GPT-6 Astra Material Constraint Score Plummets 29.1 Points; Code Execution Rises 19.5 Points

In today's Smoke evaluation, GPT-6 Astra's material constraint score fell from 89.80 to 60.70, a drop of 29.1 points, while code execution rose from 75.00 to 94.50; its overall main leaderboard score declined from 81.66 to 79.29.

Score Comparison Data

On the code execution dimension, the score was 75.00 yesterday and 94.50 today, up 19.5 points; on the material constraint dimension, it was 89.80 yesterday and 60.70 today, down 29.1 points; engineering judgment fell from 100.00 to 75.00; task expression rose from 87.50 to 90.00; the integrity rating remained pass.

Cause Analysis

The Smoke evaluation has only 10 questions per day, with 2 questions per dimension, so single-day score standard deviation is naturally large. The large changes in material constraints and code execution moving in completely opposite directions most likely stem from random fluctuation in question sampling rather than a real deterioration in model capability. The material constraint questions may have concentrated on strict material fidelity requirements, precisely hitting the model's weakness under specific constraints; the code execution questions, by contrast, may have drawn scenarios the model excels at, leading to polarization.

The Engineering Judgment side leaderboard (AI-assisted evaluation) fell by 25 points, further confirming that the day's question mix had high difficulty variance. The small 2.5-point rise in task expression shows that the model remained relatively stable at the expression level.

Implications for Users

Enterprises in scenarios that heavily rely on material constraints (such as contract generation and knowledge base Q&A) should note that a single Smoke score of 60.70 means there may be more hallucinations or deviations in tasks requiring strict quotation of source text. Developers who rely on code execution can continue to use the model; 94.50 shows it still has an advantage in code correctness.

For teams with mixed scenarios requiring both material constraints and code execution, it is advisable to observe over several consecutive days before making a selection decision, to avoid being misled by single-day fluctuations.

Strategic Judgment

Based on the available score comparison, the 29.1-point drop in material constraints is most likely caused by question sampling rather than a real deterioration in the model. The main leaderboard's decline of only 2.4 points also supports this judgment. If material constraints rebound above 80 in the next evaluation period, it can be confirmed that the day's result was normal fluctuation; if it remains below 70, further verification will be needed of the model's stability in dimensions concerning source material.

Current data does not support a negative strategic downgrade of GPT-6 Astra; continued observation is recommended.

Data source: YZ Index | Run #414 | View raw data