In today's Smoke evaluation, GPT-6 Astra's main leaderboard score fell from 82.33 to 71.01, and the code execution dimension dropped directly from 97.00 to 72.00.
Data Facts
Smoke evaluation has 10 fixed questions each day, with 2 questions per dimension. Today's code execution dimension score was 72.00, down 25 points from yesterday's 97.00; the material constraint dimension rose from 64.40 to 69.80, up 5.4 points; the two side-leaderboard dimensions, engineering judgment and task expression, remained unchanged at 75.00 and 100.00; the integrity rating remained pass. The weighted result of the main leaderboard therefore fell by 11.3 points.
The single-day drop in the code execution dimension far exceeded the rise in the material constraint dimension, causing the overall main leaderboard to decline. Engineering judgment and task expression showed no fluctuation, indicating that the model remained stable in side-leaderboard capabilities.
Possible Causes
Smoke evaluation questions are drawn daily, and a sample size of 2 questions is small, so single-day score fluctuations are within the normal range. The large drop in the code execution dimension today is most likely because the drawn questions placed higher demands on the model's code generation or debugging capabilities, and the model failed to provide correct solutions to these specific questions.
The slight rise in the material constraint dimension shows that the model has no systematic problems in citation constraints and faithfulness. Engineering judgment and task expression kept their scores, indicating that the model's overall architecture and expression capabilities were not affected by this draw.
If this decline was caused only by question difficulty, it is statistical fluctuation; if similar low scores in the same dimension continue for multiple days, it may point to real model degradation. Currently there is only one day of data, so the two cannot be distinguished.
Implications for Users
Development teams that rely heavily on code execution should note that GPT-6 Astra's stability in code generation scenarios has shown obvious fluctuation. Today's 72.00 means that in similar code tasks, the model's accuracy may be significantly lower than the level corresponding to yesterday's 97.00.
Scenarios sensitive to material constraints, such as document generation or knowledge extraction, are less affected because this dimension's score actually rose. Engineering judgment and task expression remain at full or high scores, making the model suitable for tasks requiring structured output and judgment.
When enterprises select models, they need to include single-day fluctuations in the code execution dimension in risk assessment. Especially in automated testing or script-generation pipelines, it is recommended to add manual review steps.
Strategic Judgment
The 25-point drop in the code execution dimension has exceeded the normal threshold for draw fluctuation and deserves key verification in the next Smoke evaluation. If this dimension rebounds above 90 the next day, the decline is more likely attributable to question drawing; if it remains below 80, changes in the model's real capabilities should be considered.
The 11.3-point decline in the main leaderboard was mainly driven by the single dimension of code execution, while other dimensions did not deteriorate simultaneously, indicating that the model has not comprehensively degraded. The zero fluctuation in engineering judgment and task expression further confirms this.
For core users who rely on code execution, the current data suggest that trust in GPT-6 Astra should be temporarily reduced. It is preferable to observe score changes in the same dimension over the next two days before deciding whether to adjust production environment configurations.
Data source: YZ Index | Run #387 | View raw data
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接