In today’s Smoke evaluation, GPT-o3’s main leaderboard score fell from 78.08 yesterday to 57.92, a drop of 20.2 points; its code execution dimension fell from 73.80 to 45.50.
Score Details and Direct Comparison
The code execution dimension scored 73.80 yesterday and 45.50 today, a drop of 28.3 points; material constraints scored 83.30 yesterday and 73.10 today, a drop of 10.2 points; engineering judgment scored 75.00 yesterday and 97.00 today, an increase of 22 points; task expression scored 85.00 yesterday and 81.70 today, a drop of 3.3 points. The integrity rating remains pass.
Breaking Down the Data Facts
The Smoke evaluation uses a fixed set of 10 questions each day, with 2 questions per dimension. The code execution dimension’s single-day score fluctuated by 28.3 points, far exceeding the 10.2-point drop in material constraints. Engineering judgment moved in the opposite direction, rising by 22 points, showing that the direction in which different dimensions are affected by question draws is not consistent.
Cause Analysis: Sampling Fluctuation or Model Degradation
With only 2 code execution questions each day, question difficulty and type are highly random, and each question carries significant weight. The 28.3-point drop may stem from today’s draw containing a combination of questions with higher code execution requirements, rather than an overall decline in model capability. The smaller drop in material constraints indicates that the model’s performance on material faithfulness is relatively stable. The large rise in engineering judgment further supports the explanation of question draws rather than model degradation, because the same model showing opposite changes in different dimensions at the same time is more consistent with sampling fluctuation.
Specific Implications for Users
Development teams that rely heavily on code execution should note that a single-day score of 45.50 means GPT-o3’s performance today in scenarios requiring continuous generation of runnable code is significantly lower than yesterday. Material constraints remain at 73.10, so the impact on scenarios requiring strict adherence to input documents is relatively small. Engineering judgment at 97.00 indicates that the model remains highly usable in tasks requiring engineering trade-offs.
Strategic Judgment
Based on the current single-day data, the 28.3-point drop in the code execution dimension has exceeded the range of normal small fluctuations and is worth continuing to track in the next Smoke evaluation. If code execution scores rebound above 70 over the next two days, this decline is more likely attributable to question draws; if they remain below 50, the model’s real changes in the code execution direction should be considered. The 22-point rise in engineering judgment does not change the main leaderboard judgment, because the main leaderboard includes only the code execution and material constraints dimensions.
This evaluation shows that GPT-o3 recorded a clear single-day low in the code execution dimension, while the drop in material constraints was relatively moderate. Users can decide, based on their own scenario priorities, whether to postpone using GPT-o3 for tasks with high code-execution intensity until the next round of evaluation results is available.
Data source: YZ Index | Run #436 | View raw data
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接