GPT-o3 Main Leaderboard Plummets 15 Points: Code Execution Falls from 100 to 75

In today's Smoke evaluation, GPT-o3's main leaderboard score fell from 85.15 to 70.19, a drop of 15 points, mainly driven by the code execution dimension falling from 100.00 to 75.00.

Breaking Down the Score Facts

The code execution dimension fell 25 points in a single day; material constraints dropped from 67.00 to 64.30, a decline of 2.7 points. Engineering judgment remained unchanged at 100.00, while task expression rose from 85.80 to 100.00. The integrity rating remained pass. The Smoke evaluation uses a fixed set of 10 questions per day, 2 per dimension; the sample size is small, so single-day fluctuations are within the normal range.

Analysis of the Causes of the Fluctuation

The code execution dimension saw a 25-point drop, while material constraints fell only 2.7 points, and the other two side-leaderboard dimensions were flat or up, indicating the issue is concentrated in the randomly drawn code execution questions. The Smoke evaluation has only 2 questions per dimension, so one high-difficulty or edge-case question can pull the score down by 25 points. The small decline in material constraints is out of sync with the code execution drop, pointing to question randomness rather than an overall degradation in model capability. Engineering judgment and task expression were unaffected, further supporting that this change mainly comes from the draw of code execution questions.

Implications for Users

Teams that rely heavily on code execution should add local validation steps after today; a Smoke evaluation score of 75 means the model may make obvious errors on certain code tasks. A material constraints score of 64.30 still poses risks for scenarios that require strict adherence to input constraints, and enterprises should retain manual review steps when selecting models. A perfect score in engineering judgment and 100 in task expression indicate the model remains stable in judgment and expression, making it suitable for applications that require strong reasoning but relatively little code.

Strategic Assessment

The current data only shows a single-dimension, single-day fluctuation and does not provide a multi-day trend, so it is impossible to determine whether the model has truly degraded. The difference between the 25-point drop in code execution and the 2.7-point drop in material constraints suggests that random draw is the more likely factor. It is recommended that the next Smoke evaluation focus on whether code execution rebounds to above 90; if it remains below 80 for two consecutive days, then initiate deeper re-testing. The stable performance of engineering judgment and task expression can serve as a short-term usage reference, while the code execution dimension requires additional monitoring.

In this evaluation, the comparison between code execution's 75.00 today and 100.00 yesterday directly reflects a change in question difficulty, not an adjustment to model parameters. The slight difference between material constraints' 64.30 and 67.00 also fits the characteristics of small-sample random fluctuation. Although the overall main leaderboard score of 70.19 is jarring, the side leaderboard dimensions show no systemic decline; therefore, this event is judged to be primarily random draw fluctuation, with insufficient evidence of true capability degradation.

For developers who rely on GPT-o3, today's data suggests increasing unit test coverage in code generation scenarios. When selecting models, enterprises can continue to use this model for engineering judgment and task expression tasks, but the code execution step should have manual or tool-based secondary validation. If code execution rebounds in the next round of data, then this 70.19 is merely the lower bound of the normal range; if it remains low, the weighting of the code execution dimension should be reassessed.

Data source: YZ Index | Run #322 | View raw data