GPT-o3 Code Execution Plunges 22.5 Points as Main Leaderboard Slips 7.8 Points

In today's Smoke evaluation, GPT-o3's code execution score fell from 94.50 to 72.00, while the overall main leaderboard dropped from 92.39 to 84.60.

Score Comparison and Dimension Breakdown

Material Constraints rose from 89.80 to 100.00, Task Expression rose from 70.00 to 91.70, Engineering Judgment held steady at 100.00, and the integrity rating remained "pass." The main leaderboard is composed of only two auditable dimensions, code execution and material constraints; today's 7.8-point decline on the main leaderboard was driven mainly by a 22.5-point drop in the single dimension of code execution.

Possible Causes

The Smoke evaluation includes only 2 code execution questions per day, an extremely small sample, so a single missed question can cause large swings in the dimension score. The gap between today's 72.00 in code execution and yesterday's 94.50 most likely stems from differences in the difficulty of the questions drawn, rather than a degradation in model capability. The simultaneous rise in material constraints and task expression shows that the model's answer stability on other dimensions has not run into systemic problems.

If this were genuine model degradation, it would typically be accompanied by simultaneous declines across multiple dimensions or a change in the integrity rating. But in today's data only the code execution dimension was abnormal, while the remaining dimensions either improved or held flat, so the draw-variance explanation is more compelling.

What This Means for Users

Development teams that rely heavily on code execution should add a manual review step after today, especially when generating complex algorithms or multi-step reasoning code. A perfect score on material constraints indicates that GPT-o3 performs stably in scenarios requiring strict adherence to prompts and output formats, making it suitable for document generation tasks that demand high fidelity.

Engineering Judgment remained at a perfect score, so the impact on scenarios requiring structured decision-making is limited. Task Expression improved by 21.7 points, indicating that the model has become better at clearly expressing task results, which can be put to use in report writing that requires high-quality output.

Strategic Assessment

Based on a single day of data, the decline in GPT-o3's code execution score is more likely draw variance than genuine degradation, so there is no need to immediately lower the model's priority. If code execution remains below 80 in the next evaluation round, the model's consistency should be re-verified. The current main leaderboard score of 84.60 is still higher than that of some competing products, and enterprise customers evaluating options can continue to observe three consecutive days of data before making adjustments.

Data source: YZ Index | Run #380 | View raw data