In today's Smoke evaluation, GPT-o3's code execution score fell from 98.80 to 71.60, a drop of 27.2 points, while its material constraints score rose from 51.70 to 86.70, an increase of 35 points; its main leaderboard score rose from 77.61 to 78.40.
Score Details and Direct Comparison
On the code execution dimension, the score was 98.80 yesterday and 71.60 today; engineering judgment fell from 87.50 to 75.00, task expression rose from 77.90 to 100.00, and the integrity rating remained pass. The main leaderboard rose by only 0.8 points, indicating that after one decline and one increase in the two core auditable dimensions, the overall result remained stable.
Analysis of the Causes of the Fluctuation
The Smoke evaluation has only 10 questions per day and 2 questions per dimension, so the sample size is extremely small, and the standard deviation of daily scores is easily amplified. The large moves in opposite directions for code execution and material constraints most likely stem from changes in the difficulty distribution of the day's question draw, rather than degradation of the model parameters themselves. Engineering judgment fell 12.5 points and task expression rose 22.1 points, further confirming the independence with which different dimensions are affected by question-bank randomness.
What This Means for Users
Development teams that rely heavily on code execution need to add local verification steps when calling GPT-o3, especially when handling complex algorithms or multi-step reasoning tasks, because the uncertainty of a single answer has risen from yesterday's low level. In scenarios where the material constraints score has risen sharply, automated workflows that depend on the model to strictly follow input constraints or formatting requirements can achieve a higher success rate.
Strategic Judgment
Based on single-day data, GPT-o3's main leaderboard score remains in the 78.40 range, and the 27.2-point drop in code execution is not yet evidence of a trend decline on the main leaderboard. The stability dimension score of 31.7 already shows that the model has low consistency in small-sample tests. It is advisable to continue tracking the standard deviation of scores on the same dimension for three consecutive days in the next period; if similar fluctuations occur repeatedly, its reliability in production environments will need to be reassessed.
Current data support only the judgment that "sampling fluctuation is the main cause" and do not support the conclusion that "the model has truly degraded." Enterprises can continue to include GPT-o3 in their candidate list during model selection, but they need to set up a daily spot-check mechanism for code execution scenarios.
Data source: YZ Index | Run #438 | View raw data
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接