In today's Smoke evaluation, GPT-5.5's code execution dimension fell from yesterday's 99.80 to 75.00, a drop of 24.8 points, while the overall main leaderboard score declined only from 81.58 to 78.74.
Data Facts: Dimension Score Comparison
Code execution: 99.80→75.00 (-24.8)
Material constraints: 59.30→83.30 (+24)
Engineering judgment (side leaderboard, AI-assisted evaluation): 100.00→75.00 (-25)
Task expression (side leaderboard, AI-assisted evaluation): 100.00→90.00 (-10)
Main leaderboard: 81.58→78.74 (-2.8)
Integrity rating remains pass.
The Smoke evaluation has only 2 questions per dimension each day, so the sample size is extremely small, and the daily score standard deviation is naturally large. Code execution and engineering judgment both showed roughly 25-point declines at the same time, pointing to the possibility that the 2 code questions sampled that day triggered execution failures by the model under specific constraints.
Cause Analysis: Sampling Volatility or Real Degradation
Material constraints rose by 24 points instead, indicating that the model's control over material faithfulness actually strengthened on the same batch of questions. These two main-leaderboard dimensions moved in opposite directions, ruling out the possibility of systemic degradation in the model's overall capabilities. The simultaneous sharp decline in engineering judgment (side leaderboard, AI-assisted evaluation) is more likely because code execution failures directly broke the subsequent engineering decision chain.
The most likely mechanism is question sampling: that day's code questions may have included stricter boundary conditions or multi-step dependencies, precisely hitting the model's weak points in long-chain execution. The rise in material constraints shows that the model did not become "less intelligent"; rather, its performance diverged under different constraints.
Implications for Users
Teams that place heavy emphasis on code execution should factor GPT-5.5's daily volatility into their planning when selecting models. If the application scenario requires multiple consecutive rounds of code generation and debugging, today's 75.00 indicates that the model has clear weaknesses on specific questions, and it is advisable to add a human review step or call other models in parallel.
For scenarios sensitive to material faithfulness (such as contract extraction and report generation), today's 83.30 provides a positive signal instead, indicating that GPT-5.5 can still maintain relatively high output quality when constraints tighten.
Strategic Judgment
Based on current data, GPT-5.5's main leaderboard ranking is dragged down by single-day code execution volatility, but the simultaneous rise in material constraints shows that its core capabilities have not collapsed. The next evaluation should focus on whether code execution rebounds above 90; if it stays below 80 for two consecutive days, long-term expectations for its code execution should be lowered.
The same-magnitude decline in engineering judgment (side leaderboard, AI-assisted evaluation) and code execution confirms strong coupling between the two in real tasks. Downstream applications that rely on engineering judgment should activate a contingency plan to switch to an alternative model when Smoke evaluation volatility exceeds 20 points consecutively.
The integrity rating remains pass, indicating that the model has not shown systemic answer inconsistency or a surge in hallucinations. Overall, this sharp drop is more likely a sampling event than model degradation and warrants continued tracking rather than an immediate conclusion.
Data source: YZ Index | Run #432 | View raw data
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接