GPT-5.5 Smoke Evaluation Main Leaderboard Plummets 10.1 Points; Material Constraints Drop 16.3 in Single Day

GPT-5.5 scored 88.27 on the main leaderboard in today's Smoke evaluation, down from 98.35 — a decline of 10.1 points.

Score Change Breakdown

The code execution dimension fell from 97.00 to 92.00, the material constraints dimension dropped from 100.00 to 83.70, the engineering judgment dimension held steady at 75.00, and the task expression dimension declined from 91.70 to 81.70. The main leaderboard is weighted solely from the code execution and material constraints dimensions, so the 16.3-point drop in material constraints directly dragged down the overall score.

Potential Causes

The Smoke evaluation samples only 2 questions per dimension daily, and with such a small sample size, single-day fluctuations are within normal range. The material constraints dimension saw the largest drop, suggesting that the 2 material constraint questions drawn that day may have included higher-difficulty fidelity requirements or edge cases that pushed the model's output away from the constraints. The code execution dimension fell by only 5 points, a relatively mild decline, indicating a fairly stable difficulty distribution for code-related questions. The engineering judgment dimension showed zero change, suggesting that this side-leaderboard dimension was minimally affected by the day's questions. The task expression dimension fell 10 points, possibly compounded by implicit expression requirements embedded in the material constraint questions.

Both question-draw fluctuation and genuine model degradation are plausible explanations. The current data only represents a single-day snapshot and cannot distinguish between the two. If material constraints continue to stay below 90 in the coming days, it would point more toward a real capability change; if scores rebound within a single day, the draw is the more likely culprit.

Implications for Users

Teams that prioritize code execution should note that GPT-5.5 maintained a 92.00 score on that dimension for the day and can be used as a code generation aid in the short term, but manual verification steps should be added to hedge against potential fluctuations of around 5 points. For scenarios sensitive to material fidelity — such as contract extraction and knowledge base Q&A — the 83.70 material constraint score means the risk of output deviating from constraints has increased. It is advisable to add secondary verification or rollback mechanisms to production pipelines.

The engineering judgment dimension remained stable at 75.00, limiting its impact on architecture decision-making scenarios that rely on this side leaderboard. The task expression dimension at 81.70 suggests that more explicit prompt engineering is needed to maintain output structure.

Strategic Assessment

Based on single-day data, GPT-5.5's main leaderboard score was amplified by that day's material constraint questions. The current 88.27 may either underestimate its typical performance or reflect genuine volatility. The next evaluation round should focus on verifying whether the material constraints dimension rebounds above 95. If material constraints remain below 85 for two consecutive days, its priority in high-constraint scenarios should be reassessed.

The unchanged engineering judgment score indicates the model's capability on this side leaderboard is relatively robust and can serve as a comparison baseline. Overall, this drop was driven primarily by the material constraints dimension alone and does not yet constitute a signal of multi-dimensional systemic degradation.

The material constraints drop from 100.00 to 83.70 is the core driver behind the main leaderboard's sharp decline.

For developers relying on GPT-5.5, the short-term strategy is to increase test case coverage density for material constraint scenarios and track the three-day rolling standard deviation of scores. If the standard deviation exceeds 8 points, the stability dimension will face further pressure.

This Smoke evaluation shows that GPT-5.5 exhibited pronounced single-day volatility in the material constraints dimension, while code execution proved relatively resilient. When selecting models, enterprises should treat material constraints as an independent monitoring metric rather than relying solely on the main leaderboard average.

Data source: YZ Index | Run #286 | View raw data