GPT-6 Sol's main leaderboard score in the Smoke evaluation dropped from 90.51 to 74.14, a decline of 16.4 points, driven primarily by the code execution dimension falling from 100.00 to 66.80.
Breaking Down the Data
The code execution dimension lost 33.2 points in a single day, while the material constraints dimension rose from 78.90 to 83.10, a net gain of 4.2 points. Engineering judgment fell from 100.00 to 75.00, and task expression rose from 90.00 to 91.00. The weighted main leaderboard result directly reflects the sharp drop in code execution. The integrity rating remains pass.
Cause Analysis
The Smoke evaluation contains only 10 questions per day, 2 per dimension, making draw variance the primary possible factor. The code execution dimension may have drawn questions today with stricter requirements on specific syntax and boundary conditions, causing the score to fall directly from a perfect score to 66.80. The material constraints dimension rose instead, indicating no systematic degradation in the model's constraint-following. Engineering judgment fell by 25 points in tandem, possibly overlapping with code questions in the same batch that require multi-step reasoning, while task expression edged up 1 point, showing that expression capability was unaffected.
Distinguishing genuine degradation from draw variance requires subsequent multi-day data. A single-day drop of 33.2 points is in the extreme range at a scale of 10 questions, but the positive or flat performance in material constraints and task expression rules out the possibility of a consistent decline across all dimensions.
Implications for Users
Development teams that rely heavily on code execution should immediately add local verification steps. GPT-6 Sol still holds 83.10 in the material constraints dimension and remains usable for scenarios requiring strict adherence to prompts. An engineering judgment score of 75.00 means higher risk for code tasks involving multi-step decisions; splitting tasks and calling the model separately is recommended.
For enterprises choosing models, single-day data is insufficient to change an overall assessment, but if code execution falls below 80 for two consecutive days, testing of alternative models should be initiated.
Strategic Assessment
This decline is most likely driven by question-draw variance rather than genuine model degradation. The stable performance in material constraints and task expression supports this judgment. If code execution recovers above 90 in the next Smoke evaluation, the fluctuation interpretation can be confirmed; if it remains below 75, it should be flagged as a potential degradation signal. At the current stage, GPT-6 Sol remains usable for scenarios with high material constraint requirements, but code-execution-intensive tasks should be deprioritized.
Data source: YZ Index | Run #404 | View raw data
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接