GPT-6.1 Sol's Material Constraints Score Plunges 19 Points, While Code Execution Score Surges 25 Points

In today's Smoke evaluation, GPT-6.1 Sol's Material Constraints score fell from 89.30 to 70.30, a drop of 19 points, while its Code Execution score rose from 72.00 to 97.00, a gain of 25 points; its overall main leaderboard score rose from 79.79 to 84.99.

Score Comparison and Key Findings

Code Execution rose from 72.00 to 97.00, Material Constraints fell from 89.30 to 70.30, Engineering Judgment rose from 75.00 to 100.00, Task Expression fell from 84.70 to 79.70, the main leaderboard rose from 79.79 to 84.99, and the integrity rating remained pass. The 19-point single-day drop in Material Constraints is the most notable anomaly in this evaluation.

Analysis of Fluctuation Causes

The Smoke evaluation has only 10 questions per day, with 2 questions per dimension, so the randomness of the question draw directly affects scores. This sharp drop in Material Constraints may stem from the two Material Constraints questions drawn that day being more difficult or having stricter constraint requirements than yesterday's sample; the simultaneous large increases in Code Execution and Engineering Judgment indicate that the model responded more consistently to code-related and judgment-related tasks in this draw. Task Expression fell by only 5 points, a much smaller decline than Material Constraints, indicating that the regression is not global. The available data cannot distinguish whether this is question-draw fluctuation or genuine model regression; a retest with questions from the same dimensions next period is needed to confirm it.

Specific Implications for Users

Scenarios that heavily emphasize material faithfulness (such as contract extraction, literature summarization, and regulatory compliance checks) depend on the Material Constraints dimension. GPT-6.1 Sol's score of 70.30 today means that in these scenarios the model is more likely to ignore or misinterpret the provided material, and enterprise users are advised to add a manual review step before critical outputs. The Code Execution performance of 97.00 is favorable for algorithm development and data processing pipelines; teams in these areas can continue to use it, but should separately monitor Material Constraints performance. The Engineering Judgment score of 100.00 is helpful for system design review scenarios, while the Task Expression score of 79.70 has limited impact on requirements document writing.

Strategic Assessment

The 5.2-point rise in the main leaderboard was driven mainly by Code Execution and Engineering Judgment, which partially offset the 19-point drop in Material Constraints. Given the single-day sample size of 10 questions, the 19-point fluctuation is more likely to come from the question draw rather than model regression. It is recommended that next period focus on whether Material Constraints recovers above 85 points; if it remains below 75 points for two consecutive periods, then the model's priority for material-sensitive tasks should be reassessed. The current data does not support a negative conclusion about GPT-6.1 Sol's overall capability.

Data source: YZ Index | Run #373 | View raw data