GPT-6.1 Sol Material Constraint Score Plummets 29.1 Points as Code Execution Rises 19.5 Points

In today's Smoke evaluation, GPT-6.1 Sol's material constraint score fell from 89.80 to 60.70, a drop of 29.1 points, while its code execution score rose from 75.00 to 94.50, and its main leaderboard score fell from 81.66 to 79.29.

Core Dimension Comparison Data

The main leaderboard consists solely of two auditable dimensions: code execution and material constraint. Today's material constraint score of 60.70 compares with yesterday's 89.80, a single-day drop of 29.1 points; code execution at 94.50 compares with yesterday's 75.00, a single-day rise of 19.5 points. After weighting the two, the main leaderboard fell by a net 2.4 points. Engineering judgment fell from 100.00 to 75.00, task expression fell from 87.50 to 85.00, and the integrity rating remained pass.

Analysis of Fluctuation Causes

The Smoke evaluation has only 10 questions per day, with 2 per dimension, so the sample size is extremely small. The simultaneous 29.1-point drop in material constraint and 19.5-point rise in code execution most likely stems from the random distribution of question sampling rather than systematic degradation of model parameters. If material constraint questions cluster in scenarios requiring strict citation of provided materials and prohibiting external knowledge, a single model error could lower the overall score for that dimension. If code execution questions happen to fall into algorithm or debugging question types the model is good at, the score naturally rises. The opposite movement of the two dimensions further supports the sampling-fluctuation explanation.

The contrast between the 60.70 material constraint score and the 94.50 code execution score points to a structural change in question content rather than in the model's capabilities themselves.

Specific Implications for Users

For enterprise teams that rely heavily on material faithfulness, GPT-6.1 Sol's material constraint performance of 60.70 today means that in scenarios requiring strict adherence to input documents and prohibiting hallucinations, the risk of a single call rises significantly. The 94.50 code execution score shows the model remains highly usable for pure programming tasks. Developers relying on engineering judgment should note its side-leaderboard score of 75.00, suggesting that in stages requiring architectural decisions or trade-offs, the model's output consistency may be lower than expected.

Strategic Judgment

Based on current data, GPT-6.1 Sol's main leaderboard score of 79.29 remains in the usable range, but the single-day 29.1-point fluctuation in material constraint has exceeded the boundary of what normal sampling can explain. If material constraint remains below 70 in the next Smoke evaluation, the possibility of genuine model degradation should be considered; if it rebounds above 80, today can be judged an isolated sampling event. The side-leaderboard engineering judgment score of 75.00 and material constraint score of 60.70 declining in tandem are worth focusing on next period to verify whether the two share a common mechanism.

The integrity rating remains pass, indicating that the model has not shown systematic fabrication or avoidance of answering questions. The stability dimension did not provide a specific value this time, so response consistency cannot be judged on that basis. Overall, the most direct signal from today's data is that the material constraint dimension is highly sensitive to small-sample questions; users should add manual review steps in critical tasks.

Data source: YZ Index | Run #414 | View raw data