GPT-6.1 Sol Code Execution Plunges 27.3 Points as Overall Ranking Slips to 78.69

In today's Smoke evaluation, GPT-6.1 Sol's code execution score fell from 97.30 to 70.00, and its overall ranking dropped from 84.70 to 78.69.

Score Change Facts

The code execution dimension moved by -27.3 points, material constraints rose from 69.30 to 89.30, engineering judgment fell from 100.00 to 94.80, and task expression fell from 100.00 to 85.00. The integrity rating remained "pass."

Causes

The Smoke evaluation contains only 10 questions per day, with 2 questions per dimension. A single error on a code execution question can produce a 27.3-point swing. Material constraints rose by 20 points over the same period, indicating no systematic degradation in the model's adherence to constraints. The modest declines in engineering judgment and task expression are likewise consistent with small-sample question sampling rather than a collapse in overall capability.

If this were genuine model degradation, it would typically be accompanied by simultaneous declines across multiple dimensions, but the reverse movement in material constraints is inconsistent with that. Question sampling variation is therefore the more likely cause.

Implications for Users

Development teams that rely heavily on code execution should note that a single-day score of 70.00 may indicate reduced reliability in generated code. In scenarios requiring stable code output, adding a manual review step is advisable.

For document generation or compliance scenarios sensitive to material constraints, today's 89.30 score actually offers stronger assurance and can serve as a short-term alternative.

Strategic Assessment

The 6-point decline in the overall ranking was driven mainly by code execution. Given the limited scale of the test, this change is not sufficient to conclude that the model has entered a degradation phase. If code execution rebounds above 90 in the next Smoke evaluation, this can be confirmed as a sampling event; if it remains below 80, consistency will need further verification.

Current data do not support a broad downgrade of GPT-6.1 Sol; simply observing the standard deviation of code execution scores over the next two rounds should yield a more reliable conclusion.


Data source: YZ Index | Run #429 | View raw data

This article is from Winzheng Index blog, translated in full by Winzheng (winzheng.com). Click here to view the original When republishing the translation, please credit the source. Thank you!