Claude Sonnet 4.6 Main Leaderboard Plunges 12.9 Points as Code Execution Falls from 100 to 76.1

In today's Smoke evaluation, Claude Sonnet 4.6's main leaderboard score fell from 87.36 yesterday to 74.44, a single-day drop of 12.9 points. The code execution dimension alone dropped from 100.00 to 76.10, a decline of 23.9 points.

Data Breakdown: A Single Dimension Drives the Decline

The material constraint dimension rose from 71.90 to 72.40, an increase of just 0.5 points. The engineering judgment side leaderboard fell from 57.50 to 42.90, while the task expression side leaderboard rose from 57.50 to 63.80. The integrity rating remained at pass. The main leaderboard is weighted from only two dimensions—code execution and material constraint—so the 23.9-point drop in code execution directly pulled the overall score down by 12.9 points.

Cause Analysis: Question Sampling Variance Is the More Likely Explanation

The Smoke evaluation uses only 2 questions per dimension each day, an extremely small sample. A 23.9-point single-day loss in the code execution dimension most likely stems from drawing two questions that the model currently handles poorly, rather than from any degradation in model parameters or reasoning capability within a 24-hour window. The material constraint dimension remained stable over the same period, indicating no systemic problem with the model's underlying ability to stay faithful to source materials.

Genuine degradation would typically show up across multiple dimensions simultaneously. Here, only code execution swung sharply, while the task expression side leaderboard actually rose 6.3 points—further supporting the conclusion that the fluctuation comes from question randomness rather than an overall decline in model capability.

What This Means for Users

Teams that rely heavily on code generation should note that Claude Sonnet 4.6's code execution score in the Smoke evaluation has shown single-day swings on the order of 23.9 points. In production environments, complex algorithm or multi-step reasoning tasks should include an added human review step or run parallel calls to other models.

For scenarios with high demands on material fidelity—such as contract review or report generation—Claude Sonnet 4.6's material constraint score of 72.40 today remains within a usable range, so the impact is relatively limited.

Strategic Assessment

Based on the current single-day data, there is a risk that Claude Sonnet 4.6's code execution capability has been overestimated, but the evidence is still insufficient to conclude that the model has entered a sustained degradation phase. If the code execution score rebounds above 90 in the next Smoke evaluation, this episode can be confirmed as question sampling variance; if it remains below 80, the model's true stability will require further validation.

The only signal that can be confirmed right now is this: the code execution dimension shows high variance in small-sample rapid testing, and developers who depend on it should build a multi-day average score tracking mechanism rather than adjusting their model selection based on single-day results.

Data source: YZ Index | Run #402 | View raw data