Claude Sonnet 4.6 Smoke Evaluation Main Leaderboard Plunges 11.7 Points; Material Constraint Falls 19.9 in a Single Day

Claude Sonnet 4.6's main leaderboard score fell from 82.31 to 70.61 in today's Smoke evaluation, a decline of 11.7 points.

Score Breakdown: Material Constraint Drives the Drop

The main leaderboard is weighted solely from code execution and material constraint. Today code execution fell from 99.50 to 94.50, down 5 points; material constraint fell from 61.30 to 41.40, down 19.9 points. The single-dimension decline in material constraint accounts for the overwhelming majority of the main leaderboard's total drop, directly dragging down the overall ranking.

Engineering judgment rose from 60.80 to 75.00, while task expression held steady at 37.50. The integrity rating remains pass.

Cause Analysis: Sampling Noise or Genuine Degradation

The Smoke evaluation uses only 10 questions per day, 2 per dimension. A 19.9-point single-day drop in material constraint could be caused by question sampling under such a small sample, or it could reflect a systematic deviation in how the model handles fidelity to source material. Code execution fell only 5 points, a magnitude within normal fluctuation range. Engineering judgment moved in the opposite direction, rising 14.2 points, indicating that the model's performance on side-leaderboard tasks did not decline in tandem.

The current data cannot distinguish a one-off sampling anomaly from genuine capability degradation. If material constraint questions involve long-text citation or instruction conflicts, a single question error can cause a large score swing.

Implications for Users

Scenarios that rely heavily on material constraint are the most affected. Enterprise RAG systems, contract review tools, and knowledge-base Q&A applications that require the model to strictly follow uploaded documents or instructions should immediately add manual verification steps. Code execution scenarios are less affected; routine script generation and debugging can continue.

For developers, the rise in engineering judgment shows the model still has headroom on architecture design tasks, and it can be retained in hybrid workflows to handle side-leaderboard tasks.

Strategic Assessment

The 19.9-point single-day drop in material constraint exceeds the normal sampling range and constitutes a signal that requires verification in the next period. If the material constraint score rebounds above 55 tomorrow, the decline is more likely attributable to question fluctuation; if it remains below 45, genuine model degradation must be considered.

With only one period of data so far, a conclusion of "continuous degradation" is not supported. It is recommended that Claude Sonnet 4.6's material constraint performance be placed on the daily tracking list, with model selection adjustments made only after three consecutive periods of observation.

Data source: YZ Index | Run #426 | View raw data