Claude Sonnet 4.6 Smoke Evaluation Main Leaderboard Score Plunges 16.9 Points; Material Constraints Fall 30 Points in One Day

In today's Smoke evaluation, Claude Sonnet 4.6's main leaderboard score fell from 80.99 to 64.13, a drop of 16.9 points, mainly due to the material constraints dimension falling from 88.30 to 58.30.

Score Comparison and Data Facts

The code execution dimension fell from 75.00 to 68.90, down 6.1 points; task expression fell from 90.80 to 80.80, down 10 points; engineering judgment remained unchanged at 100.00; the integrity rating remained pass. The main leaderboard score is weighted from two auditable dimensions, code execution and material constraints, and material constraints accounted for the vast majority of this decline.

Possible Causes

The Smoke evaluation has only 10 questions per day, 2 per dimension, so single-day score fluctuations are themselves within the normal range. The material constraints dimension fell by 30 points, far exceeding the 6.1-point drop in code execution, suggesting that the two material-constraint questions drawn that day may have imposed higher requirements on the model's output fidelity, and that the model made clear errors in citation or constraint adherence. The 10-point drop in task expression may be related to expression accuracy in the same set of questions, but the zero fluctuation in engineering judgment shows that the model remains stable in side-leaderboard capabilities.

From the perspective of dimension composition, material constraints directly correspond to the model's fidelity to the given material. A 30-point single-day drop is more likely to stem from sample differences caused by question sampling rather than a systematic degradation in model capability. The simultaneous small declines in code execution and task expression further support the sampling-fluctuation explanation.

Specific Implications for Users

For scenarios that rely heavily on material constraints, such as document summarization, contract review, or knowledge-base question answering, teams that emphasize code execution need to additionally verify whether Claude Sonnet 4.6's output today shows material deviation. Engineering judgment remains at full marks, indicating that the model can still provide stable reference in auxiliary scenarios requiring engineering decisions.

If developers use Claude Sonnet 4.6 for multi-turn material-processing tasks, it is advisable to add manual spot checks, especially in finance or legal scenarios that are sensitive to material constraints. The 10-point drop in task expression also suggests that the degree of output structuring may fluctuate, requiring consistency checks in the production workflow.

Strategic Judgment

Based on the current score comparison, the 30-point drop in material constraints exceeds the normal single-day fluctuation range and merits close tracking in the next Smoke evaluation. If the material constraints score rebounds the next day, it can be judged as sampling fluctuation; if it remains below 70, then a genuine decline in the model's stability in that dimension should be considered. Engineering judgment and the integrity rating remain unchanged, indicating that core capabilities are unaffected; enterprises selecting models can continue to use it, but should set up additional verification mechanisms for material-constraint scenarios.

Data source: YZ Index | Run #391 | View raw data