Claude Sonnet 4.6 Falls 15.2 Points on Material Constraints, Main Leaderboard Slips 5.5 Points

In today's Smoke evaluation, Claude Sonnet 4.6's Material Constraints score fell from 88.30 to 73.10, a drop of 15.2 points, directly pulling the main leaderboard down from 77.41 to 71.89.

Score Comparison and Main Leaderboard Composition

The Code Execution dimension rose from 68.50 to 70.90, a gain of 2.4 points. The Material Constraints dimension fell from 88.30 to 73.10, a drop of 15.2 points. The main leaderboard is composed solely of the weighted Code Execution and Material Constraints scores, so the single-day plunge in Material Constraints is the only source of the 5.5-point decline in the main leaderboard. Engineering Judgment rose from 75.00 to 100.00, and Task Expression rose from 38.80 to 70.00; both are side-leaderboard metrics and are not counted toward the main leaderboard.

Analysis of Possible Causes

The Smoke evaluation uses only 10 questions per day, 2 per dimension, and the randomness of question selection is the primary factor. The 2 questions drawn for the Material Constraints dimension that day may have placed higher demands on instruction following or citation fidelity, causing large score swings. Code Execution edged up over the same period, indicating that the model has not shown systematic degradation in that direction. Engineering Judgment and Task Expression on the side leaderboards both rose sharply at the same time, further indicating that the model's overall output style has not collapsed, and that the problem is more likely concentrated in the sampling of specific Material Constraints questions.

Material Constraints at 73.10 points vs. 88.30 yesterday; the single-day swing far exceeds Code Execution's 2.4-point gain.

Implications for Users

Scenarios that rely heavily on source fidelity should be on alert. For enterprises that need the model to generate answers strictly from provided materials or to quote source text, today's 73.10 level means a higher risk of omissions or rewrites in outputs. Code Execution still holds at 70.90, so teams focused mainly on code generation are less affected. The integrity rating remains pass, and the model showed no systematic fabrication or refusal to answer questions.

Strategic Assessment

The current data most likely points to fluctuation in question selection rather than genuine model degradation. The combination of a 15.2-point single-day drop in Material Constraints and sharp simultaneous rises in two side-leaderboard metrics is consistent with normal fluctuation caused by random differences in question difficulty. We recommend continuing to track the Material Constraints score in the next period, and only considering a change in model capability if it stays below 80 for two consecutive days. The main leaderboard's 71.89 is already below yesterday's level, but Code Execution remains on an upward trend; when selecting models, continue to observe how Material Constraints performs going forward.


Data source: YZ Index | Run #436 | View raw data

This article is from Winzheng Index blog, translated in full by Winzheng (winzheng.com). Click here to view the original When republishing the translation, please credit the source. Thank you!