In today's Smoke evaluation, Claude Sonnet 4.6's main leaderboard score fell from 84.90 to 73.76, a decline of 11.1 points.
Breaking Down the Score Change
The code execution dimension dropped from 88.90 to 78.80, a single-day loss of 10.1 points; material constraint fell from 80.00 to 67.60, a loss of 12.4 points. Both main leaderboard dimensions posted double-digit declines at the same time, directly dragging down the overall main leaderboard. Engineering judgment rose from 50.00 to 60.50, and task expression rose from 32.50 to 65.00, marking a clear rebound on the side leaderboard. The integrity rating changed from pass to warn.
Analysis of the Volatility
The Smoke evaluation uses only 10 questions per day, 2 per dimension, so the random drawing of questions is itself subject to chance. The simultaneous declines in code execution and material constraint suggest that the questions drawn that day may have concentrated their scrutiny on the model's specific execution paths or fidelity to source material. The rise in side leaderboard scores may reflect that the model happened to match the day's drawn content on engineering judgment and task expression questions, but this does not offset the losses in the main leaderboard's core dimensions. The shift of the integrity rating to warn indicates recordable issues in the model's answer consistency or boundary handling, which echoes the main leaderboard decline.
What This Means for Users
Development teams that rely heavily on code execution may encounter more execution failures or output deviations today and should add extra manual review steps. In scenarios sensitive to material constraint, such as contract summarization and knowledge-base question answering, model output may show more omissions or deviations from the source text; temporarily switching to a more stable alternative model is recommended. Higher scores in engineering judgment and task expression offer some help for scenarios requiring structured output, but they cannot cover the decline in the main leaderboard's core capabilities.
Strategic Assessment
This 11.1-point main leaderboard decline was driven jointly by code execution and material constraint, and was accompanied by a change in the integrity rating; the single-day swing already exceeds the normal range of random question drawing and constitutes a signal that warrants continued tracking. If the main leaderboard remains below the 75-point range in the next evaluation round, a systematic performance regression can be confirmed; if it rebounds quickly, this decline is more likely attributable to a concentrated draw of questions. The current data is sufficient to place Claude Sonnet 4.6 on a priority watch list.
Data source: YZ Index | Run #410 | View raw data
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接