Claude Sonnet 4.6 Material Constraints Plunge 24.8 Points, While Code Execution Surges 50 Points

In today's Smoke evaluation, Claude Sonnet 4.6's Material Constraints score fell from 77.40 yesterday to 52.60, a drop of 24.8 points, while Code Execution rose from 48.60 to 98.60, an increase of 50 points; the overall main leaderboard score rose from 61.56 to 77.90.

Score Comparison and Key Facts

The specific changes from yesterday to today are: Code Execution 48.60→98.60, Material Constraints 77.40→52.60, Engineering Judgment 50.00→62.50, Task Expression 56.30→76.30, and Integrity Rating remains pass. The main leaderboard gained a net 16.3 points, driven mainly by the substantial improvement in Code Execution.

Possible Causes

The Smoke evaluation has only 10 questions per day, with 2 questions per dimension, so single-day fluctuations are within the normal range. The simultaneous plunge in Material Constraints and surge in Code Execution is most likely due to the randomness of question selection: today's Material Constraints questions may have imposed higher fidelity requirements, while the Code Execution questions were relatively simple or better matched the model's current strengths. Another possibility is temporary instability in the model's material-handling stage, but Engineering Judgment and Task Expression both improved, indicating that the overall response framework did not collapse.

Because there is no repeated-test data for the same type of questions over multiple consecutive days, it is impossible to distinguish question variation from genuine degradation. A single-day drop of 24.8 points is an extreme value in a 10-question quick test, but the opposite 50-point rise in Code Execution is equally extreme, pointing to a higher probability of question-selection effects.

Practical Implications for Users

For teams in material-constraint-heavy scenarios—for example, RAG applications that need to generate answers strictly based on given documents or avoid hallucinations—today's 52.60 means the model has a clear weakness in fidelity. It is advisable to temporarily switch to a model with a higher Material Constraints score or add a manual verification step.

Developers who rely on Code Execution can take advantage of today's 98.60 score; success rates may be higher when handling complex algorithms or debugging tasks. With both Engineering Judgment and Task Expression up, this has a positive impact on scenarios requiring structured output and decision support.

Strategic Assessment

The Material Constraints dimension is one of the two auditable cores of the main leaderboard. A single-day drop of 24.8 points has exceeded the normal fluctuation range and is worth continued tracking in the next period. If Material Constraints rebounds tomorrow, it can basically be determined to be caused by question selection; if it remains low, the risk of genuine model degradation needs to be considered. The current main leaderboard score of 77.90 is being propped up by Code Execution, and the shortcoming in Material Constraints has been partially masked. When selecting models, users should look at this dimension separately rather than only at the main leaderboard.

The Integrity Rating remains pass, indicating that no systematic fabrication or refusal issues appeared in the responses, and data credibility has not been affected. The Stability dimension did not provide a specific value this time, but the extreme single-day fluctuation already indicates that consistency may be low, and production environments that rely on stable output need additional verification.


Data source: YZ Index | Run #439 | View raw data

This article is from Winzheng Index blog, translated in full by Winzheng (winzheng.com). Click here to view the original When republishing the translation, please credit the source. Thank you!