Claude Opus 4.7 Material Constraint Score Plunges 17.6 Points While Code Execution Jumps 24.8, Main Leaderboard Rises 5.7

In today's Smoke evaluation, Claude Opus 4.7's material constraint score fell from 70.00 yesterday to 52.40, a drop of 17.6 points, while code execution rose from 74.30 to 99.10, a gain of 24.8 points. The main leaderboard rose overall from 72.37 to 78.09.

Score Comparison Facts

The specific changes from yesterday to today are: code execution 74.30→99.10, material constraint 70.00→52.40, engineering judgment 87.50→33.30, task expression 60.40→50.00, main leaderboard 72.37→78.09, with the integrity rating remaining at pass.

Analysis of the Causes of the Fluctuation

The Smoke evaluation contains only 10 questions per day, 2 per dimension, so day-to-day sampling variation is within the normal range. The sharp opposing moves of 24.8 points in code execution and 17.6 points in material constraint most likely stem from the randomness of that day's question sampling rather than a real degradation in model capability. The 54.2-point plunge in engineering judgment further supports sample bias from sampling rather than systematic degradation.

Implications for Users

Teams that prioritize code execution may consider continuing to use Claude Opus 4.7 in the short term, since that dimension has reached 99.10. For scenarios sensitive to material constraints—such as tasks requiring content to be generated strictly from a given document—additional manual verification is needed, because 52.40 is already below yesterday's level.

Strategic Assessment

Engineering judgment falling from 87.50 to 33.30 indicates that the model shows considerable inconsistency on secondary leaderboard dimensions. The rise in the main leaderboard score is driven mainly by the single dimension of code execution, while the simultaneous declines in material constraint and engineering judgment constitute a potential risk signal. The next evaluation round should focus on verifying whether engineering judgment recovers; otherwise, the model's value may be underestimated in tasks requiring comprehensive judgment.

Based on the available score comparisons, the anomaly in Claude Opus 4.7 manifests mainly as a synchronized decline in material constraint and engineering judgment, rather than a collapse in overall capability. When relying on it for multi-step reasoning tasks, developers should add intermediate-result checkpoints.

The simultaneous appearance of a 99.10 code execution score and a 52.40 material constraint score shows that the model is markedly divergent across different capability axes.

For enterprises choosing a model, if the current workflow is mainly code generation, Claude Opus 4.7 remains competitive based on today's performance; if the workflow involves extensive fidelity-to-material requirements, it is better to observe several more days of data before deciding.

The integrity rating remains at pass, indicating no integrity issues in the model's responses and that the fluctuation is concentrated in the capability dimensions themselves.

Taking all comparison data together, the modest rise on the main leaderboard masks significant declines on secondary leaderboard dimensions—a signal worth tracking closely in the next round.


Data source: YZ Index | Run #438 | View raw data

This article is from Winzheng Index blog, translated in full by Winzheng (winzheng.com). Click here to view the original When republishing the translation, please credit the source. Thank you!