Claude Opus 4.7 Code Execution Drops 19.5 Points as Material Constraints Rise 19.3 Points

In today's Smoke evaluation, Claude Opus 4.7's code execution dimension fell from 94.50 to 75.00, a drop of 19.5 points; material constraints rose from 70.00 to 89.30, an increase of 19.3 points; and the main leaderboard total score fell from 83.48 to 81.44, down only 2 points.

Data Facts and Dimension Breakdown

The Smoke evaluation runs only 10 questions per day, with 2 questions per dimension. Code execution and material constraints make up the main leaderboard, while engineering judgment and task expression form the secondary leaderboard. Today, code execution at 75.00 and material constraints at 89.30 jointly determine the main leaderboard score of 81.44. Engineering judgment rose from 55.60 to 75.00, task expression rose from 37.50 to 50.00, and the integrity rating remained pass.

Cause Analysis: Sampling Fluctuation or Model Degradation

With a single-day sample of 10 questions, randomness in question sampling is the primary factor. The near-20-point moves in opposite directions for code execution and material constraints at the same time indicate that the model's performance differences across question types were amplified. If this were real degradation, it would usually be accompanied by synchronized declines across multiple dimensions; today, however, both secondary-leaderboard items rose, pointing to question distribution rather than an overall decline in capability.

The 19.3-point rise in the material constraints score may stem from today's sample containing questions with higher requirements for citation fidelity; the sharp drop in code execution may be due to sampled coding questions that require complex multi-step reasoning or edge cases. The two scores offset each other in opposite directions, leaving the main leaderboard down only slightly by 2 points.

Specific Implications for Users

Teams that rely heavily on code generation should note that a single-day code execution score of 75.00 means that in complex debugging or multi-file refactoring scenarios, the model's output stability may be lower than the previous day's 94.50 level. A material constraints score of 89.30, meanwhile, shows that in RAG scenarios requiring strict adherence to reference documents, today's performance was better than yesterday's.

The rise in the two secondary-leaderboard items, engineering judgment and task expression, means that for developers who need the model to make architecture decisions or break down requirements, today's usability improved somewhat. However, the main leaderboard fluctuation is a reminder that when selecting a model, one should not look only at single-day scores and should observe multi-day consistency.

Strategic Assessment

Based on today's data, the change in Claude Opus 4.7's main leaderboard score was mainly caused by random offsetting between code execution and material constraints, and there is currently no evidence to support real model degradation. The daily Smoke evaluation's design of 2 questions per dimension naturally amplifies fluctuations, and a single-day drop of 19.5 points is within the normal range.

If code execution remains below 80 in the next run and material constraints fall back, it will be necessary to focus on verifying the model's sensitivity to question types in coding tasks. Current data show only sampling fluctuation and do not yet constitute a signal for sustained attention.

Data: YZ Index | Run #427 | View raw data