Grok 4 Code Execution Plunges 20.2 Points, Main Leaderboard Rises 2.4 Points Against Trend

In today's Smoke evaluation, Grok 4's code execution dimension fell from 94.60 to 74.40, a drop of 20.2 points, while material constraints rose from 45.00 to 75.00 and the main leaderboard score rose from 72.28 to 74.67.

Data Fact: Severe Opposite Moves Across Dimensions in a Single Day

Comparing yesterday's and today's scores, code execution fell 20.2 points, material constraints rose 30 points, engineering judgment rose 31.4 points, and task expression rose 6.6 points. The main leaderboard rose only 2.4 points, and the integrity rating remained at pass. The Smoke evaluation has only 10 questions per day, 2 per dimension, so single-day score standard deviation is naturally relatively large.

Cause Analysis: Question-Draw Fluctuation More Likely

The opposite moves of more than 20 points in code execution and material constraints most likely stem from the two code questions drawn that day being significantly harder than yesterday's sample, while the material-constraint questions happened to match the model's strengths. Engineering judgment also rose 31.4 points in sync, further supporting draw randomness rather than overall model degradation. If this were a real capability decline, it would usually be accompanied by simultaneous declines in multiple core dimensions, rather than a slight rise on the main leaderboard.

Code execution 74.40 vs. 94.60 yesterday, material constraints 75.00 vs. 45.00 yesterday—the two dimensions moved in completely opposite directions, pointing to differences in question distribution.

Implications for Users

For developer teams that rely heavily on code execution, today's data suggest that multiple independent validations should be conducted before critical tasks to avoid being misled by a single Smoke result. In scenarios sensitive to material constraints (such as document processing with high long-text faithfulness requirements), Grok 4's 75.00 today actually provides a more reliable reference. When selecting models, enterprises should prioritize the multi-day average of the main leaderboard rather than single-day dimension extremes.

Strategic Judgment: Main Leaderboard Resilience Worth Continued Tracking

The main leaderboard score rose 2.4 points against the trend, indicating that the plunge in code execution was fully offset by gains in material constraints and engineering judgment. Based on current data, Grok 4 shows no signal of systematic degradation, but a single-day fluctuation of 20.2 points has exceeded the normal draw range. It is recommended that the next evaluation check whether the standard deviation of code execution over three consecutive days falls back to within 15 points. If the standard deviation remains high, redundant validation mechanisms should be added in the production environment.

Engineering judgment rose from 30.60 to 62.00, indicating that the day's engineering questions better matched the model's reasoning path. This complements the decline in code execution and further confirms that randomness was dominant. Task expression maintained a moderate rise from 71.30 to 77.90, providing stable support for application scenarios requiring clear output.

Across all dimensions, Grok 4's performance today is closer to normal draw-driven fluctuation rather than capability degradation. Teams that prioritize code execution should focus in the next evaluation round on whether this dimension rebounds into the 90-point range; the slight rise on the main leaderboard provides buffer room for model selection.

Data source: YZ Index | Run #399 | View raw data