Grok 4 Code Execution Plunges 15.8 Points, Main Leaderboard Drops Only 1.1

In today's Smoke evaluation, Grok 4's code execution score fell from 94.00 to 78.20, a drop of 15.8 points; material constraints rose from 71.40 to 88.30, an increase of 16.9 points; and the main leaderboard score fell from 83.83 to 82.75, a drop of only 1.1 points.

Data Facts: Divergent Trends on the Main and Side Leaderboards

This Smoke evaluation included only 10 questions, with code execution and material constraints as auditable main-leaderboard dimensions. Code execution lost 15.8 points in a single day, while material constraints gained 16.9 points over the same period; after offsetting, the main leaderboard showed a net loss of 1.1 points. Engineering judgment fell from 75.00 to 22.20, and task expression fell from 100.00 to 70.40; both side-leaderboard dimensions dropped by more than 29 points. The integrity rating remained pass and did not trigger the threshold.

Cause Analysis: Question-Sampling Volatility or Genuine Model Degradation

The Smoke evaluation includes only 2 questions per dimension each day, so the sample size is very small and the single-day score standard deviation is naturally high. Code execution and material constraints moved in almost mirror-opposite directions, suggesting that this fluctuation most likely stems from random question sampling rather than a systemic degradation in the model's capability structure. Genuine degradation would usually be accompanied by same-direction declines across multiple main-leaderboard dimensions, rather than a complementary pattern of one rising and one falling.

The simultaneous sharp declines in the engineering judgment and task expression side leaderboards likewise point to a concentrated shift in question difficulty or task-type distribution. Side-leaderboard questions may include more tasks requiring long-chain reasoning or strict formatting, while the code execution questions drawn that day happened to expose the model's execution instability under specific constraints.

Implications for Users

Developer teams that rely heavily on code execution should note that the single-day score of 78.20 is already below the previous day's level of 94.00; in scenarios requiring highly consistent code generation, they should add a manual review step. RAG or document-processing scenarios that are sensitive to material constraints received a positive signal: 88.30 indicates that the model's fidelity to input materials improved that day.

The sharp fluctuations in the engineering judgment and task expression side leaderboards pose a potential risk to automated workflows that require stable output formats or engineering decisions. Developers using Grok 4 for multiple iterations should retain manual verification checkpoints on critical paths to prevent side-leaderboard volatility from directly propagating to final deliverables.

Strategic Judgment: Is Continued Attention Warranted?

The main leaderboard fell by only 1.1 points this time, and core capabilities did not show systemic collapse. The opposite movements in code execution and material constraints are more consistent with small-sample sampling volatility than with a substantive change in model weights or alignment strategy. It is advisable to focus the next Smoke evaluation on whether code execution rebounds into the 90-point range; if it remains below 80 for two consecutive periods, then consider launching a larger-sample targeted validation.

Current data do not support the conclusion that Grok 4's overall capability should be downgraded. Selection-oriented enterprises can continue to treat it as a candidate model, but should set up redundant validation mechanisms in code-execution-intensive scenarios. The stability dimension did not provide data in this period, so long-term consistency cannot be judged on that basis.

Data: YZ Index | Run #405 | View raw data