Grok 4 Material Constraint Score Plunges 17.4 Points; Code Execution Rises 7.9; Smoke Main Leaderboard Down 3.5

In today's Smoke evaluation, Grok 4's material constraint score fell from 75.00 to 57.60, a drop of 17.4 points, while code execution rose from 74.40 to 82.30, and the overall main leaderboard fell from 74.67 to 71.19.

Score Comparison and Key Facts

The specific changes from yesterday to today are as follows: code execution 74.40 → 82.30 (+7.9), material constraints 75.00 → 57.60 (-17.4), engineering judgment 62.00 → 55.60 (-6.4), task articulation 77.90 → 85.00 (+7.1), main leaderboard 74.67 → 71.19 (-3.5). The integrity rating remains pass.

Possible Causes of the Sharp Drop in Material Constraints

The Smoke evaluation has only 10 questions per day, 2 per dimension, so single-day score fluctuations are within the normal range. The material constraints dimension mainly assesses the model's fidelity to the given material and compliance with constraints. Today's 57.60 versus yesterday's 75.00, a drop of 17.4 points, most likely stems from a higher proportion of strict material-constraint question types in the sampled questions. Code execution rose by 7.9 points at the same time, indicating that the model's performance on code-related tasks did not degrade in tandem; therefore, this decline in material constraints is closer to question-sampling fluctuation than to a real degradation in model capability.

Specific Implications for Users

For enterprise scenarios that rely heavily on material fidelity, such as contract review, policy interpretation, and internal document Q&A, Grok 4's performance today suggests the need to add a manual review step. The code execution score rose to 82.30; for developer teams focused mainly on code generation and debugging, today's data instead indicates improved usability. Engineering judgment fell to 55.60; for scenarios requiring complex decision chains, it is advisable to temporarily reduce its weight in reliance.

Strategic Judgment

Based on single-day data, the 3.5-point decline in the main leaderboard was mainly dragged down by material constraints. Material constraints and code execution show clear opposite fluctuations, indicating that the model's stability differs across dimensions. The stability dimension measures the standard deviation of scores across multiple answers to the same type of question; today's large fluctuation in material constraints may push up the standard deviation for this dimension and lower the overall consistency score. Whether continuous attention is needed depends on whether material constraints can rebound to the 70-plus range in the next evaluation. If material constraints remain below 60 for two consecutive days, that can be judged as a signal requiring close tracking.

This evaluation once again confirms that the Smoke test is suitable for capturing short-term anomalies, but it should not be directly equated with long-term capability ranking. Grok 4 rose to 85.00 in the task articulation dimension, showing that it still has an advantage in task decomposition and clarity of expression. When enterprises select models, they can use the material constraints score of 57.60 as an immediate reference value for material-heavy scenarios, rather than a final conclusion.

Data: YZ Index | Run #401 | View raw data