Grok 4 Smoke Evaluation Main Score Plunges 17.5 Points, Material Compliance Drops 21.9 in a Single Day

Grok 4's main score in today's Smoke evaluation fell from 94.15 yesterday to 76.65, a decrease of 17.5 points.

Score Breakdown and Direct Comparison

The Code Execution dimension dropped from 100.00 to 86.10, a decrease of 13.9 points; the Material Compliance dimension dropped from 87.00 to 65.10, a decrease of 21.9 points; Engineering Judgment dropped from 88.90 to 75.00; Task Expression dropped from 100.00 to 70.00. The integrity rating remains pass.

Data Fact Decomposition

The Smoke evaluation consists of 10 fixed questions daily, with 2 questions per dimension. The main score is weighted only from Code Execution and Material Compliance. Today, these two dimensions together contributed the entire 17.5-point decline. The Material Compliance dimension saw the largest single-day drop, indicating that the 2 Material Compliance questions drawn today scored significantly lower than yesterday's corresponding questions.

Cause Analysis: Lottery Fluctuation or True Degradation

The randomness of the daily 10-question draw is the primary possible cause. Smoke evaluation questions change daily. Material Compliance questions demand strict citation accuracy and compliance adherence; differences in difficulty or trap distribution between just 2 questions can cause fluctuations of over 20 points. The 13.9-point drop in Code Execution also aligns with small-sample variance characteristics. Current data does not show continuous declines in the same dimension over multiple days, so model parameter-level degradation cannot be confirmed.

Implications for Users

RAG systems and compliance review scenarios that heavily rely on Material Compliance need heightened vigilance. Today's score of 65.10 indicates significant errors in citation fidelity. Developers should add manual verification steps in critical processes. Code execution scenarios are relatively less affected, but the 86.10 score is still below yesterday's perfect score, so automated test scripts should retain fallback mechanisms.

Strategic Judgment

Based on single-day data, this score drop for Grok 4 is more likely due to question draw rather than model capability degradation. It is recommended to compare whether the Material Compliance and Code Execution dimensions recover after the next Smoke evaluation results are released. If Material Compliance remains below 75 points for two consecutive days, then consider actual model stability issues. There is no need to immediately downgrade Grok 4's priority in the selection ranking, but single-day fluctuation risks should be included in the monitoring checklist.

Data source: YZ Index (YZ Index) | Run #235 | View raw data