Grok 4 Main Leaderboard Plummets 17.2 Points, Material Constraints Crashes 24.8 Points in a Single Day

In today’s Smoke evaluation, Grok 4’s main leaderboard score fell from 83.51 to 66.35, a drop of 17.2 points.

Core Data Comparison

The Code Execution dimension fell from 88.50 yesterday to 77.60, a drop of 10.9 points; the Material Constraints dimension fell from 77.40 to 52.60, a drop of 24.8 points. Engineering Judgment rose from 42.80 to 56.00, and Task Expression rose from 93.80 to 95.00. The integrity rating changed from pass to warn.

Drivers Breakdown

The main leaderboard is weighted solely by Code Execution and Material Constraints. The 24.8-point single-day decline in Material Constraints directly pulled the main leaderboard down by about 12.4 points, while the Code Execution decline contributed about 5.45 points; together, the two explain most of the 17.2-point drop. The modest rebounds in Engineering Judgment and Task Expression failed to offset the main leaderboard losses.

The decline most likely stems from question-sampling volatility. The Smoke evaluation includes only 10 questions per day, and the Material Constraints dimension includes only 2 questions, so a single question error can cause a fluctuation of more than 20 points. Genuine model degradation usually comes with simultaneous declines across multiple dimensions, whereas this time Engineering Judgment and Task Expression actually rose, pointing more strongly to changes in question difficulty or topic distribution.

Specific Impact on Users

Scenarios that rely heavily on Material Constraints—such as faithful rewriting of long documents, citation checking, and strictly formatted output—will face a higher risk of failure. Stability in Code Execution scenarios has also declined, and teams with heavy code execution needs should add manual review steps. Engineering Judgment and Task Expression remain at mid-to-high levels, so the impact is smaller for developers whose tasks require creative expression or judgment.

Strategic Assessment

The integrity rating changing to warn is the only signal in this evaluation that requires immediate attention. Such rating changes are usually related to consistency or boundary behavior, and it is recommended that the next Smoke evaluation focus on tracking whether this dimension persists. Under the single-day sampling framework, the 17.2-point main leaderboard decline remains within an explicable range, but the absolute Material Constraints score of 52.60 has entered a low range and is worth continued observation in the next period to confirm whether it is accidental fluctuation.

Overall, Grok 4’s performance this time is closer to sampling volatility than systematic degradation, but the integrity rating change and the low absolute Material Constraints score constitute a double warning. Users should pause large-scale deployment in scenarios sensitive to material faithfulness.


Data source: YZ Index | Run #439 | View raw data

This article is from Winzheng Index blog, translated in full by Winzheng (winzheng.com). Click here to view the original When republishing the translation, please credit the source. Thank you!