Gemini 3.1 Pro's Code Execution Plunges 27.2 Points, Main Leaderboard Slips 6.5

In today's Smoke evaluation, Gemini 3.1 Pro's code execution score dropped directly from yesterday's 91.70 to 64.50, a decline of 27.2 points, while the main leaderboard as a whole fell back from 82.39 to 75.89.

Raw Data Behind the Score Change

A day-over-day comparison shows the largest swing in the code execution dimension, while the materials constraint dimension rose 18.8 points from 71.00 to 89.80. The engineering judgment dimension held flat at 100.00, and the task expression dimension edged up 8.3 points to 60.00. The integrity rating remains a pass.

Analyzing the Cause of the Fluctuation

The Smoke evaluation contains only 10 questions per day, with two questions per dimension, so day-to-day differences in the questions drawn can easily cause large score swings. The sharp drop in the code execution dimension most likely stems from the two questions drawn today placing higher demands on code correctness or execution efficiency, tasks on which the model failed to deliver the expected output. The simultaneous surge in the materials constraint dimension shows the model was more faithful to the given materials on today's questions. These opposing movements in the two dimensions point to randomness from question sampling rather than a decline in the model's overall capability.

Code execution 91.70 → 64.50, materials constraint 71.00 → 89.80 — this kind of offsetting movement is common in small-sample rapid evaluations.

If a model were genuinely regressing, multiple dimensions would typically decline in tandem; this time engineering judgment held at a perfect score and task expression actually rose, which favors the question-variation explanation.

What This Means for Users

Teams that rely heavily on code generation and debugging should note that Gemini 3.1 Pro can swing by more than 27 points in a single day on the code execution dimension, meaning the same kind of programming task may produce noticeably different results on different days. The rise in the materials constraint dimension indicates the model still performs strongly in scenarios that require strict adherence to input documents or instructions, making it suited to tasks such as document summarization and compliance checks.

For developers, the current data suggests not adjusting model choices in production environments based on a single Smoke result; instead, watch the score distribution for the same type of task across several consecutive days to judge whether the movement is incidental.

Strategic Assessment

Based on this score comparison, the high volatility in Gemini 3.1 Pro's code execution dimension is worth tracking in the next round, but the available evidence is not enough to conclude that the model's capability has declined systematically. The marked improvement in the materials constraint dimension may be masking the main leaderboard's pullback, so the next evaluation round should focus on the difficulty distribution of the code execution questions. Only if code execution scores fall below 70 on two consecutive days should deeper stability testing be considered.

The engineering judgment dimension remains at a perfect score, showing the model still maintains a high standard on engineering decision-making problems and that this capability was unaffected by the current fluctuation. Overall, the main leaderboard's 6.5-point decline was driven primarily by the single code execution dimension, with no simultaneous deterioration in the model's other core capabilities.

Data source: YZ Index | Run #378 | View raw data