Gemini 2.5 Pro Code Execution Plummets 25 Points, Main Leaderboard Falls 7.2 Points

In today's Smoke evaluation, Gemini 2.5 Pro's code execution score fell from 100.00 to 75.00, and its main leaderboard score dropped from 88.75 to 81.53.

Data Breakdown

The score comparison shows that code execution fell 25 points, while engineering judgment also fell 25 points to 50.00. Task expression rose 8.3 points to 90.00, and material constraints rose 14.5 points to 89.50. The main leaderboard declined by 7.2 points overall. The integrity rating remained pass.

Cause Analysis

The Smoke evaluation contains only 10 questions per day, with 2 per dimension. The sample size is small, so question-draw variance is the primary likely cause. The simultaneous 25-point drops in code execution and engineering judgment suggest that the questions drawn this time may have concentrated on exposing inconsistent handling by the model in complex code paths or under constrained conditions. The higher scores in material constraints and task expression indicate no systematic degradation in the model's text faithfulness or clarity of expression.

The possibility of genuine model regression is low. Single-day data cannot support a sustained-trend conclusion; simultaneous declines in two dimensions are more consistent with small-sample randomness than with changes at the parameter level. As a side-leaderboard dimension, engineering judgment is strongly correlated with code execution, and their same-magnitude movement further points to question characteristics rather than an overall decline in model capability.

Implications for Users

Teams that rely heavily on code execution should add a manual review step when depending on Gemini 2.5 Pro for multi-step debugging or algorithm implementation. This 75.00 score is below yesterday's 100.00 baseline, meaning the probability of errors in similar tasks has increased. Material constraints rose to 89.50, which instead provides more stable output for scenarios requiring strict adherence to input instructions.

When integrating Gemini 2.5 Pro, developers can prioritize assigning code execution tasks to other models, or strengthen constraints in prompts to reduce the impact of variance. An engineering judgment score of 50.00 indicates that the model's ability to assess the pros and cons of technical solutions has become clearly unstable; scenarios that rely on such judgment for architectural decisions should be cautious.

Strategic Judgment

Based on a single-day comparison, Gemini 2.5 Pro's code execution capability was amplified by this round's question draw, and there is insufficient evidence of actual regression. The main leaderboard score of 81.53 remains within a usable range, but if the same dimensions show large fluctuations for two consecutive days, priority verification will be needed. The rises in material constraints and task expression show that the model remains resilient in non-code dimensions.

For the next Smoke evaluation, it is recommended to focus on whether code execution and engineering judgment rebound. If both dimensions remain below 80, then a genuine decline in the model's stability on such tasks should be considered. The current data does not support a conclusion that Gemini 2.5 Pro's overall capability should be downgraded; it only suggests adding redundant verification in code-intensive scenarios.

Data source: YZ Index | Run #318 | View raw data