Gemini 2.5 Pro Main Leaderboard Jumps from 18.37 to 83.39, Material Constraint Leaps from 12.1 to 100

In today's Smoke evaluation, Gemini 2.5 Pro's main leaderboard score rose from 18.37 points yesterday to 83.39 points, the code execution dimension rose from 23.50 to 69.80 points, the material constraint dimension rose from 12.10 to 100.00 points, and the integrity rating simultaneously switched from fail to pass.

Data Facts: Single-Day Dimension Score Changes

The code execution dimension scored 23.50 yesterday and 69.80 today, the material constraint dimension scored 12.10 yesterday and 100.00 today, engineering judgment (side leaderboard, AI-assisted evaluation) rose from 24.40 to 88.90 points, and task expression (side leaderboard, AI-assisted evaluation) rose from 50.00 to 65.00 points. The main leaderboard is composed solely of the weighted combination of the two auditable dimensions—code execution and material constraint—so today's 83.39 points directly reflects the joint improvement in both dimensions.

Cause Analysis: Question Draw Fluctuation Is the Most Likely Explanation

The Smoke evaluation uses only 10 questions per day (2 questions per dimension), making the sample size extremely small. The material constraint dimension jumping from 12.10 to a perfect score means that both material constraint questions drawn today were handled correctly, while the questions drawn yesterday may have included stricter constraint conditions or boundary cases more likely to trigger violations. The code execution dimension also saw a 46.3-point increase, which is consistent with the drastic score swings caused by the random distribution of question difficulty in a small sample. Existing data shows no change in model parameters or training, so there is no basis for attributing this improvement to a genuine degradation or advancement in the model's capabilities.

Today's material constraint dimension reached 100.00 points, while yesterday it was only 12.10 points—a difference of 87.9 points, far exceeding the normal range that could be explained by the standard deviation of single-question scores.

Implications for Users

Teams that prioritize code execution should note when making selection decisions that Gemini 2.5 Pro's code execution score in the Smoke evaluation remains at 69.80 points, still a clear gap from a perfect score. For scenarios that depend on material fidelity, it can be observed that the model reached 100.00 points in the material constraint dimension today, but scored only 12.10 in the same dimension yesterday, indicating that a single test result is insufficient as a basis for long-term selection decisions.

The integrity rating switching from fail to pass means that today's responses passed the integrity threshold, but the threshold itself is only an entry condition and does not represent consistent output quality across all scenarios. Developers planning to deploy Gemini 2.5 Pro to production environments should require integrity ratings of pass across at least 3 consecutive independent Smoke evaluation cycles to reduce the risk of misjudgment caused by single-draw fluctuation.

Strategic Assessment

Based on the existing score comparison, Gemini 2.5 Pro's main leaderboard improvement this time is most likely driven by question draw fluctuation rather than a systemic change in model capabilities. The simultaneous rise in both side leaderboard dimensions—engineering judgment and task expression—further confirms the randomness of overall test question difficulty distribution. It is recommended that the next cycle focus on verifying whether Gemini 2.5 Pro can maintain a main leaderboard score above 80 points when the difficulty of material constraint and code execution questions remains at similar levels. If similar large fluctuations appear in two consecutive cycles, the model's measurement stability should be added to the key watchlist.

The current data does not support judging Gemini 2.5 Pro as having "solved the material constraint problem," nor does it support determining that its code execution capabilities have substantially regressed. The only conclusion that can be confirmed is that the Smoke evaluation's single-day sample size is too small, causing a single model's score to fluctuate by more than 60 points on the main leaderboard between different days—and this fluctuation itself is worth continued tracking.

Data Source: YZ Index | Run #289 | View Raw Data