Gemini 3.1 Pro Material Constraint Score Falls 15.3 Points While Code Execution Rises to 100

In today's Smoke evaluation, Gemini 3.1 Pro's material constraint score fell from 88.10 to 72.80, a drop of 15.3 points, while its code execution score rose from 54.50 to 100.00 and its main leaderboard score increased from 69.62 to 87.76.

Data Facts: Dimension Score Comparison

The code execution dimension reached a perfect 100.00 today, up 45.5 points from yesterday. The material constraint dimension fell to 72.80, down 15.3 points from yesterday. The engineering judgment dimension remained unchanged at 75.00. The task expression dimension rose to 81.70, an increase of 6.7 points. As a result, the main leaderboard score rose 18.1 points to 87.76. The integrity rating remained at pass.

Cause Analysis: Sampling Fluctuation or Model Degradation

The Smoke evaluation includes only 10 questions per day, with 2 questions per dimension, so a single question's score has an outsized effect on the dimension total. The code execution dimension jumping from 54.50 straight to a perfect score suggests that the 2 questions sampled today may have happened to match the model's strengths. The 15.3-point drop in material constraint may be because the questions sampled required strict adherence to material constraints, and the model deviated. Engineering judgment and task expression changed relatively little, indicating that questions in these two dimensions are relatively stable.

The sharp rise in the main leaderboard score was driven mainly by the perfect code execution score. The decline in material constraint did not offset the overall improvement, indicating that this change is more likely random fluctuation caused by question sampling rather than a genuine decline in the model's capabilities. A single day's data is insufficient to prove sustained degradation.

Implications for Users

Teams that focus heavily on code execution can have higher confidence under today's data, as Gemini 3.1 Pro has reached a perfect score in that dimension. Scenarios that depend on material fidelity require additional validation of outputs, as a score of 72.80 means the model may deviate from the provided material on some questions. Engineering judgment remains at 75.00, with limited impact on developers who need engineering decisions. Task expression rose to 81.70, making it suitable for scenarios that require clear task statements.

Enterprises selecting models should consider multi-day data; a single-day decline in material constraint should not be used directly as a reason for exclusion, but teams sensitive to material constraints need to add a manual review step to their production workflows.

Strategic Judgment

Based on today's score comparison, the rise in Gemini 3.1 Pro's main leaderboard score comes mainly from a perfect score in the code execution dimension, and the decline in material constraint is more likely sampling fluctuation. There is no need to immediately conclude that the model has degraded, but if material constraint remains below 80 in the next period, it will be necessary to verify whether a systemic problem has emerged. The current data supports continued observation rather than a downgrade in priority.

Overall, Gemini 3.1 Pro's perfect performance in the code execution dimension offset its losses in material constraint, and its main leaderboard score still achieved a net increase of 18.1 points. Users can decide whether to adjust their usage strategy based on the weighting of their own scenarios.

Data source: YZ Index | Run #382 | View raw data