Gemini 3.1 Pro Material Constraint Drops 26.6 Points, Main Ranking Still Up 5.4 Points

Gemini 3.1 Pro saw its material constraint score drop from 90.90 to 64.30 in today's Smoke evaluation, a decrease of 26.6 points, while its task expression score also fell from 85.00 to 66.70.

Key Data

Code execution score rose from 18.40 to 50.00, engineering judgment remained unchanged at 75.00, the main ranking overall increased from 51.03 to 56.44, and the integrity rating remained pass.

Cause Analysis

Smoke evaluations only include 2 questions per dimension per day, with an extremely small sample size. Changes in individual question scores can cause significant fluctuations in dimension scores. The simultaneous decline in material constraint and task expression suggests that the two material fidelity questions drawn today may have required stricter boundary adherence from the model, which the model failed to fully satisfy. The sharp recovery in code execution indicates that the other two code questions were either easier or the model performed consistently on that task type. Combined, the main ranking still achieved a net increase.

Implications for Users

For RAG scenarios heavily reliant on material citation and long-document rewriting teams, today's results show that Gemini 3.1 Pro may exhibit noticeable output deviation under small sample conditions, necessitating additional manual verification steps in critical workflows. For developers primarily focused on code generation, today's data provides a positive signal.

Strategic Assessment

The steep single-day drop in material constraint is more likely due to question draw fluctuations rather than genuine model degradation. The fact that the main ranking still rose indicates that the gain from code execution currently outweighs the loss from material constraint. It is recommended to continue tracking whether material constraint rebounds above 80 points in the next Smoke evaluation; if it remains below 70 points for two consecutive days, consider model consistency issues.

Data Source: YZ Index | Run #237 | View Raw Data