Doubao Pro Main Leaderboard Plunges 8 Points; Material Constraint Falls 25.8 Points in One Day

In today's Smoke evaluation, Doubao Pro's main leaderboard score fell from 87.60 yesterday to 79.57, an 8-point decline. The core reason was a sharp drop in the material constraint dimension from 84.30 to 58.50.

Breakdown of the Data

The code execution dimension rose from 90.30 to 96.80, engineering judgment fell from 69.50 to 63.60, task expression remained unchanged at 95.00, and the integrity rating remained at pass. The main leaderboard is weighted only by code execution and material constraint, so the 25.8-point drop in material constraint directly pulled the overall score down by 8 points.

Cause Analysis: Sampling Volatility or Real Degradation

The Smoke evaluation uses only 2 questions per dimension each day, so each question carries very high score weight. If both material constraint questions happened to require strict source citation or refusal of hallucinations, a single large-scale deviation by the model could cost more than 20 points. Code execution rose by 6.5 points in the opposite direction, indicating no systematic problem in the model's code path. Engineering judgment declined slightly by 5.9 points, far less than material constraint, so the most likely cause is that the question draw in the material constraint dimension concentrated exposure of inconsistency in the model's faithfulness control, rather than an overall capability regression.

The gap between the material constraint score of 58.50 and yesterday's 84.30 exceeds the normal range of single-day standard deviation, suggesting that consistency in this dimension is currently low.

Specific Implications for Users

Scenarios that rely heavily on material constraint—such as legal contract review, financial statement reconciliation, and technical document alignment—need a higher proportion of human review. The code execution score of 96.80 still has reference value for purely programming-assistance tasks, but when a task also requires "answering based on provided materials," the material constraint performance of 58.50 significantly increases the risk of output deviation. When developers call Doubao Pro in RAG workflows, it is advisable to add an additional consistency-check layer.

Strategic Judgment

Based on the current score comparison, the sharp single-day fluctuation in Doubao Pro's material constraint dimension deserves continued tracking in the next Smoke evaluation. If material constraint remains below 65 points for two consecutive days, it can be preliminarily judged that this dimension has a stability problem; if it rebounds above 80 points the next day, this decline is more likely attributable to question sampling. The current data do not support the conclusion of "overall model degradation," but they do support the observation of "insufficient material constraint consistency."

Changes in the two side-leaderboard dimensions, engineering judgment and task expression, were both smaller than those in the main leaderboard's core dimensions, confirming that the main leaderboard decline was driven mainly by material constraint. In enterprise model selection, if material faithfulness is a core requirement, it is advisable to wait for at least 3 days of Smoke data before deciding; if the main use is code generation, the current score of 96.80 remains in a usable range.

Data source: YZ Index | Run #415 | View raw data