Doubao Pro Main Leaderboard Plunges 8 Points: Code Execution 94.5→45.8, Material Constraints 47.6→89.3

Doubao Pro's main leaderboard score fell from 73.40 yesterday to 65.38 in today's Smoke evaluation, a drop of 8.02 points. The code execution dimension fell from 94.50 to 45.80, a decline of 48.70 points; material constraints rose from 47.60 to 89.30, an increase of 41.70 points; engineering judgment climbed from 75.00 to 95.50, up 20.50 points; task expression held steady at 100.00; the integrity rating remained pass.

Breaking Down the Data

The main leaderboard is a weighted combination of the code execution and material constraints dimensions. Today's code execution score directly dragged down the main leaderboard, while the simultaneous rise in material constraints only partially offset the decline. Engineering judgment and task expression are not counted toward the main leaderboard, but their changes show the model's performance diverging across secondary dimensions. All figures come from the same day's 10-question quick test (2 questions per dimension).

Causes: Sampling Noise or Genuine Degradation

The Smoke evaluation uses only 2 questions per dimension each day, an extremely small sample in which each individual question carries high weight. The most likely reason code execution fell from 94.50 to 45.80 is that the two code questions drawn today happened to hit the model's weak spots, causing severe point loss on individual questions. Material constraints rose 41.70 points over the same period, indicating that the model actually performed better on faithfulness to the provided materials within the same question set. The 20.50-point gain in engineering judgment likewise points to randomness from question sampling rather than an overall degradation in model capability. Task expression has been perfect for two consecutive days, further confirming that the fluctuation is concentrated in specific dimensions.

Genuine degradation would typically come with simultaneous declines across multiple dimensions or a change in the integrity rating, but today's integrity rating remains pass, and both material constraints and engineering judgment improved markedly. The current data therefore better support the question-sampling fluctuation explanation.

What This Means for Users

Teams that rely heavily on code execution should note: today's 45.80 shows that Doubao Pro can fail badly on particular programming tasks. We recommend adding a local code testing step before deploying to production, rather than trusting a single output directly. A material constraints score of 89.30 indicates the model is more stable in scenarios requiring strict adherence to input documents (such as contract review or report generation), making it a good first choice for such tasks.

An engineering judgment score of 95.50 is friendly to developers who need model assistance with architecture decisions, but because this is a secondary dimension, human review is still required in practice. A perfect task expression score means lower risk in everyday conversation and requirement-statement tasks.

Strategic Assessment

A single-day 8-point swing on the main leaderboard is within the normal range for a small-sample quick test, but the extreme 48.70-point drop in code execution warrants continued verification next period. If the code execution score rebounds above 80 tomorrow, it can essentially be judged as sampling chance; if it stays below 60 for two consecutive days, potential problems with the model's code-related training data should be considered. The opposite movements of material constraints and code execution also suggest that future evaluations could examine the correlation in question difficulty between the two dimensions.

Based on the available data, Doubao Pro's short-term pressure on the main leaderboard stems mainly from an unfavorable draw on code execution rather than an overall capability decline. Enterprises making model selections should add redundant verification in code-heavy scenarios and may moderately raise usage priority in material-faithfulness scenarios.

Data source: YZ Index | Run #427 | View raw data