Doubao Pro Main Leaderboard Plunges 10.6 Points as Engineering Judgment Falls from 100 to 63.6

In today's Smoke evaluation, Doubao Pro's main leaderboard score fell from 79.18 to 68.54, a decline of 10.6 points.

Data Facts

The Code Execution dimension dropped from 78.50 yesterday to 68.40, down 10.1 points; Material Constraints fell from 80.00 to 68.70, down 11.3 points. Engineering Judgment fell from 100.00 to 63.60, down 36.4 points; Task Expression slipped from 87.90 to 86.70, down 1.2 points. The integrity rating remained pass.

Cause Analysis

The 36.4-point swing in Engineering Judgment most likely stems from the sensitivity of that day's randomly drawn questions to secondary leaderboard dimensions. The Smoke evaluation uses only 10 questions per day, 2 per dimension, so the single-day sample is limited, and secondary dimensions such as Engineering Judgment are more easily influenced by specific questions. The two core main-leaderboard dimensions, Code Execution and Material Constraints, fell in tandem by roughly 10–11 points, indicating that this fluctuation is not a single secondary-dimension anomaly but a problem with overall answer consistency.

The contrast with Task Expression's mere 1.2-point decline suggests that the model's output style on Task Expression is relatively stable, while clear degradation appears in scenarios requiring strict adherence to materials or code execution. The current data cannot distinguish whether this reflects random variation from question drawing or a genuine capability regression, and multi-day data is needed for verification.

Implications for Users

Teams that rely heavily on code execution should add a manual verification step when selecting Doubao Pro, because a Code Execution score of 68.40 means more than 3 out of every 10 questions may contain execution errors. The Material Constraints score of 68.70 likewise signals that in scenarios sensitive to material fidelity, such as contract review and data extraction, the model may omit or rewrite content.

The Engineering Judgment score of 63.60 has a greater impact on developers who rely on the model for architecture decisions or solution evaluation; such tasks are best assigned to models with higher Engineering Judgment scores. Task Expression at 86.70 remains in the usable range, so everyday conversation and content generation scenarios are less affected.

Strategic Assessment

The 10.6-point decline on the main leaderboard was driven jointly by the two core dimensions, Code Execution and Material Constraints, and the sharp 36.4-point swing in Engineering Judgment further amplified the overall drop. The integrity rating remained pass, indicating no obvious hallucination or false-information issues.

The current data supports placing Doubao Pro on a watch list, with close tracking of whether Code Execution and Material Constraints recover in the next period. If both dimensions remain below 70, consideration should be given to reducing its weight in production environments. The Stability dimension score of 31.7 already shows that this model's single-day scores fluctuate considerably, so model selection should rely on multi-day averages rather than single-day results.

Data source: YZ Index | Run #423 | View raw data