Doubao Pro Code Execution Plummets 26.2 Points, Main Leaderboard Rises Only 0.03 Points

In today’s Smoke evaluation, Doubao Pro’s code execution score fell from 75.00 yesterday to 48.80, a drop of 26.2 points; its material constraint score rose from 67.90 to 100.00, an increase of 32.1 points; and its main leaderboard score rose from 71.81 to 71.84, an increase of only 0.03 points.

Data Facts: Dimension Scores Moved Sharply in Opposite Directions

This Smoke evaluation contained 10 questions, with 2 questions per dimension. The code execution dimension directly determines 50% of the main leaderboard weight, and the material constraint dimension also accounts for 50%. The two core dimensions moved in opposite directions by 26.2 points and 32.1 points, leaving the main leaderboard score almost unchanged. The engineering judgment dimension remained unchanged at 75.00, the task expression dimension fell from 95.00 to 85.00, and the integrity rating remained pass.

Cause Analysis: Question Sampling Volatility Is More Likely

The Smoke evaluation covers the code execution dimension with only 2 questions per day, so each question carries extremely high score weight. The gap between yesterday’s 75.00 and today’s 48.80 most likely stems from the 2 questions selected today placing higher demands on code execution, or from model errors in specific syntax or library calls. The material constraint dimension simultaneously rose to a perfect score, suggesting that today’s questions may have focused more on material faithfulness, a dimension in which Doubao Pro performed steadily. The main leaderboard score rose by only 0.03 points, indicating that the opposite movements in the two dimensions offset each other rather than reflecting a systematic degradation in the model’s overall capability.

If this were genuine model degradation, multiple dimensions would typically decline in sync. In this case, however, the engineering judgment dimension was unchanged, and the task expression dimension fell by only 10 points, a smaller change than in code execution. Based on the available score comparison, random volatility from question sampling is the more reasonable explanation.

Implications for Users

Teams that rely heavily on code execution should add redundant verification steps when choosing Doubao Pro, especially when handling complex algorithms or multi-file projects. The material constraint dimension rose to 100.00, indicating that the model is more reliable in scenarios requiring strict adherence to input materials, such as contract review and code comment generation. The task expression dimension fell to 85.00, so developers with strict output-format requirements should perform additional checks.

Strategic Judgment

Based on today’s score comparison, the main leaderboard remained almost unchanged, indicating no obvious decline in Doubao Pro’s overall capability. The single-day 26.2-point drop in code execution falls within the known normal volatility range of the Smoke evaluation. It is advisable to continue tracking scores in the same dimension in the next period; only if the code execution score remains below 55 for two consecutive days should the possibility of genuine model degradation be considered. At present, there is no need to downgrade the assessment of Doubao Pro.


Data source: YZ Index | Run #435 | View raw data

This article is from Winzheng Index blog, translated in full by Winzheng (winzheng.com). Click here to view the original When republishing the translation, please credit the source. Thank you!