Doubao Pro Main Score Plunges 15 Points: Code Execution Drops from 75 to 58.3

Doubao Pro's main score in today's Smoke evaluation dropped from 86.25 to 71.22, with the code execution dimension falling from 75.00 to 58.30 and material adherence dropping from 100.00 to 87.00.

Score Change Data Breakdown

The code execution dimension lost 16.7 points in a single day, material adherence lost 13 points, and together they dragged down the main score by 15.03 points. The engineering judgment side score dropped from 100.00 to 50.00, while the task expression side score rose slightly by 3.3 to 95.00. The integrity rating remained pass.

Cause Analysis of Fluctuation

The Smoke evaluation uses only 2 questions per dimension daily, giving each question a high weight. The code execution dimension fell from 75 to 58.3, most likely due to encountering a more difficult code question or the model outputting errors under specific constraints. The material adherence dropping from a perfect score to 87 indicates at least one question exhibited material deviation. The sharp decline in the engineering judgment side score to 50 also points to randomness from question selection rather than model-level degradation.

The task expression side score rose instead, showing stable performance on expression-type tasks in this drawing. Overall fluctuation fits the statistical characteristics of a quick daily 10-question test, but consecutive core dimensions declining simultaneously exceeds the typical single-question error range.

Specific Implications for Users

Teams heavily reliant on code generation should perform additional verification on Doubao Pro's outputs today. A score of 58.3 implies approximately 41.7% of code execution tasks may contain errors. Scenarios sensitive to material fidelity (e.g., contract extraction, report generation) should also add manual review steps; a score of 87 corresponds to a 13% risk of material deviation.

The engineering judgment side score of 50 indicates unstable performance in scenarios requiring multi-step reasoning and trade-offs. Developers relying on it for architectural or solution judgments should switch to other models or add manual validation.

Strategic Assessment

This 15-point main score drop is primarily driven by question selection volatility and does not yet constitute sufficient evidence of true model degradation. However, the simultaneous decline of the engineering judgment side score to 50 and code execution's 16.7-point drop produces a signal strength exceeding normal daily fluctuation, warranting focused consistency validation in the next Smoke evaluation. If both code execution and material adherence recover to above 80 in the next round, this event was random; if they remain below 70, Doubao Pro's suitability for scenarios with high code and material requirements should be reassessed.

Data source: YZ Index | Run #234 | View raw data