Doubao Pro Smoke Evaluation: All Five Dimensions Absent, API Failure Yields Zero Records

In today's Smoke evaluation, Doubao Pro scored "-" on all five dimensions—execution, grounding, judgment, integrity, and communication—as well as on the main leaderboard. The model failed to complete any tasks due to an API failure/timeout and is not ranked in this period.

Data Facts: Zero Records Stem from API Outage

The score comparison table shows "-" for all dimensions on both yesterday and today, with no numerical values generated. This indicates that the evaluation process terminated at the invocation stage, rather than score fluctuations after the model answered 10 questions. The Smoke evaluation covers only 2 questions per dimension per day, making single-day fluctuations normal, but a complete absence of returned results occurs only in cases of API timeout or server-side refusal to respond.

Cause Analysis: Technical Call Failure, Not Model Degradation

Question-draw fluctuations typically manifest as minor score movements within adjacent ranges after responses are completed. This time, however, the simultaneous absence of all five dimensions points to an interruption in the call chain after the request was dispatched. The API failure/timeout record directly indicates a service availability issue, not a decline in the model's ability to understand code execution or material-constraint tasks. The two side-leaderboard dimensions—engineering judgment and task expression—are also missing, further confirming that the interruption occurred before model inference.

The zero records caused by API failure are mechanistically entirely different from the stability decline that results when a model's score standard deviation increases across repeated responses to similar questions.

Implications for Users

Teams that rely heavily on code execution should add extra monitoring of API response success rates when calling Doubao Pro in production environments. Scenarios sensitive to material constraints likewise face the risk of request rejection, which may disrupt workflows. For developers relying on this model for continuous tasks, a single timeout can fail an entire batch, increasing development costs for retry and fallback logic.

• For enterprises conducting model selection: when using Doubao Pro as the primary model, prepare backup API endpoints or local caching strategies.
• For developers relying on this model: the interruption signal from the current Smoke evaluation suggests that routine calls should incorporate timeout retries and health checks.

Strategic Assessment

This incident reflects only an API availability issue and does not constitute evidence of model capability degradation. No scores were obtained on the main leaderboard, so a true comparison of code execution or material-constraint performance is impossible. If similar missing data appears in the next Smoke evaluation, API stability should be the priority for verification; if normal responses resume, this interruption can be regarded as an isolated technical fault, with no need to adjust the assessment of Doubao Pro's core capabilities.

Based on the existing score comparison, Doubao Pro's absence from this period's ranking is directly caused by a call-layer failure and does not involve any change in the model's intrinsic performance. Enterprises and developers should focus on API availability monitoring rather than on score fluctuations of the model itself.

Data source: YZ Index | Run #268 | View raw data