GLM-4.6 Smoke Evaluation: Material Constraint 51.80, Task Expression 50.00; API Failure Leaves Two Dimensions Missing

In today's Smoke evaluation, GLM-4.6 experienced missing data in the execution and judgment dimensions due to an API failure/timeout. An automatic re-run has been initiated, and this round will not be included in the main leaderboard ranking. The material constraint dimension scored 51.80, the task expression dimension scored 50.00, and the remaining dimensions were recorded as empty.

Data Facts: Score Comparison and Missing Status

The day-over-day comparison shows that the code execution dimension went from no record to no record, the material constraint dimension went from no record to 51.80, the engineering judgment dimension went from no record to no record, the task expression dimension went from no record to 50.00, and the main leaderboard went from no record to no record. The Smoke evaluation uses only 10 questions per day, with 2 questions mapped to each dimension, so single-day score fluctuations are within the normal range.

Cause Analysis: API Failure, Not Model Degradation

The complete absence of both the execution and judgment dimensions in the score comparison points directly to API call timeouts or failures, rather than real changes in the model's code execution or engineering judgment capabilities. The two recorded dimensions—material constraint at 51.80 and task expression at 50.00—each come from a limited sample of 2 questions. With only 10 questions in total, the sample size is too small and easily affected by question randomness. There is no evidence of systematic degradation in the model's material fidelity or task expression; the missing data is itself a technical recording interruption.

GLM-4.6's Smoke evaluation today: 51.80 in material constraint and 50.00 in task expression; the remaining dimensions are missing due to the API failure.

Implications for Users: Impact on Model Selection and Dependent Scenarios

For teams that rely heavily on code execution, the missing execution dimension means that GLM-4.6's usability in that scenario cannot be assessed from today's Smoke data, and the re-run results are needed. The material constraint score of 51.80 provides a reference for enterprises that depend on the model strictly adhering to input constraints: if the current application is sensitive to output boundaries, this score suggests that additional manual verification is required. Meanwhile, the task expression score of 50.00 indicates that the model's performance in task instruction parsing sits in the lower-middle range, and developers should reserve fault-tolerance mechanisms when building multi-step instruction pipelines.

• Developers in material-constraint-heavy scenarios should first review the complete execution score after the re-run before deciding whether to upgrade the production environment.
• Automated workflows that rely on task expression should add manual review checkpoints on top of today's data.

Strategic Assessment: Focus on Completeness, Not Single-Day Scores

Based on the score comparison, the anomaly in this round of GLM-4.6 is primarily caused by the API failure, not model capability degradation. Given the inherent volatility of the single-day 10-question quick test, the two isolated scores of 51.80 and 50.00 are insufficient to support a conclusion of genuine performance decline. The signal that warrants attention is data completeness itself: consecutive API timeouts will directly affect the comparability of future Smoke evaluations. It is recommended that the next round focus on verifying whether the re-run execution and judgment scores return to the historical range, to confirm that this missing data was an isolated incident.

For teams currently evaluating options, the available data does not support removing GLM-4.6 from the candidate list, but cross-validation of multiple days of complete Smoke records is required before deployment. Developers relying on this model should treat today's missing data as an operational signal, prioritizing API availability monitoring over immediately adjusting prompt strategies.

Daily fluctuations in Smoke evaluations are normal; the core issue in this anomaly is missing dimensions rather than score levels. Based on the existing comparison, GLM-4.6 should re-enter ranking observation after the re-run to confirm whether the material constraint score of 51.80 and the task expression score of 50.00 represent stable levels.

Data source: YZ Index | Run #260 | View raw data