GLM-4.6 Material Constraint Score Plummets 30.2 Points, Code Execution Rises 22 Points

In today's Smoke evaluation, GLM-4.6's material constraint score fell from 80.20 yesterday to 50.00, a drop of 30.2 points, while code execution rose from 25.00 to 47.00, and the main leaderboard score changed from 49.84 to 48.35.

Direct Data Comparison of Score Changes

The material constraint dimension showed a -30.2 point change, the code execution dimension showed a +22 point change, engineering judgment fell from 75.00 to 0.00, and task expression remained unchanged at 25.00. The main leaderboard is weighted from the code execution and material constraint dimensions, and ultimately dropped only 1.5 points.

Distinguishing Question Sampling Fluctuation from Real Model Degradation

The Smoke evaluation has only 10 questions per day, with 2 questions per dimension, so single-day scores are heavily affected by the specific questions drawn. The drop in material constraint from 80.20 to 50.00 and the zeroing out of engineering judgment may stem from the day's material fidelity questions being concentrated in difficulty or having stricter answer format requirements. The 22-point rise in code execution shows that the model performed better on the code questions drawn that day. The opposite changes in the two dimensions point to question sampling fluctuation rather than overall capability degradation.

If this were real model degradation, multiple dimensions would usually decline simultaneously, but this time the task expression dimension was unchanged and code execution actually improved, further supporting the sampling fluctuation explanation. The integrity rating remains pass, with no new issues.

Specific Implications for Users

For teams that rely on material constraints, today's 50.00 means an increased probability of model errors in scenarios that require strictly following input materials to generate answers. Developers focused on code execution can gain some confidence from the 47.00 score, but should note that single-day data is unstable.

The zeroing out of engineering judgment has the greatest impact on scenarios requiring engineering decision support; users should add manual review steps to critical processes.

Strategic Assessment

Based on the current score comparison, the extreme fluctuations in material constraint and engineering judgment are more likely caused by question sampling rather than systematic degradation of model capability. The fact that the main leaderboard dropped only 1.5 points also confirms the offsetting effect of the two core dimensions.

It is recommended that the next Smoke evaluation continue to observe whether material constraint rebounds to the 70-plus range. If material constraint remains around 50 for two consecutive days, then consider the possibility of real model degradation. At present, there is no need to downgrade the overall capability assessment of GLM-4.6.

Data source: YZ Index | Run #385 | View raw data