Qwen3 Max Material Constraint Drops 20 Points to 47.70, Code Execution Surges 37.8 Points, Main Leaderboard Rises 11.8 Points

Qwen3 Max in today's Smoke evaluation saw its material constraint score drop from 67.70 yesterday to 47.70, a decline of 20 points, while code execution rose from 54.70 to 92.50, an increase of 37.8 points. The overall main leaderboard rose from 60.55 to 72.34.

Main Leaderboard Score Breakdown and Direct Comparison

The main leaderboard is solely weighted by two dimensions: code execution and material constraint. Today's code execution score of 92.50 compared to yesterday's 54.70, and material constraint of 47.70 compared to yesterday's 67.70. After weighting, the main leaderboard net increased by 11.8 points. Engineering judgment dropped from 72.20 to 55.60, while task expression rose from 31.70 to 66.70—both are side leaderboard dimensions and not included in the main leaderboard.

Possible Mechanism Behind the 20-Point Drop in Material Constraint

Smoke evaluation covers the material constraint dimension with only 2 questions per day. Score fluctuations from a single question can directly lead to dimensional differences of over 20 points. Today's material constraint score of 47.70 corresponds to an average accuracy of approximately 47.7% on the two questions, while yesterday's 67.70 corresponds to approximately 67.7%. The randomness of question sampling is the primary explanation: if today's two questions both involve strict citation of the original text or multi-paragraph consistency requirements, model output deviations will magnify the scoring deduction. The simultaneous 37.8-point surge in code execution also points to changes in question difficulty distribution rather than an overall model capability shift.

The simultaneous occurrence of a 92.50 code execution score and a 47.70 material constraint score indicates that the model's performance differences across task types were amplified by the day's questions.

Scenario Implications for Users

For scenarios heavily reliant on material fidelity—such as teams working on legal contract extraction, academic citation verification, or long-document fact checking—today's 47.70 score suggests the need to increase manual review steps. Developers focused on code generation and debugging can continue using outputs at the 92.50-score level, but still need to verify material citation parts. The engineering judgment side leaderboard score of 55.60 reduces its value for system architecture decisions, while the task expression score of 66.70 provides some assistance for prompt iteration.

Differentiating Stability from Genuine Degradation

The stability dimension measures the standard deviation across multiple similar questions. Today's single-day data cannot be used to calculate standard deviation. The coexistence of a -20 point drop in material constraint and a +37.8 point gain in code execution is more consistent with sampling fluctuation than with model parameter-level degradation. The integrity rating remains pass, with no issues of inconsistency or fact fabrication.

Strategic Assessment

Based on the current score comparison, the rise in Qwen3 Max's main leaderboard is primarily driven by the single dimension of code execution. The 20-point drop in material constraint, under a 2-question sample, does not constitute a signal of sustained degradation. If the next Smoke evaluation shows material constraint still below 55, it will require focused verification; if it rebounds above 65, today's result will be confirmed as a sampling anomaly. The current data does not support a conclusion of overall model capability downgrade.

When selecting models for enterprise use, it is necessary to observe the standard deviation of material constraint scores over three consecutive days. If the standard deviation exceeds 15 points, it indicates insufficient consistency in the model's material constraint dimension, making it suitable only as a code assistance tool rather than a core knowledge base. Developers can design secondary verification workflows targeting material constraint weaknesses to mitigate potential risks at the 47.70-score level.

Data source: YZ Index | Run #255 | View raw data