DeepSeek V4 Pro’s main leaderboard score fell from 68.16 to 49.52 points in today’s Smoke evaluation, a drop of 18.6 points.
Score Change Details
The code execution dimension fell from 58.30 to 47.00 points, a drop of 11.3 points. The material constraint dimension fell from 80.20 to 52.60 points, a drop of 27.6 points. Engineering judgment fell from 100.00 to 75.00 points, a drop of 25 points. Task expression rose from 66.70 to 75.00 points, an increase of 8.3 points. The integrity rating remained pass.
Data Fact Breakdown
The material constraint dimension had the largest single-day decline, accounting for the main part of the main leaderboard drop. The code execution dimension also fell by 11.3 points. The two main-leaderboard dimensions fell by a combined 38.9 points, far exceeding the overall main leaderboard decline, indicating that the main leaderboard score is mainly weighted by these two dimensions. The task expression dimension showed a positive change, but its weight is relatively low and cannot offset the overall main leaderboard decline.
Cause Analysis
The sharp decline in the material constraint dimension most likely stems from the questions sampled that day placing higher requirements on consistency between model output and the given material. The simultaneous decline in code execution suggests that the day’s questions may have involved dual constraints of code generation and material citation. Engineering judgment, as a side leaderboard (AI-assisted evaluation), fell by 25 points, further confirming volatility in the model’s performance under multi-constraint scenarios. The slight rise in task expression indicates that the model has not shown systemic problems in task description clarity.
The Smoke evaluation contains only 10 questions per day, so single-day fluctuation is itself within the normal range. However, the 27.6-point drop in the material constraint dimension has exceeded the typical sampling fluctuation range, so it is necessary to distinguish changes in question difficulty from genuine capability degradation. The current data only show a single-day result and cannot directly determine whether model parameters or training have changed.
Implications for Users
Developers working in scenarios with heavy material-fidelity requirements need to add output validation steps in production environments. Teams that rely on both code execution and material constraints should avoid adjusting deployment strategies based on single-day evaluation results. The engineering judgment side leaderboard (AI-assisted evaluation) at 75 points suggests that the model’s judgment consistency under complex constraints has declined, and workflows that rely on this model for multi-step reasoning may face higher retry costs.
Strategic Judgment
The 27.6-point single-day drop in the material constraint dimension is the most direct warning signal, indicating that the model is highly sensitive to constraint-type questions. The main leaderboard score of 49.52 is already below yesterday’s 68.16, and in the short term it is necessary to observe whether the material constraint and code execution dimensions rebound simultaneously in the next Smoke evaluation. If both dimensions remain below the 55-point band, the model’s performance on dual-constraint tasks may have entered a new range.
The slight rise in the task expression dimension cannot change the overall main leaderboard trend, indicating that current problems are concentrated in the two core dimensions of material and code. Based on the existing score comparison, the stability of DeepSeek V4 Pro in material constraint scenarios deserves focused verification in the next period.
Data source: YZ Index | Run #385 | View raw data
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接