DeepSeek V4 Pro Code Execution Plunges 41.7 Points; Main Leaderboard Down 19.2 in a Day

DeepSeek V4 Pro's main leaderboard score in today's Smoke evaluation fell from 70.44 to 51.24, a decline of 19.2 points, driven primarily by the code execution dimension dropping from 66.70 to 25.00.

Data Breakdown: Single Dimension Dominates the Decline

The code execution dimension lost 41.7 points in a single day, while the material constraint dimension rose from 75.00 to 83.30, an increase of 8.3 points. Engineering judgment fell from 75.00 to 50.00, and task expression dropped from 66.70 to 58.30. The main leaderboard is weighted solely by code execution and material constraint, so the sharp decline in code execution directly dragged down the overall ranking.

Cause Analysis: Sampling Fluctuation or Model Degradation

The Smoke evaluation includes only 2 questions per dimension per day, making the sample size extremely small; a single question failure can cause 30–40 point swings. Today's 25.00 in code execution means both questions failed to pass, while yesterday's 66.70 corresponds to at least one question passing. The material constraint dimension improved in the opposite direction, indicating that the model performed consistently on constraint-following questions without signs of systematic degradation. The declines in engineering judgment and task expression were both smaller than that of code execution, pointing to issues concentrated in code generation and debugging.

Code execution 66.70→25.00, material constraint 75.00→83.30, showing that today's point losses are concentrated in execution correctness, not material citation accuracy.

Specific Implications for Users

Teams that heavily rely on code execution should immediately retest the same or similar questions locally. A 25.00 score in the Smoke evaluation corresponds to failing both questions; if the model is used for automated script generation or algorithm verification, the error rate may rise significantly. RAG scenarios sensitive to material constraints are less affected, as that dimension actually improved today.

• Code generation tasks: manual review steps are recommended
• Long-chain tool calls: prioritize switching to models that performed more stably yesterday
• Scenarios requiring high material fidelity: can continue using without urgent switching

Strategic Assessment

A single-day 41.7-point drop in code execution is an extreme case under a testing framework of only two questions per day, but at least three consecutive days of data are needed to distinguish sampling fluctuation from genuine capability changes. Based solely on today's data, it cannot be concluded that the model has entered a degradation trajectory. It is recommended to add the DeepSeek V4 Pro code execution dimension to the priority tracking list for the next period; only consider adjusting model selection priority if it scores below 40 for two consecutive days.

The integrity rating remains pass, indicating that the model did not exhibit refusal-to-answer or format violation issues; the problem is strictly concentrated on execution correctness.

Data source: YZ Index | Run #281 | View raw data