DeepSeek V4 Pro Code Execution Plummets 22.5 Points to 72; Main Leaderboard Falls to 84.6

In today's Smoke evaluation, DeepSeek V4 Pro's Code Execution dimension score fell from 94.50 to 72.00, while its overall main leaderboard score dropped from 92.39 to 84.60.

Score Comparison Data

Material Constraints rose from 89.80 to 100.00, Task Expression rose from 65.00 to 87.50, Engineering Judgment remained unchanged at 100.00, and Integrity Rating remained pass.

Breaking Down the Data

The Smoke evaluation has only 10 questions per day, 2 per dimension, so single-day fluctuations are within the normal range. The simultaneous appearance of a -22.5-point move in Code Execution alongside +10.2 points in Material Constraints and +22.5 points in Task Expression shows clear divergence in the model's performance across dimensions.

Possible Causes Analysis

Question sampling variance is the primary explanation. With only 2 Code Execution questions per day, the sample size is extremely small; one complex multi-step reasoning question or one simple calculation question can cause swings of more than 20 points. The simultaneous improvement in Material Constraints and Task Expression indicates no systemic degradation in the model's overall output capability; more likely, the Code Execution questions sampled this time happened to expose a specific weakness.

Actual degradation is unlikely. Engineering Judgment remained at full score and Material Constraints reached full score, indicating that the model remains stable in constraint adherence and judgment logic. If degradation were at the parameter level, it would usually be accompanied by simultaneous declines across multiple dimensions, rather than gains in some areas offsetting losses in others.

Implications for Users

Teams heavily reliant on Code Execution need to additionally validate outputs today. In scenarios such as financial modeling and data-processing script generation, a score of 72.00 means error rates may rise, and additional manual review is recommended.

Scenarios sensitive to material fidelity can be used with confidence. A Material Constraints score of 100.00 shows that the model has reached its best level in citation and constraint adherence, making it suitable for document generation and compliance-checking tasks.

Strategic Assessment

This fluctuation is more likely due to question sampling than model degradation. Single-day Smoke data is insufficient to determine a long-term trend; it is recommended to observe for three consecutive days whether the Code Execution dimension rebounds above 90. If it remains below 80, an official long leaderboard retest should be initiated.

The main leaderboard score of 84.60 remains above the daily level of most competitors, and DeepSeek V4 Pro's overall competitiveness has not been fundamentally affected. Developers can continue using it, but should increase unit test coverage for Code Execution tasks.

The simultaneous occurrence of a 72.00 in Code Execution and a 100.00 in Material Constraints points to question sampling rather than model degradation.

Engineering Judgment and Integrity Rating remained at full score and pass, ruling out systemic capability decline. If Code Execution rebounds in the next Smoke run, this incident can be classified as normal statistical noise.

Enterprises selecting models should incorporate daily Smoke fluctuations into their monitoring systems rather than drawing conclusions directly from single-day data. DeepSeek V4 Pro's stable performance in Material Constraints and Engineering Judgment remains a core advantage.

Data source: YZ Index | Run #380 | View raw data