DeepSeek V4 Pro Main Score Plummets 11.9 Points, Code Execution Drops 13.3

DeepSeek V4 Pro's main score in today's Smoke evaluation fell from yesterday's 93.84 to 81.93, a drop of 11.9 points.

Core Data Comparison

The code execution dimension dropped from 100.00 to 86.70, a decrease of 13.3 points; the material constraints dimension fell from 86.30 to 76.10, a drop of 10.2 points. The engineering judgment dimension remained unchanged at 75.00 points, and the task expression dimension stayed at 95.00 points, with the integrity rating remaining pass.

Possible Causes for Score Changes

The Smoke evaluation only includes 10 questions per day, 2 per dimension, so daily draw fluctuations are within normal range. The simultaneous significant decline in code execution and material constraints, with zero fluctuation in the two side dimensions, indicates that the issues are concentrated in the auditable dimensions of the main scoreboard. The loss in code execution may be due to the two questions drawn today involving more complex multi-step reasoning or edge cases, while the loss in material constraints could be related to long-context citations or format constraints in the questions. Current data does not indicate any changes in model parameters or training, so this is more likely a score fluctuation caused by question draw randomness rather than actual model degradation.

Specific Implications for Users

Teams heavily reliant on code execution should perform additional manual review of DeepSeek V4 Pro's generated code today and in the following days, especially in automated testing or production script scenarios. Scenarios sensitive to material constraints (such as legal contract generation or long-document summarization) should also increase verification frequency to avoid downstream errors due to format or citation deviations. Teams with stable engineering judgment and task expression can continue using the model as planned and do not need to adjust their workflows immediately.

Strategic Assessment

A single-day drop of 11.9 points is within the historical fluctuation range of the Smoke evaluation, but if core dimensions of the main scoreboard continue to lose points simultaneously for more than two consecutive days, it warrants focused verification in the next period. If code execution and material constraints rebound above 90 points tomorrow, the change is more likely attributable to draw randomness; if they remain below 85 points, stability issues on specific question types should be considered. Current data does not support a full downgrade of DeepSeek V4 Pro, but users relying on its code execution capability are advised to increase redundant checks in the short term.

Data source: YZ Index | Run #235 | View raw data