DeepSeek V4 Pro Drops 16.9 Points in Smoke Evaluation Main Rankings, Code Execution Down 28 Points in a Single Day

DeepSeek V4 Pro's main ranking score in today's Smoke evaluation dropped from 96.99 to 80.10, a decrease of 16.9 points.

Score Breakdown and Direct Drivers

The code execution dimension fell from 100.00 to 72.00, accounting for nearly all of the main ranking decline; material adherence dropped from 93.30 to 90.00, a decrease of only 3.3 points. Engineering judgment rose from 75.00 to 100.00, while task expression fell from 91.70 to 79.70. Since the main ranking is weighted solely by code execution and material adherence, the 28-point single-day loss in code execution directly determined the overall ranking drop.

Question Sampling Volatility or Genuine Model Degradation

The Smoke evaluation uses only 2 questions per dimension per day, resulting in a very small sample size; a single question failure can cause fluctuations of 28 points. Code execution scored full marks yesterday and 72 today, indicating that at least one of the two questions had a clear error or was incomplete. Material adherence dropped only 3.3 points over the same period, suggesting no systemic decline in the model's basic ability to stay faithful to source materials. Engineering judgment conversely rose to full marks, further showing that the model's performance across different ability axes does not degrade synchronously. Therefore, current data supports volatility from question sampling rather than a sustained deterioration in model parameters or post-training capabilities.

Implications for Users

Teams heavily reliant on code generation should increase manual verification steps today and over the next few days, especially for scenarios involving multi-step reasoning or API calls. Material adherence remains above 90, meaning limited impact on RAG applications requiring strict citation or formatting constraints. Engineering judgment has risen to full marks, indicating that in tasks requiring system design and trade-off judgment, the model actually produces better outputs and can be prioritized during the prototyping phase.

Strategic Assessment

A single-day 28-point fluctuation in code execution is explainable under a 2-question sample, but if code execution continues to fall below 85 in the next evaluation round, multi-day continuous tracking should be initiated to confirm whether genuine degradation exists. Current data does not support conclusions about DeepSeek V4 Pro's overall capability; it merely indicates that the code execution dimension is sensitive to small-sample fluctuations. Enterprises should treat the Smoke evaluation as an early warning signal rather than a final proof of capability.

Data source: YZ Index | Run #229 | View Raw Data