DeepSeek V4 Pro Material Constraint Score Plunges 18.1 Points, Yet Code Execution Hits a Perfect 100

In today's Smoke evaluation, DeepSeek V4 Pro's material constraint score fell from 88.10 to 70.00, a drop of 18.1 points, while code execution rose from 72.00 to 100.00 and the main leaderboard score climbed from 79.25 to 86.50.

Breaking Down the Score Facts

This evaluation covered only the daily 10 questions, with 2 questions per dimension. The material constraint dimension scored 70.00 today, down 18.1 points from yesterday's 88.10; the code execution dimension jumped from 72.00 to a perfect 100.00; engineering judgment held steady at 100.00; task expression rose from 75.00 to 81.70; and the integrity rating remained at pass.

Cause Analysis: Sampling Volatility, Not Model Degradation

The simultaneous large opposing moves in the material constraint and code execution dimensions most likely stem from the randomness of question sampling. The Smoke evaluation uses only 2 questions per dimension, so a single misstep can cause a swing of roughly 15–20 points. Today's material constraint losses were concentrated in judgments of fidelity to the provided material, while the code execution questions may simply have fallen along an execution path the model handles well. Engineering judgment and task expression held steady or rose slightly, further confirming that overall capability has not undergone systematic degradation.

If a model were truly degrading, multiple dimensions would typically decline in sync, rather than one dimension collapsing while another hits a perfect score.

What This Means in Practice for Users

Teams that rely heavily on material constraint scenarios—such as contract clause extraction, policy text comparison, or faithful summarization of long documents—should add a manual review step based on today's data. Teams with stable code execution performance can continue using the model for script generation and debugging tasks, but should set up a second round of validation in material-constraint-related workflows.

• Material-constraint-sensitive scenarios: compare key outputs sentence by sentence against the source material to prevent the accumulation of localized errors at the 18.1-point level.
• Code-execution-dominant scenarios: the 100.00 score can go straight into production pipelines.

Strategic Judgment

Based on a single Smoke dataset, the 18.1-point drop in DeepSeek V4 Pro's material constraint score is more likely sampling volatility than a signal of capability degradation. The main leaderboard score still rose 7.3 points, indicating overall usability is unaffected. If material constraint remains below 75 in the next evaluation round, further validation will be needed; if it rebounds above 85, today's result is confirmed as random fluctuation. The current data does not support a downward revision of the model's overall capability assessment.

When enterprises select models, they can use DeepSeek V4 Pro as their primary model for code execution while retaining human review or cross-validation across multiple models in the material constraint stage. No data on the stability dimension was provided this time, so multi-day standard deviation should be monitored continuously to confirm consistency.

Data source: YZ Index | Run #382 | View raw data