GPT-o3 Smoke Evaluation Main Index Plunges 9 Points; Material Constraint Drops 20 Points in a Single Day

GPT-o3 scored 87.93 on the main index in today's Smoke evaluation, down 9 points from yesterday's 96.93, primarily driven by the material constraint dimension falling from 95.00 to 75.00.

Score Comparison Data

The code execution dimension held steady at 98.50. The material constraint dimension fell 20 points. Engineering judgment rose from 65.30 to 75.00, and task expression rose from 65.00 to 91.70. The integrity rating remained "pass."

Data Fact Breakdown

The Smoke evaluation includes only 10 questions per day, with 2 questions per dimension. The 20-point single-day loss in the material constraint dimension directly pulled the main index down by 9 points. The other two side dimensions rebounded noticeably, with task expression improving by 26.7 points.

Possible Cause Analysis

Question sampling fluctuation is the primary explanation. The two daily material constraint questions may have involved stricter citation requirements or longer context, and the model failed to satisfy the constraints in a single response, causing the dimension score to drop directly from 95.00 to 75.00. The likelihood of genuine model degradation is low, since the code execution dimension showed no decline and both side dimensions rose in tandem, indicating no systematic deterioration in the model's overall output capability.

The gains in engineering judgment and task expression may reflect that today's sampled questions better matched the model's strengths in structured expression, complementing the material constraint questions. Single-day data cannot distinguish random fluctuation from genuine capability change, but a sharp 20-point drop in one dimension is an extreme sample within the normal range for a test with only 2 questions per dimension.

Implications for Users

In scenarios with heavy material constraints, enterprises need to add extra manual verification steps. For teams that rely on GPT-o3 to generate reports with source citations, contract clauses, or technical documentation, today's data suggests that constraint failure can occur in a single output; a secondary review step is recommended after critical outputs.

For developers with high reliance on code execution, the 98.50 score remains unchanged, so the model can continue to be used for programming tasks in the short term. For mixed scenarios requiring both material fidelity and engineering judgment, it is advisable to delegate the material constraint portion to other models or manual processing.

Strategic Judgment

This 9-point decline in the main index primarily stems from extreme single-dimension volatility in material constraints rather than overall capability degradation. The concurrent rise in engineering judgment and task expression further supports the volatility explanation. The next Smoke evaluation should track whether the material constraint dimension recovers to above 90 points; if it stays around 75 points for two consecutive days, the model's suitability for material-constraint scenarios should be reassessed.

Current data does not support classifying GPT-o3's performance on the material constraint dimension as a systemic issue. Selection teams are advised to continue tracking the standard deviation of scores for the same dimension over the next 3-5 days, rather than adjusting deployment plans based on a single day's results.

Data source: YZ Index | Run #284 | View raw data