GPT-6 Luna Code Execution Plunges 24.1 Points as Material Constraint Climbs 32

In today's Smoke evaluation, GPT-6 Luna's code execution score dropped from 99.10 yesterday to 75.00, its material constraint score rose from 62.80 to 94.80, and its main leaderboard score climbed from 82.77 to 83.91.

The Facts Behind the Score Changes

The code execution dimension stood at 99.10 yesterday and 75.00 today, a single-day decline of 24.1 points. The material constraint dimension stood at 62.80 yesterday and 94.80 today, a single-day rise of 32 points. Engineering judgment (secondary leaderboard, AI-assisted assessment) fell from 100.00 to 63.90, and task expression (secondary leaderboard, AI-assisted assessment) fell from 100.00 to 83.80. The main leaderboard score was 82.77 yesterday and 83.91 today, up 1.1 points. The integrity rating remains at pass.

Analysis of Possible Causes

The Smoke evaluation contains only 10 questions per day, 2 per dimension, so the small sample size means question-draw volatility is the main mechanism. The large changes in opposite directions for code execution and material constraint indicate that the code execution questions drawn that day differed significantly in difficulty or type from the previous day, while the material constraint questions were relatively easy. Engineering judgment and task expression declined at the same time, but both are secondary leaderboard dimensions and do not directly affect the main leaderboard calculation. The main leaderboard score rose only slightly, indicating that after the rises and falls of the two core auditable dimensions offset each other, the overall result remained stable.

There is no multi-day consecutive data to support a conclusion of genuine model degradation; a single-day opposing change is more consistent with random sampling characteristics than with a systematic decline in model capability.

Implications for Users

Teams that rely heavily on code execution may encounter a higher failure rate in that day's tasks and should prepare backup plans or manual review. Scenarios sensitive to material constraints, such as document generation and citation checking, may see better performance. Enterprises that select models based on main leaderboard rankings can continue to observe, since the main leaderboard score has not shown any substantial decline.

Strategic Judgment

Based on the existing single-day data, GPT-6 Luna's main leaderboard score remains stable, and the sharp drop in code execution is more likely a result of the question draw than a signal of model degradation. The next evaluation round needs to verify whether code execution rebounds above 90 points; if it stays below 80 for two consecutive days, that would warrant further attention. The large fluctuation in the engineering judgment secondary leaderboard does not change the main leaderboard conclusion.

Overall, the current data does not support a downward revision of the long-term assessment of GPT-6 Luna's code execution capability, and continued tracking of dimension consistency in subsequent Smoke evaluations is recommended.


Data source: YZ Index | Run #432 | View raw data

This article is from Winzheng Index blog, translated in full by Winzheng (winzheng.com). Click here to view the original When republishing the translation, please credit the source. Thank you!