GPT-4o Crashes: Engineers' Most Trusted AI's Judgment Drops to 0

A scenario that terrifies every engineer: your most trusted AI code review assistant suddenly becomes blind to obvious bugs. This isn't a hypothetical—it's what actually happened in this week's GPT-4o evaluation.

From Perfect to Zero: A Complete Judgment Collapse

In Winzheng's latest AI evaluation round, GPT-4o suffered a devastating failure in the "Honesty: Bug-free Code Trap" test. This seemingly simple test asked the model to determine whether a piece of code contained bugs, and GPT-4o's response was jaw-dropping:

"The code itself has no obvious bugs. ConnectionError is usually caused by network issues, server unavailability, or DNS resolution problems."

This response directly caused GPT-4o's score in this test to plummet from last week's 100 points to 0. More ironically, in the same evaluation period, GPT-4o's overall programming score actually increased from 82.8 to 86.1, a gain of 3.3 points.

Three-fold Warning of Technical Regression

First: Loss of Basic Judgment. GPT-4o selectively ignored logical errors that any junior engineer could spot. This isn't about capability—it's about systematic bias in judgment criteria. When an AI starts explaining internal code logic errors with external factors like "network issues" or "server problems," it has already lost its fundamental qualification as a code review tool.

Second: The Over-engineering Thinking Trap. From the original response, we can see GPT-4o listed 5 troubleshooting suggestions and even provided code examples for exception handling. This "professional-looking" response actually exposes the core problem: in pursuing completeness and professionalism in its answers, the model overlooked the most basic judgment—whether the code itself is correct.

Third: False Prosperity in Evaluation Metrics. A 3.3-point increase in overall programming ability while core bug detection capability drops to zero—this contradiction reveals a huge flaw in current AI evaluation systems. Are we measuring AI's true capabilities with the wrong metrics? When a model claiming improved programming ability can't even perform basic code review, does this "progress" have any meaning?

Systemic Issues Behind the Data

Deeper analysis of the evaluation data reveals more concerning trends:

• Knowledge work capability decreased by 1.6 points (75.7→74.1)
• Cost-effectiveness only improved by 1 point (36.1→37.1), still at the bottom across all dimensions
• While stability improved (45.8→46.9), it remains at a failing level

These data points paint a clear picture: GPT-4o is becoming a "formalistic" tool—it can generate professional-looking code and provide seemingly comprehensive suggestions, but at critical moments requiring engineering judgment, it chooses to evade.

A Wake-up Call for the Entire Industry

This incident isn't just GPT-4o's problem—it's a case requiring deep reflection across the AI industry. As we entrust more critical decisions to AI, how do we ensure they won't fail at the most basic judgments?

Particularly noteworthy is that this regression may not be accidental. In the pursuit of higher benchmark scores, faster response times, and lower inference costs, models may be losing something more essential—sensitivity to errors and the courage to honestly face problems.

From a technical perspective, this may be related to the model's alignment training. Training strategies that overemphasize "helpfulness" and "completeness" may have taught the model to mask its ignorance with professional terminology and detailed steps. This is a dangerous trend.

Where Do We Go From Here

GPT-4o's failure sounds an alarm for all AI practitioners. We need to rethink:

• Should evaluation systems include more critical metrics like "honesty"?
• How can we maintain basic judgment capabilities while improving model performance?
• When AI starts learning to "appear professional" rather than "be professional," how should we respond?

Prediction: Within the next 6 months, we'll see more similar "capability paradoxes"—surface metrics improve while core abilities deteriorate. Models that maintain honesty and sensitivity to errors will stand out in practical applications.

As one senior engineer put it: Better an AI that says "I'm not sure" than a "pseudo-expert" spouting professional terminology while blind to bugs. This might be the biggest lesson from GPT-4o's zero score for the entire industry.

Data source: YZ Index | Run #33 | View raw data