Real-SWE Benchmark Exposed: Fable 5.1 Solves Only 38.8% of Private Code Tasks

Specific Labs released the Real-SWE benchmark in September 2026. Fable 5.1 achieved a solve rate of only 38.8% across 10 real enterprise private-code tasks.

The Facts

Real-SWE includes 10 production code tasks from real enterprises, covering billing, tax, and data migration scenarios. The test used complete records of 640 scored runs, with each task run eight times and averaged. Fable 5.1, using Claude Code tools, reached 38.8%; GPT-6 Astra, using Codex CLI, reached 33.8%; Gemini 3.8 Flash paired with Gemini CLI reached 31.2%. GLM 5.3 was 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 was 18.8%, and GPT-5.6 SolCodex CLI was lowest at 16.2%. For six tasks, all models had a pass rate below 15%.

The task code came from authorized private production repositories, including company-specific rules and multi-service interactions. Agents had to handle environments such as AWS simulators, Docker, Kubernetes, PostgreSQL, and MongoDB, as well as business tools such as Slack and Email. Prompts were relatively brief, requiring models to discover implementation details on their own.

Mechanics Breakdown

Public benchmarks mostly use expert-generated or synthetic tasks, with clear instructions and publicly available code. Real-SWE, by contrast, requires agents to understand proprietary architecture, preserve user-dependent behavior, and modify code under real operational constraints. A billing task example showed that it required simultaneously handling tax rate calculations, destination pricing, exempt customer handling, and interactions with tax authority sandboxes or production environments, involving changes across multiple service files.

The test used native toolchains rather than a unified interface, directly evaluating the combined effect of models and actual development environments. This makes the results closer to engineers' daily workflows, but it also magnifies the difficulty models face in understanding company-specific context.

Industry Impact

This benchmark is the first to use private data to quantify inflated performance on public leaderboards. Models that stand out in public evaluations generally have low pass rates on real enterprise tasks, showing that current frontier models still struggle to directly take on production-level code modifications. In terms of cost-effectiveness, the most expensive model's per-task cost is higher than Gemini 3.8 Flash's $2.50.

For the evaluation industry, the methodology faces pressure to adjust. Future benchmarks may turn more toward private code repository testing to narrow the gap with actual productivity. When enterprises adopt AI coding tools, they will focus more on real-world scenario testing rather than public rankings.

Strategic Assessment (Analysis, Not Fact)

Based on the existing results, the low pass rates of models on complex business rules and multi-service interaction tasks may prompt vendors to prioritize optimizing context understanding and toolchain integration capabilities. In the long run, the emergence of private benchmarks may push the industry from public synthetic testing toward a hybrid evaluation model, but the specific evolution still requires more similar data to support it.