SWE Refactor Bench Test: Only 5.4% of 520 Runs Across 8 Models Completed Full Repository Migration

On August 24, 2026, the SWE Refactor Bench paper was released on arXiv, conducting full-repository technology stack migration tests across 520 runs of 8 frontier models, with only 28 runs (5.4%) passing all evaluation stages.

Fact Restoration

The benchmark includes 20 full-repository migration tasks covering 4 types of technical debt. The three-stage evaluation protocol consists of migration audit, behavior testing, and agent verification. Among 520 runs, no model produced an accepted solution for 13 tasks, and the best model, Claude Opus 5, scored 47.0/100. Language rewriting scored only 5.6, while toolchain rewriting scored 31.4. Among 340 runs that passed the migration audit, 58% reached 99% of fixed checks, but only 26% reached 100%.

Mechanism Analysis

The paper notes that "migration completeness" and "behavioral correctness" are two completely independent capability dimensions. Some runs skipped migration and preserved original behavior, being intercepted at the migration audit stage; most runs attempted migration but introduced behavioral defects, failing at the behavior testing stage. Existing benchmarks only evaluate behavioral correctness, allowing agents to pass tests by copying the original implementation—what the paper calls "Blindness." The three-stage protocol simultaneously measures whether migration actually occurred and whether behavior is correct, avoiding this vulnerability.

Industry Impact

This result directly challenges the current narrative that AI coding agents can replace engineers. Models show significant performance variation across migration categories, with toolchain rewriting being relatively easy while language rewriting is nearly infeasible. Most of the 20 tasks received no successful solution, indicating that long-cycle, full-repository refactoring remains beyond the capabilities of current frontier models. Developers relying solely on behavioral tests may underestimate the actual risks of migration.

Strategic Assessment

(The following is analysis, not fact) The difficulty gradient design of benchmarks is crucial for discovering capability boundaries. By introducing the migration audit stage, SWE Refactor Bench forces models to actually perform refactoring rather than evade it, exposing independent capability defects that previous benchmarks failed to capture. To improve agent reliability in real engineering scenarios in the future, both migration completeness and behavioral correctness must be optimized simultaneously, rather than merely pursuing test pass rates.

Source: arXiv paper 2608.23564 and related reports. All data comes from publicly available results in the paper, with no additional speculation added.