Co-Design Bench Shows Strongest Coding Agent Completes Only 26.4% of Full Tasks

On November 8, 2026, Co-Design Bench, released by Alibaba together with five universities, showed that GPT-6 Astra met 69.9% of acceptance criteria on average across 273 multimodal tasks, but fully completed only 26.4% of the entire tasks.

Factual Reconstruction

The benchmark distilled 273 tasks from 26,314 real developer coding sessions, with tasks drawn from 37,463 deduplicated queries. It includes two tracks, multimodal editing and repository editing, covering localized edits and repository-level revisions. The tasks come with 2,331 hidden expert-written acceptance criteria and are scored using the judge-then-refute protocol.

Testing covered 10 model endpoints. For repository-editing tasks spanning three to four types of changes, all endpoints had lower standard completion rates than for tasks involving only one or two types of changes. The benchmark does not provide gold-standard code or target screenshots; it is audited through source-code and browser evidence and independent LLM validators.

Mechanism Breakdown

The benchmark divides capabilities into three dimensions: contextual understanding, UI transformation, and functional implementation. Real user requests often include image, audio, video, or document inputs, and agents must complete revisions without target outputs. The 43.5 percentage point gap between partial satisfaction of acceptance criteria and full task completion reflects that in multi-step coordinated execution, even after some criteria pass, later-stage failures can still prevent overall task delivery.

The performance decline on the repository-editing track indicates that as the number of change types increases, it becomes harder for agents to maintain consistency and completeness. The protocol uses separate validators to challenge positive judgments, reducing misjudgments of visual and interaction claims.

Industry Impact

These results show that the usability of current coding agents in real user scenarios is still constrained by the conversion from partial satisfaction to complete delivery. The benchmark highlights the gap between high scores on synthetic tests and delivery in real applications, which may prompt developers to focus more on multi-step coordination and error recovery mechanisms.

The testing framework, involving multiple institutions, provides a reference for evaluating agentic models’ performance in real application revisions. The low full-task completion rate may affect adoption decisions for related products in production environments.

Strategic Assessment (Analysis, Not Fact)

Based on the available facts, the gap revealed by the benchmark may push the industry to re-examine evaluation methods and prioritize model architectures that support long-sequence change coordination. If subsequent tests follow similar criteria, local optimization strategies may need to give way to improvements in end-to-end delivery capability.