MLCommons has been selected as the benchmark team for the STRIDE Ventures AI Efficiency Challenge. Supported by the U.S. National Science Foundation Directorate for Technology, Innovation, and Partnerships (NSF TIP) through a two-year, $3.5 million grant, MLCommons will develop MLPerf Agents—a new benchmark for measuring the efficiency of agentic AI systems in completing real work.
Why Agentic AI Needs a New Benchmark
Existing benchmarks primarily measure the time, cost, and energy of a single inference, but they cannot fully reflect how agentic AI actually operates. Agentic systems involve multiple rounds of interaction, tool calls, information retrieval, context growth, failure recovery, and self-checking, which is fundamentally different from a single model responding to a prompt.
MLPerf's existing inference benchmarks already capture some of this complexity in certain areas, and MLPerf Agents will extend this to full workflows, covering agent frameworks, model selection, retrieval, tool execution, orchestration, state management, failure recovery, and validation, thereby comprehensively assessing end-to-end efficiency.
Four Innovations of MLPerf Agents
- Efficiency tied to success: Cost, energy, latency, and throughput are counted only for successfully completed tasks, with validation methods informed by the MLCommons AI Risk and Accountability Working Group.
- Full-stack measurement scope: Resource accounting covers the entire agent technology stack, precisely identifying inefficiencies.
- Comparable results across architectures: Unified workloads, tracking rules, and validation standards make it easier for buyers to compare complete solutions.
- Trustworthy validation mechanism: A hidden validation dataset combined with MLPerf peer review and audit processes reduces the risk of benchmark cheating.
Built on Eight Years of MLPerf Experience
MLPerf Agents will build on mature benchmarks such as MLPerf Inference and MLPerf Power, follow an open collaborative process, and have workloads, rules, and reference implementations developed jointly by the MLCommons community. David Kanter and Vijay Janapa Reddi will lead the effort, with support from organizations including Qualcomm, NVIDIA, and Intel.
About the AI Efficiency Challenge
The challenge is operated by STRIDE Ventures and is a national innovation platform jointly launched by the U.S. National Science Foundation and Start2 Group. More information is available at stride-ventures.com/ai-efficiency-challenge.
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接