Our mission, updated: standardize AI measurement and enable informed decisions

Over the past eight years, both the AI industry and MLCommons have undergone tremendous growth and change. None of this would have been possible without the support of all our members.

Mission Update: Standardizing AI Measurement

Today, MLCommons officially updates its mission to clarify its core focus: Standardize AI measurement and enable informed decisions.

Measurement is the cornerstone of responsibly unlocking AI's enormous potential. AI systems are like the ultimate black box, relying on billions of parameters to generate complex behavior. Our explanations of how they work remain vague and inadequate, and system complexity continues to rise. Training and running AI is costly and energy-intensive, and better measurement can help us empirically characterize system behavior and its operational costs.

Scope and Value of Measurement

We are committed to measuring any characteristic that affects the value of AI hardware, software, or data—speed, power consumption, reliability, or quality. This will support informed decisions about current AI systems, cost-benefit trade-offs, guidance for innovation, and risk management (such as assurance and insurance). The work benefits academic researchers, hardware and software vendors, deployers, end users, and even policymakers.

We do this to reduce risk in the AI market and promote its growth.

Building and Operating Industrial-Grade Benchmarks

MLCommons focuses on developing and operating standardized measurements—industrial-grade benchmarks. These tests combine detailed protocols with test tools, retained datasets, evaluation models, and other artifacts. Because AI testing is complex and subjective, multi-party implementations based only on paper standards rarely produce comparable measurement results. Artifacts need technical and legal protection against overfitting, and measurements must evolve with technology—this requires an organization that can operate credibly and sustainably.

Our work includes defining, building, operating, validating, explaining, evolving, and connecting benchmarks:

• Defining: Clarifying what is measured and which metrics are used, and developing rigorous protocols through a transparent community process.
• Building: Translating real-world use cases into rigorous test scenarios and maintaining evaluations that evolve as AI advances.
• Operating: Delivering high-integrity services through infrastructure such as Candela.
• Validating: Providing third-party confirmation of results to support procurement, insurance, and compliance.
• Explaining: Helping users understand results and working with standards organizations.
• Evolving: Continuously improving to reflect technological progress.
• Connecting: Aligning with global standards and policy frameworks.

Organizational Changes and Future Plans

MLCommons will implement several changes: improving its website and resources, adopting a product marketing mindset, supporting performance and reliability trade-offs, adjusting membership fees to attract deployers, and expanding into new areas such as healthcare and financial services.

We welcome feedback and will continue to evolve as fields such as AI data science develop. Thank you to all our members for your support.