Standardizing Generative AI Service Evaluation: An API-Centric Benchmarking Approach

The adoption rate of generative AI has exploded, with ChatGPT alone growing its user base by approximately 8x from mid-2023 to early 2025. The rapid release pace of models from giants like Anthropic, Google, Meta, Microsoft, Mistral, and OpenAI makes semi-annual benchmark cycles feel as slow as geological eras. For enterprises investing millions of dollars in inference infrastructure, one question grows increasingly urgent: How can we compare these systems in a way that reflects real-world production?

At GTC, MLCommons co-founder David Kanter unveiled the answer: MLPerf Endpoints, a complete reimagining of industry benchmark records. With over 125 member organizations, more than 90,000 reproducible results, and recognition from IEEE and ISO/IEC SC42, MLPerf has become a critical pillar for procurement decisions across government, industry, and academia. Endpoints aims to maintain that trust while adapting to an ecosystem transformed in just two years. You can try it out here.

Why Traditional Methods Need Change

Traditional MLPerf inference benchmarks used a tightly coupled architecture: the load generator and model server shared dependencies as a single local process. This works for classic ML, but generative AI deployments are API-first — whether on-premises, in the cloud, or as managed services.

Meanwhile, measuring GenAI performance is far from trivial. Real services need to combine accuracy, latency, throughput, and sequence length into a nonlinear multidimensional surface. Simple scenarios fail to capture long-tail queries, variable arrival patterns, and interactions with strict SLAs.

API-Centric Architecture

MLPerf Endpoints abandons the monolithic design in favor of a decoupled client that communicates with any model serving API endpoint via standard interfaces such as HTTP or gRPC. The benchmark client is lightweight and production-ready; the system under test only needs a URL. Submitters enjoy zero-friction integration — point to an endpoint and run. This architecture also allows managed cloud services and bare-metal deployments to compete fairly, something previous frameworks struggled to achieve.

Under the hood, a new generation of scalable load generators uses independent worker processes, pre-warmed connection pools, and ZeroMQ-based IPC to ensure the test framework itself never becomes a bottleneck, even when testing rack-scale systems.

Pareto Curves and Step Functions: New Metrics and Intuitive Comparison Visualization

The most innovative aspect is how results are presented. Each round varies concurrency and captures key metrics including TTFT (time to first token), throughput (tokens/sec), interactivity (user tokens/sec), and response latency. Submitters tune parallelism and batching for each working point, and visualization tools plot Pareto curves (e.g., throughput vs. interactivity) — allowing buyers to instantly see real-world trade-offs, such as serving more users versus maintaining responsiveness.

Critically, MLPerf Endpoints uses step functions instead of interpolated trend lines. GenAI performance is highly nonlinear; interpolation could fabricate unachieved levels and mask out-of-memory errors or P99 latency spikes. Step functions only show verified working points, eliminating "paper performance." Customers can easily compare these functions and match them to their own scenarios — high concurrency during the day, extreme interactivity at night.

Rolling Submissions: Benchmarking at the Pace of Software Updates

The most disruptive change is in the operating model. Traditionally, MLPerf releases followed a biannual cycle (training, inference, storage, etc.). In a market where models are updated every few weeks, this is too slow for RFP writing and hardware launches. Starting in Q2 2026, MLPerf Endpoints shifts to continuous rolling submissions: submitters can publish peer-reviewed and audited results at any time. Incremental submissions allow starting from a baseline Pareto curve and adding points as the software stack matures.

This approach borrows from mature methods in standards bodies like SPEC and TPC, adapted for the AI world. Review and audit requirements remain unchanged, ensuring industry-level robustness.

Future Outlook

The initial MLPerf Endpoints v0.5 demo results come from AMD, Google, Intel, KRAI, NVIDIA, and others, with support from over 30 organizations including Argonne National Laboratory, Broadcom, Dell, HPE, Lambda, Lenovo, Oracle, Red Hat, and the University of Florida. Results include models such as DeepSeek-R1, GPT OSS 120B, Llama 3.1 8B, QWEN 3 Coder 480B, running on nearly ten different systems.

Looking ahead, MLCommons invites the ecosystem to shape the next steps. Enterprise and IT buyers can join the advisory board; OEMs, CSPs, and ODMs contribute results to the rolling leaderboard; model developers and API providers integrate SOTA models and plan hosting roadmaps; researchers use Endpoints to anchor reproducible baselines. New models — especially practical commercial hot sellers — are continuously evaluated and included. You can try it yourself.

How to Get Involved: Rolling submissions for MLPerf Endpoints open in Q2 2026. To participate, contribute, or learn more, visit https://mlcommons.org/benchmarks/endpoints/ or join the working group.