What is MLPerf Endpoints, and why it matters to buyers

What MLPerf Endpoints Measures

MLPerf® Endpoints focuses on evaluating the real-world delivery capability of inference systems in production environments. Rather than testing a single tuned configuration in isolation, it measures the model, hardware, serving software, and network as an integrated endpoint under real workloads, simulating how customers use inference services day to day.

What Buyers Gain

A complete operating frontier rather than a single data point. Each Endpoints result presents a Pareto front across concurrency levels, giving buyers a clear view of throughput and response performance under different loads, from low-concurrency interactive applications to high-concurrency service scenarios.

Metrics directly relevant to decisions. Five core metrics include: concurrency (the number of queries the system serves simultaneously), first-token latency P90, system throughput (tokens/s), system throughput per watt (a TCO proxy metric), and user interactivity (tokens/s/user).

Results that keep pace with the market. Endpoints uses a rolling submission mechanism, publishing a batch of results every few weeks to ensure the benchmark keeps up with the latest models and software.

A credible peer-review process. Results are released through a peer-review process, giving buyers a basis for assessing reliability.

How Endpoints Differs from Other Benchmarks

Traditional MLPerf Inference tests a single inference run in fixed scenarios and at preset points, whereas Endpoints measures endpoints as they are actually sold and used, covering the full range of concurrency and both single-turn and multi-turn agentic tasks. It shows the complete frontier curve, includes the full serving stack, and clearly reveals the trade-off between throughput and user experience, with rolling updates.

How to Read Endpoints Results

Buyers can follow four steps: locate the concurrency region corresponding to their workload; set an SLA (TTFT P90 and interactivity); compare throughput among systems that meet the SLA; and use provisioned power for normalized comparisons.

Why MLCommons Moved to Endpoints

This evolution reflects a real shift in how enterprises buy, deploy, and consume AI. Modern buyers purchase a fully integrated serving stack rather than bare hardware, and Endpoints better predicts real operational performance under dynamic, multi-turn production workloads.