Mlperf Inference V5 1 Results

MLCommons announced the official release of the MLPerf Inference v5.1 benchmark results, the latest iteration following v5.0, focusing on evaluating inference performance of generative AI models. Driven by contributors such as LMSYS Org, this round expands the benchmark scope and introduces more challenging workloads to help the industry measure system performance in real-world scenarios.

New Benchmark Models and Scenarios

This test adds multiple new benchmarks, emphasizing the inference capabilities of large language models (LLMs):

• Llama 3.1 405B: Offline, Server, and Single Stream tests in data center scenarios, with input sequence length up to 2048 tokens and output of 128 tokens.
• Llama 3.2 1B / 3B: Edge device scenarios, supporting image-text multimodal tasks.
• Classic benchmarks such as BERT, ResNet-50, Stable Diffusion are retained to ensure comprehensive coverage of computer vision, NLP, and generative tasks.

The test rules are strict, including performance counter verification and power measurement standardization, and submitted systems must disclose their hardware configurations.

Performance Highlights and Record Holders

In the data center category, NVIDIA dominates multiple records:

• H100 SXM achieves the highest system throughput in the Llama 3.1 405B Offline scenario.
• H200 NVL performs well in Server and Single Stream, with significant low-latency optimization.

AMD MI300X breaks records in the Llama 405B Server scenario, while Intel Gaudi3 shows enhanced competitiveness on small and medium-sized models. On the edge side, Qualcomm and MediaTek's SoCs achieve efficient inference on Llama 3.2.

Key Data and Trends

Compared to v5.0, results show inference performance improvements of 20–50%, thanks to optimization techniques such as FP8 quantization and FlashAttention. Vendors have also explored runtime frameworks like SGLang and vLLM.

Industry Significance

The MLPerf Inference v5.1 results highlight the intense competition in AI hardware, driving the paradigm shift from training to inference. Future tests will further integrate more open-source models, such as DeepSeek and Qwen, to promote fair comparisons. Developers can download the full dataset and submission guidelines via the official link.