Mlperf Auto V0 5 Results

MLPerf Auto v0.5 Benchmark Overview

MLCommons has released the latest results of MLPerf Auto v0.5, the authoritative benchmark in the AutoML domain, focusing on automated hyperparameter tuning and model selection. Unlike traditional CV or NLP tasks, this edition specifically targets large language model (LLM) scenarios, introducing the Elo Rating from Chatbot Arena as a quality proxy metric to reflect real user preferences.

LMSYS Org’s Leading Submission

LMSYS Org (lmsys.org) submitted the top-ranked result. They developed an automated pipeline based on Chatbot Arena Hard Auto, utilizing millions of user voting data to drive the optimization process.

• Core Tech Stack: SGLang runtime + vLLM backend, supporting efficient continuous batching and PagedAttention.
• Target Model: Llama 3.1 405B-Instruct, trained on 8x H100 GPUs.
• Key Results: Elo Rating: 1312 (highest score), quality-time product of 1.28e6, far exceeding the baseline.

Their approach achieved optimal Elo improvement under a limited computational budget through reinforcement learning (RL) fine-tuning and architecture search.

Other Competitor Performances

Google Cloud used AutoML Tables, and NVIDIA submitted optimized results based on NeMo, achieving Elo scores of 1275 and 1280, respectively. However, in multi-objective optimization (quality vs. efficiency), LMSYS’s Pareto front was superior.

Technical Highlights and Insights

This benchmark introduced distributed AutoML, supporting multi-node collaborative search. LMSYS’s innovation lies in the real-time integration of Arena data, avoiding biases from synthetic datasets. Additionally, SGLang’s zero-overhead batching significantly reduces inference latency.

The results indicate that LLM AutoML is shifting from static benchmarks to dynamic user-driven optimization, with future plans to incorporate multimodal tasks.

Full results are available on the MLCommons official website.