Mlperf Client V0 6

The MLCommons organization recently released the benchmark results for MLPerf Client v0.6, the first standardized benchmark suite focused on client-side inference performance, targeting scenarios such as smartphones, laptops, and edge devices.

New Version Highlights

MLPerf Client v0.6 expands the workload to support more cutting-edge AI models, including classic tasks and emerging large models:

• Image Classification: ResNet-50
• Object Detection: RetinaNet
• Speech-to-Text: RNN-T
• Medical Image Segmentation: 3D-Unet
• Recommendation System: DLRM v2
• Natural Language Processing: BERT
• Large Language Model: Llama 2 70B
• Generative AI: Stable Diffusion

Test scenarios include Offline, Server, SingleStream, and the newly added MultiStream (2, 4, 8 streams), with precision requirements of INT8 quantization or FP16.

Submission Results Overview

Multiple vendors actively participated, including Arm, Intel, MediaTek, NVIDIA, Qualcomm, and Samsung. Below are some top performance highlights:

Llama 2 70B (Offline Scenario)

• NVIDIA (H100 TensorRT-LLM): Leading samples/second throughput
• Qualcomm (Snapdragon 8 Gen 3): Best on mobile platforms

Stable Diffusion (SingleStream)

• NVIDIA dominates generation speed
• MediaTek (Dimensity 9300): Efficient mobile performance

BERT (Server Scenario)

Detailed results can be viewed on the official page. Organizations such as LMSYS Org contributed benchmark framework optimizations.

Significance and Outlook

This benchmark reveals the rapid development of client-side AI hardware, especially breakthroughs in quantized large models. Future versions are expected to include more multimodal tasks, driving AI migration from the cloud to the edge.