Mlperf Client V1 0

The MLCommons organization has released the first results of the MLPerf Client v1.0 benchmark, the first benchmark specifically designed for AI inference performance on client and edge devices, aiming to evaluate generative AI and traditional AI task performance on mobile devices.

Benchmark Overview

MLPerf Client v1.0 focuses on mobile and edge computing scenarios, introducing six key tasks:

• Image Super-Resolution: Uses the Real-ESRGAN x4 model on the DIV2K validation set.
• Image Classification: MobileNet V3 on the ImageNet dataset.
• Object Detection: YOLOv5 on the COCO val2017 dataset.
• Speech-to-Text: Conformer model on the LibriSpeech dataset.
• Chatbot: First introduction of the Llama 3.1 8B Instruct model, supporting conversation generation on the ShareGPT-4o dataset.
• Image Generation: Stable Diffusion XL Turbo generates images from MS COCO 2014 prompts.

Test scenarios cover Offline (batch processing), Server (real-time requests), SingleStream (single stream), and MultipleStream (multiple streams), with quality target thresholds ensuring result comparability.

Participants and Result Highlights

Submissions for this round come from vendors such as Arm, Intel, MediaTek, Qualcomm, Samsung, and Tenstorrent. Key findings include:

• Qualcomm's Snapdragon platform leads in multiple tasks, such as the Chatbot SingleStream scenario achieving high throughput.
• Arm Total Design excels in the Chatbot task, enabling efficient Llama 3.1 8B inference.
• MediaTek and Samsung submit strong results in image generation and speech tasks.
• The first Chatbot results highlight the importance of quantization techniques (e.g., INT4) and optimized frameworks.

See the full results table on the official page, including performance metrics such as Samples/s and Latency.

Significance and Outlook

MLPerf Client v1.0 fills the gap in client AI benchmarks, driving SoC vendors to optimize generative AI deployment on devices like phones and tablets. Future versions will expand with more models and scenarios, promoting standardization in AI edge computing.