Mlperfmobile Android

The MLCommons organization recently announced the latest benchmark results for MLPerf Mobile v4.0 for Android, a milestone update in the field of mobile AI performance evaluation. This benchmark focuses on the Android platform, simulating real user scenarios to help vendors optimize hardware and accelerate AI application deployment.

Benchmark Overview

MLPerf Mobile v4.0 for Android introduces seven core tasks covering key areas such as computer vision, natural language processing, and generative AI:

• Image Classification (ResNet50): Image classification using the ImageNet dataset.
• Object Detection (SSD-MobileNet): Object detection based on the COCO dataset.
• Super Resolution (EDSR): Image super-resolution to enhance low-resolution image quality.
• Speech Recognition (RNN-T): Speech recognition supporting real-time transcription.
• 3D-UNet: Medical image segmentation task.
• BERT Squad: Question answering task for evaluating NLP capabilities.
• Stable Diffusion: First introduction of text-to-image generation to test generative AI performance.

The tests emphasize low power consumption and high throughput, suitable for devices such as smartphones and tablets.

Top Performance

Submissions for this round came from vendors including Qualcomm, Samsung, MediaTek, and Google. The following are some highlights (using the Offline scenario as an example, unit: Samples/second):

• Qualcomm Snapdragon 8 Gen 3 achieved a peak throughput of 1,200+ on ResNet50, winning multiple tasks.
• Samsung Galaxy S24 Ultra (powered by Snapdragon 8 Gen 3) performed exceptionally well on the Stable Diffusion task, leading in generation speed.
• MediaTek Dimensity 9300+ closely followed in the speech recognition task.
• Google Tensor G3 demonstrated strong NLP capabilities on BERT Squad.

The full results table is available on the official page, supporting three scenarios: SingleStream, Server, and Offline.

Significance and Outlook

This update optimizes the benchmark framework, supports more NPU and GPU accelerators, and promotes AI standardization in the Android ecosystem. The inclusion of Stable Diffusion marks the penetration of generative AI into mobile devices. With the results publicly available, developers can better select hardware, which is expected to accelerate on-device AI innovation, such as real-time image generation and intelligent assistants.

MLPerf Mobile continues to lead the industry, driving efficient and sustainable mobile AI development.