Mlperf Tiny V1 3 Tech

MLPerf Tiny v1.3 Overview

MLPerf Tiny v1.3 is the latest edge AI benchmark version from MLCommons, designed for resource-constrained devices such as microcontrollers (MCUs) and low-power edge processors. This benchmark aims to standardize TinyML model evaluations, helping developers compare the performance of different hardware and software stacks in terms of accuracy, latency, and power consumption.

Compared to the previous version, v1.3 introduces two new benchmarks: Image Classification (IC) and Visual Wake Words (VWW), and optimizes the existing Keyword Spotting (KWS) and Anomaly Detection (AD). These updates reflect the real-world demands of edge AI in smart homes, wearables, and IoT.

Details of New Benchmarks

Image Classification (IC)

The IC benchmark uses the CIFAR-10 dataset, with models based on the MobileNetV2 architecture. The task requires achieving high accuracy on 224x224 resolution images while keeping inference latency within 30ms. Evaluation metrics include Top-1 accuracy and throughput per watt, suitable for visual edge applications.

• Dataset: CIFAR-10 (60,000 32x32 color images)
• Model: MobileNetV2 (quantized to INT8)
• Performance Target: Accuracy > 70%, Latency < 30ms

Visual Wake Words (VWW)

VWW simulates device wake-up scenarios, using a custom dataset of 100,000 images (with/without wake gestures). The model adopts a lightweight CNN to detect wake words like gestures. It emphasizes low power consumption, suitable for always-on devices.

• Dataset: VWW v1.0 (balanced positive and negative samples)
• Model: EfficientNet-Lite (optimized version)
• Metrics: Recall > 90%, Power consumption < 1mJ/inference

Updates to Optimized Benchmarks

Keyword Spotting (KWS)

Based on the Google Speech Commands v2 dataset, it introduces multi-keyword support (e.g., 10 command classes). The model uses TC-ResNet, with accuracy target raised to above 95%.

Anomaly Detection (AD)

Targeting industrial sensor data, using the NAB dataset. The model is an Autoencoder variant, with anomaly detection threshold F1 score > 0.85.

Evaluation Rules and Framework

All benchmarks adopt the Closed Division rules, with support for custom implementations (Open Division). Inference frameworks include TensorFlow Lite Micro, TVM, and SGLang. Submissions must report offline accuracy, real-time factor (RTF), and hardware specifications. The first submission deadline is Q1 2025.

• Hardware Range: MCUs (< 1MB RAM), edge SoCs
• Quantization Support: INT8/FP16
• Power Measurement: Standard power tracing

Significance and Outlook

MLPerf Tiny v1.3 advances the maturity of the TinyML ecosystem, with initial results expected to be released in 2025. Developers can download the benchmark suite from the MLCommons official website to start optimizing deployments.