The Benchmark Behind the Next Wave of Ultra-Low-Power AI

Machine learning no longer lives only in data centers. Today, it has entered doorbell cameras, hearing aids, factory sensors, and battery-powered wearables. These devices typically have power budgets of just a few milliwatts yet need to respond in real time, so performance is no longer just about "running fast" — it must also be "power-efficient enough."

That raises a problem: How can you fairly compare the performance and efficiency of devices with vastly different form factors? In the same submission round, there could be a 60 MHz microcontroller, a RISC-V core with vector extensions, or a dedicated Neural Processing Unit.

MLPerf Tiny was designed exactly for this. Developed by the MLCommons Tiny Working Group in collaboration with EEMBC, its goal is to compare the machine learning capabilities of ultra-low-power systems using the same workloads, models, and measurement methods, while remaining architecture-agnostic.

The Rise of TinyML: Why It Matters

TinyML generally refers to machine learning models that can run on microcontrollers and resource-constrained devices, typically with fewer than 2 million parameters and power consumption ranging from sub-milliwatts to low milliwatts. This capability brings AI into scenarios that cloud inference and phone inference cannot easily cover.

By 2026, typical applications already include:

• Industrial vibration sensors that detect bearing wear before equipment failure;
• Agricultural sensors that run for months on a single charge;
• Hearing aids that suppress background noise locally;
• Always-on keyword detection in headphones;
• Patchable wearable devices that monitor ECG without uploading raw signals.

The advantage of local inference is straightforward: data stays where it is generated, resulting in lower latency, lower cost, better privacy, and longer battery life — critical for battery-powered devices.

Meanwhile, the software ecosystem has matured rapidly. TensorFlow Lite for Microcontrollers, ExecuTorch, Edge Impulse, STEdgeAI-Core, NXP eIQ, and vendor-specific compilers like AndesAIRE and RUHMI have all made it easier to deploy models on embedded hardware. But this also exposes a new challenge: different vendors often report results using different models, datasets, and test conditions, making it hard to fully interpret the numbers on their own.

The Benchmark Gap That Was Missing

Establishing a fair benchmark for ultra-low-power machine learning is not straightforward. Papers on MLPerf Tiny have pointed out several core challenges such benchmarks must face.

• Energy measurement must be fair. Power consumption varies greatly across devices, and vendor measurement methods differ. Some include peripherals, firmware startup, or I/O activity; others measure only the inference itself, which can significantly affect results.
• Must adapt to extremely small memory budgets. TinyML devices typically operate with memory in kilobytes, not gigabytes — resource constraints are about six orders of magnitude more severe than for phone ML. Reference models, test framework overhead, and quantization paths must all fit into limited space.
• Hardware and software are highly heterogeneous. Devices may be general-purpose MCUs, neural network processors, event-driven architectures, or even in-memory computing systems; each vendor often comes with its own toolchain. If the benchmark imposes too many restrictions, it will mask the true optimization capabilities of the systems under test.

Figure 1: Diversity at every layer of the Tiny Machine Learning Stack makes benchmark standardization more challenging.

In the past, embedded developers were not without benchmarks, but many did not answer the questions they truly cared about. CoreMark is a common standard for measuring Microcontroller Unit performance, but it is not designed for ML workloads. MLMark is closer to machine learning inference, but its reference models — such as ResNet-50, MobileNet, and SSD-MobileNet — target edge AI processors rather than severely resource-constrained microcontrollers; moreover, it does not treat energy as a core metric.

This is exactly the gap MLPerf Tiny aims to fill. In TinyML, speed and energy efficiency are not the same thing: a model that runs fast may also drain the battery quickly. Therefore, latency and energy per inference must be measured together.

MLPerf Tiny is driven by more than 50 organizations from industry and academia, including Harvard, Google, STMicroelectronics, Qualcomm, Syntiant, Renesas, Infineon, CERN, and Silicon Labs. The group spent about 18 months building the benchmark suite and released the first public results as v0.5 in June 2021.

What MLPerf Tiny Actually Measures

The question MLPerf Tiny answers is straightforward: if all devices run the same ML task, which hardware and software stack performs it most efficiently?

To do this, it evaluates three metrics: accuracy, latency, and energy per inference. Each task has a fixed quality target, so participating systems compare speed and energy at equivalent accuracy, rather than sacrificing accuracy for better numbers.

Among these, energy is the most distinctive metric in MLPerf Tiny. In TinyML scenarios, if a device takes only 10 ms per inference but consumes twice the energy of a competitor, it may not be the better choice — battery life is often the primary system constraint. MLPerf Tiny uses the EEMBC EnergyRunner test framework to perform calibrated power measurements on the device under test, enabling direct comparison of energy results across different chips.

MLPerf Tiny also emphasizes modular design. Optimizations can come from the chip, compiler, runtime, or the model itself, but the reference tasks remain fixed, ensuring all submissions are measured against the same ruler.

Take Visual Wake Words as an example. All submissions must determine whether a person is present in a 96×96 image and achieve at least 80% accuracy on the official test set. Because the problem, rules, and accuracy threshold are identical, results are comparable. For instance, ASYGN's ColibriNPU consumes just 22.2 microjoules per inference. At that energy level, a single CR2032 coin cell battery could theoretically support one inference per second for more than three years.

Real-World Scenarios for the Five Benchmarks

MLPerf Tiny included four tasks in its v0.5 release in 2021 and expanded to five in v1.3 in 2025. Each task corresponds to a common TinyML deployment pattern:

• Keyword Spotting (KWS): Uses a small DS-CNN trained on the Google Speech Commands dataset, representing wake-word and voice command detection in smart speakers, headphones, and hearing aids.
• Visual Wake Words (VWW): Uses a MobileNetV1 binary classifier for 96×96 images, simulating low-power "person present" detection in doorbells, security cameras, and occupancy sensors.
• Image Classification (IC): Runs a ResNet-style network on CIFAR-10, representing general low-resolution visual classification tasks on edge devices.
• Anomaly Detection (AD): Uses an autoencoder trained on ToyADMOS industrial machine sound data, representing predictive maintenance and equipment condition monitoring.
• Streaming Wake Word: Added in v1.3, employs a 1D depthwise-separable CNN to detect a target word in continuous audio streams. Unlike other benchmarks, it also tracks device performance in idle and always-listening states, better reflecting the real-world power stress of always-on voice devices.

What v1.4 Reveals: Edge AI Entering an Era of Energy Efficiency Competition

The significance of MLPerf Tiny v1.4 goes beyond releasing a new set of numbers — it further confirms the direction of ultra-low-power AI: future competition will center not only on model size and inference latency, but also on end-to-end energy efficiency, toolchain maturity, and how well hardware adapts to specific workloads.

For developers and buyers, the value of such a benchmark lies in reducing comparison costs. Facing vastly different platforms like MCUs, RISC-V, and NPUs, MLPerf Tiny uses unified models, data, accuracy targets, and energy measurements to enable discussions within the same context.

As TinyML enters more real-world scenarios, systems that can perform reliable inference at a few milliwatts — or even lower — will become the infrastructure for the next wave of edge AI applications. MLPerf Tiny provides the common language to measure that progress.