MLCommons released the Whisper Inference v5.1 benchmark in September 2025, the latest standardized evaluation of inference performance for OpenAI Whisper speech-to-text models. The benchmark aims to provide fair, comparable performance data for the industry, helping developers optimize deployment.
Key Updates and Model Support
The v5.1 version introduces the Whisper large-v3 model, while retaining variants such as tiny, base, small, medium, large-v1, and large-v2. The dataset is expanded to Multilingual LibriSpeech, containing English and multilingual audio with a total duration exceeding 1000 hours.
Key Metrics
- RTF (Real-Time Factor): Ratio of inference time to audio duration, lower is better.
- WER (Word Error Rate): Measures accuracy.
- Primary score: RTF @ target WER (e.g., RTF@WER20%), balancing speed and accuracy.
- Energy efficiency metric: RTF × Power (Watts).
Test Hardware and Frameworks
Support for CPUs (x86, ARM, e.g., Intel Xeon, AMD EPYC, Apple M series), GPUs (NVIDIA A100/H100, AMD MI300X, Intel Gaudi), and edge devices. Frameworks include ONNX Runtime, TensorRT, SGLang, and others.
Top Performance Results
Server-side top submissions:
| Hardware | Model | RTF@WER20% | Submitter |
|---|---|---|---|
| NVIDIA H100 SXM | large-v3 | 0.008 | NVIDIA |
| AMD MI300X | large-v3 | 0.012 | AMD |
| Intel Xeon 6 | medium | 0.15 | Intel |
Edge highlight: Apple M3 Max achieves 0.05 for RTF@WER20%, with best energy efficiency.
Leaderboard and Submission Guidelines
Full results available at MLCommons official website. Developers can submit via the MLPerf Inference v5.1 rules, emphasizing end-to-end performance, including pre-processing and post-processing.
This benchmark highlights the advantages of GPUs in high-accuracy STT tasks while driving low-power optimization, accelerating the deployment of AI voice applications.
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接