Bringing Text-to-Video to MLPerf Inference v6.0

Introduction

MLCommons^Ⓡ MLPerf^Ⓡ Inference benchmark suite is an industry standard for measuring performance of machine learning (ML) and artificial intelligence (AI) workloads, covering areas such as vision, speech, and natural language processing. The suite selects workloads that represent the forefront of the industry, not only providing consumers with critical information for deployment and budget decisions but also helping vendors optimize workloads under practical constraints.

Over the past year, video generation models such as OpenAI Sora2 have developed rapidly, shifting from hobbyist generation of quirky videos to core tools in professional creative workflows. To this end, the MLPerf Text-to-Video task group was established to include dedicated video generation benchmarks in the MLPerf suite.

Model Selection

The benchmark uses the Wan2.2-T2V-A14B-Diffusers model released by Alibaba in July 2025, which was one of the best open-weight models on the Text-to-Video leaderboard at that time. This model is fully open-sourced under the Apache 2.0 license and can be run via Huggingface Diffusers.

The Wan2.2 model consists of a three-part pipeline:

1. Google's UMT5 XXL text encoder for encoding prompts.
2. The Wan2.2 A14B diffusion Transformer for generating video latent representations.
3. The Wan2.2 VAE decoder for decoding the latent video into a sequence of frames.

A key feature of Wan2.2-T2V-A14B-Diffusers is the mixture-of-experts (MoE) architecture, but unlike standard MoE with a gated network, it sequentially activates two experts: a "high-noise expert" used in the early denoising stage, switching to a "low-noise expert" in the later stage.

Interestingly, most video generation models do not generate frames one by one; instead, they denoise the entire video latent representation simultaneously. For example, Wan2.2's latent representation covers 4 frames of 32×32 pixels. Generating a 5-second 720p 16fps video requires a sequence length of 19,320, resulting in a compute-intensive workload.

Performance Metrics

When designing the benchmark, the biggest challenge was selecting performance metrics. Text-to-video tasks are computationally expensive; a single query often takes minutes.

To ensure feasibility and cutting-edge relevance, we imposed the following constraints:

• Configuration: Video duration 5 seconds, fixed resolution 720p, i.e., generating 81 frames of 720×1280 images (16fps).
• Runtime target: Dataset reduced to 100/248 samples for performance mode (accuracy mode retains 248 samples).

Replacing Server Scenario with SingleStream

A major change in the benchmark is replacing the Server scenario with SingleStream to measure latency. Generating a single video requires significant computation, often taking minutes.

The Server scenario assumes near-real-time processing, but in practice it leads to system overload, with most requests experiencing excessive wait times that fail to accurately reflect hardware performance. Therefore, we adopted SingleStream, which only measures processing time and ignores wait time.

Dataset, Accuracy Metrics, and Task Selection

We selected VBench as the official dataset and accuracy framework, prioritizing licensing, robustness, and ease of use after comparative analysis.

Initial evaluations included OpenVid-1M, VidGen-1M, WebVid-10M, and ActivityNet. The first filter was commercial availability:

• VidGen-1M and WebVid-10M were eliminated due to non-commercial licenses.
• OpenVid-1M had a suitable license (CC BY 4.0), but offered only a dataset without an evaluation framework.

VBench stood out as a complete solution, providing a diverse prompt set and a pre-validated scoring suite, including 16-dimensional quality metrics such as Subject Consistency, Motion Smoothness, and Aesthetic Quality; approximately 950 standardized prompts; determinism (stable across hardware under fixed seed, e.g., NVIDIA, AMD); and wide adoption (e.g., reported in the Wan model).

The default VBench configuration takes over 80 hours. To align with MLPerf, we made adjustments:

• Dataset subset: Focused on 6 key metrics: Subject Consistency, Background Consistency, Motion Smoothness, Dynamic Degree, Appearance Style, and Scene, ensuring high discriminative power and removing redundancy.
• Reduced scale: To 248 samples, balancing accurate verification with runtime.

VBench is the only commercially viable, legally cleared, and methodologically complete framework that can be adapted to MLPerf constraints.

Reference Implementation

To ensure fairness and reproducibility, the reference implementation is based on a standardized open-source foundation:

• Model architecture: Wan2.2-T2V-A14B-Diffusers (hosted by Wan-AI), a 14B-parameter diffusion Transformer.
• Precision and computation: BF16 (BFloat16) precision.
  • Reference accuracy: 70.48 (VBench).
  • Minimum threshold: 69.77 (99% of reference).
• Generation pipeline: Based on Hugging Face Diffusers.
  • Input: Text prompt + fixed latent tensor (ensures determinism).
  • Scheduler: UniPCMultistepScheduler (Wan2.2 default).