DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0

Motivation and Architectural Relevance

As large language model (LLM) development increasingly shifts toward sparse computation, the benchmarks used to evaluate training performance must also evolve accordingly. MLPerf™ Training v6.0 adds a new large-scale pre-training benchmark based on DeepSeek-V3, which adopts a Mixture-of-Experts (MoE) architecture with 671B total parameters, of which 37B are activated per token.

This benchmark captures key innovations that have become standard in the industry, including Multi-head Latent Attention (MLA) and auxiliary-loss-free load balancing.

Technical Architecture and Implementation

DeepSeek-V3 introduces several unique computational patterns that sharply contrast with dense models in the current suite (e.g., Llama 3.1):

• Multi-head Latent Attention (MLA): Unlike standard Multi-Head Attention (MHA), MLA reduces memory bandwidth bottlenecks in training and inference by applying low-rank joint compression to the Key-Value (KV) cache.
• Fine-Grained Expert Segmentation: Each expert’s feed-forward network (FFN) is split into m smaller experts. While a typical MoE might use top-2 routing across 16 experts, DeepSeek-V3 scales to 160 routed experts plus a shared expert to capture common knowledge.
• Multi-Token Prediction (MTP): The model is trained with a 2-token prediction objective. This requires a shared backbone and two dedicated output heads, increasing the computation-to-memory ratio during backpropagation.
• Auxiliary-Loss-Free Load Balancing: To avoid routing collapse without incurring high-overhead auxiliary losses, this method dynamically adjusts each expert’s bias term based on real-time load.

Benchmark Definition and Reference Setup

The task is defined as LLM pre-training using the Mixture-of-Experts objective.

Dataset and Tokenizer

• Dataset: C4 (Colossal Clean Crawled Corpus)
• Tokenizer: Llama-3 compatible tokenizer (128k vocabulary)
• Sequence length: 4,096 tokens

Convergence and Checkpoints

The task group found that MoE models experience token imbalance in the early stages of training. Since the benchmark captures only a small portion of full training, this imbalance occupies approximately 50% of the benchmark runtime, which does not represent steady-state MoE training. To ensure the benchmark measures steady-state hardware efficiency, the task group adopted a warm-start approach.

Because the benchmark uses the Llama 8B tokenizer instead of the original DeepSeek tokenizer, initializing from Hugging Face checkpoints results in significant token imbalance. To address this, the task group fine-tuned the checkpoint for 50 steps so that the expert token distribution closely matches that of the original DeepSeek tokenizer (as shown in Figure 1). The resulting checkpoint is provided in HuggingFace format and hosted by MLCommons. This approach ensures that over 98% of the benchmark run operates in a balanced expert state, reflecting long-term training dynamics.

(Figure 1)

Global Batch Size (GBS) Selection

This benchmark requires a GBS of at least 15,360. Although internal tests show the model maintains computational efficiency at lower batch sizes (e.g., 512, 1k, or 2k), the task group set a higher lower bound for three reasons:

• Representativeness: The original DeepSeek-V3 pre-training employed a batch size scheduling strategy, peaking at 15,360. To represent the true large-scale pre-training described in the paper, the benchmark targets the 15k–18k range.
• Benchmark Fairness: Setting a minimum GBS of 15k ensures that all submitters compete fairly, avoiding “hero runs” at small batch sizes that do not reflect production-scale MoE training.
• Convergence Scaling: The task group established a square root scaling rule to maintain learning rate stability at large batch sizes: LR(GBS) = 2.4×10^{-5} × √(GBS / 16384)

Engineering Challenges and Validation

The development of the reference implementation (using NVIDIA NeMo Megatron-bridge) highlighted key requirements for convergence:

• Expert Parallelism (EP): For a single layer, routed experts are evenly deployed across 64 GPUs (8 nodes). A node-constrained routing is implemented, sending each token to at most M=4 nodes.
• Memory Management: Expert capacity must not be limited to avoid out-of-memory (OOM) errors caused by token imbalance in the initial steps after checkpoint loading, ensuring the benchmark represents real training conditions.
• Target Metric: The benchmark targets a cross-entropy validation loss of 3.6, with a coefficient of variation (CV) of 1.5% (as shown in Figure 2).

(Figure 2)

Conclusion

This benchmark provides a standardized platform for evaluating production-scale training efficiency of leading open-source MoE models. By clearly defining convergence, batch size, and expert parallelism requirements, the DeepSeek-V3 benchmark ensures that MLPerf Training continues to reflect the state of contemporary AI infrastructure.

The reference implementation is available in the MLCommons GitHub repository, and the task group welcomes community submissions and feedback. For more information about MLCommons and membership details, please visit MLCommons.org.

Category

MLPerf Training News

Authors

Denys Fridman (NVIDIA) Michal Marcinkiewicz (NVIDIA) Shriya Rishab (NVIDIA) Qinwen Xu (Google) Parmita Mehta (Google)