GPT-OSS 20B: A Sparse MoE Pretraining Benchmark for MLPerf Training v6.0

Large Language Model (LLM) pretraining typically requires enormous computational resources. In the past, pretraining tasks in the MLPerf™ Training benchmark suite, such as Llama 3.1 405B and Llama 3.1 8B, were primarily designed for dense models and often demanded large-scale multi-node infrastructure, creating a high barrier for many organizations wishing to participate in the benchmark.

To lower the barrier to entry while covering the more representative sparse architectures of today, the MLPerf Training Working Group, driven by a task force composed of members from AMD, NVIDIA, and NIT University, introduced a new pretraining benchmark: GPT-OSS 20B. This is a modern Mixture-of-Experts (MoE) benchmark capable of evaluating the complex routing logic and sparse computation patterns common in MoE architectures, and it can run on hardware configurations as small as a single 8-GPU node.

Why GPT-OSS 20B

The task force believes GPT-OSS 20B is a suitable candidate for a small-scale MoE pretraining benchmark for three main reasons:

• Sparse Computation Efficiency: GPT-OSS 20B has approximately 21B total parameters but uses an MoE design, activating only 3.6B parameters per token. This allows the model to retain a large knowledge capacity while maintaining computational costs close to those of a smaller dense model.
• Training from Scratch: To simplify the benchmark process and avoid the overhead of downloading multi-GB checkpoints, GPT-OSS 20B is trained from random weights. This makes it a pure test of the system's ability to optimize sparse models from an initial state.
• Reusable Reference Implementation: The reference code is built on AMD's Primus framework. Primus is a new general-purpose training library that supports AMD and NVIDIA backends. Primary validation work was done on AMD Instinct™ MI355X and NVIDIA B200 systems.

Dataset and Tokenization

GPT-OSS 20B uses the C4 (Colossal Cleaned Common Crawl) dataset, employing the same pre-tokenized subset and Llama-3 compatible tokenizer as the Llama 3.1 8B benchmark. This reduces additional configuration costs for submitters who have already prepared data for other MLPerf Training benchmarks.

The training data consists of approximately 80 GB of pre-shuffled C4 shards, hosted on MLCommons™ storage. To ensure stability, the benchmark requires evaluation using the first 1,024 samples of the validation set after every 12,288 training samples (768 iterations at GBS=16).

Statistical Variance: The Core Challenge of Fair Benchmarks

One of the core goals of benchmarking is fairness. In large-scale training, fairness is typically measured by the Coefficient of Variation (CV), which is the ratio of standard deviation to mean: CV = σ / μ × 100%.

Why High CV Undermines Fairness

If a benchmark has high CV, it means there is significant "statistical noise" in the results. For example, one run might require 170k samples to converge, while another might need 250k samples due to randomness, so the results no longer clearly reflect the strengths of the hardware or software stack. For a credible industry standard, users must be able to reproduce results; excessive variance makes it difficult to determine whether a performance improvement comes from an engineering breakthrough or mere luck.

Reducing CV from ~15% to Below 5%

A common way to reduce CV is to start from a pretrained checkpoint that has already passed the early unstable phase. However, the task force wanted to keep the benchmark simple and avoid requiring submitters to download large checkpoints. Therefore, they reduced the CV from approximately 15% to below 5% through three key technical interventions.

1. Eliminating Validation Set Noise

In early tests, the team observed large and unrepresentative spikes in evaluation loss.

• Problem Discovery: The validation set was being reshuffled at each evaluation. For sparse MoE models, whose routing is highly sensitive to input distribution, this introduced artificial jitter.
• Solution: The benchmark now mandates evaluation using a static, unshuffled set of the first 1,024 samples from the C4 validation set, ensuring that every run and every evaluation step sees exactly the same test input.

2. Stabilizing the Optimizer

Many modern models use Adam epsilon (ε) of 10^-8, but the task force found that this setting caused excessive divergence when training a 20B-scale MoE from scratch. By aligning with the standard used by Llama 3.1 8B and setting ε to 10^-5, the team provided the necessary numerical stability for training, reducing the probability that an "unlucky" gradient update would damage sparse experts.

3. Standardizing Initialization

To ensure all participants start from the same "statistical energy," the task force strictly defined the weight initialization standard: init_method_std = 0.008. This avoids additional variance caused by different initial points in the high-dimensional loss landscape.

Technical Configuration and Quality Metrics

The target accuracy for GPT-OSS 20B is a validation loss (log perplexity) of 3.34. This target comes from extensive sweeps on AMD MI355X and NVIDIA B200 hardware, representing a balance point that ensures stable convergence and reasonable runtime; with BFloat16 precision, convergence time is approximately 6.5 hours.

To facilitate optimization for different hardware, submitters are allowed to adjust three hyperparameters: global batch size, learning rate, and learning rate warmup.

Conclusion: Bringing MoE Pretraining into Standardized Benchmarking

GPT-OSS 20B formally brings MoE pretraining into the MLPerf Training benchmark suite. By identifying and eliminating sources of training variance—including validation set shuffling, optimizer instability, and initialization inconsistency—the task force provides a stable, high-fidelity benchmark while maintaining accessibility for submitters.

This means that results from MLPerf Training v6.0 will better reflect true hardware and software efficiency rather than random fluctuations. GPT-OSS 20B also provides the community with a standardized way to evaluate sparse pretraining performance alongside existing dense workloads. Its reference implementation is available on the MLCommons GitHub repository.

This article is categorized as MLPerf Training News. The authors include Sarthak Arora (AMD), Su-Ann Chong (AMD), Ravi Dwivedula (AMD), Miro Hodak (AMD), Michal Marcinkiewicz (NVIDIA), and Shriya Rishab (NVIDIA). For more MLCommons information, visit MLCommons.org.