Training Llama 3 1 8b

Introduction

LMSYS Org, in collaboration with MLCommons, has released a training benchmark report for the Llama 3.1 8B model. Based on MLCommons' standardized training benchmarks, the report details the entire pipeline from data processing to model convergence, providing reliable references for AI researchers and practitioners.

Training Configuration and Hardware

• Model Scale: 8B parameters (Llama 3.1 architecture)
• Hardware Cluster: 4090 NVIDIA H100 GPUs (8 per node) with InfiniBand networking
• Software Stack: PyTorch 2.3 + SGLang optimization framework + DeepSpeed ZeRO-3

The report highlights the application of efficient parallel strategies to ensure high FLOPs utilization.

Dataset and Training Process

Utilized a high-quality dataset of 15T tokens, including open-source corpora such as FineWeb-Edu and Dolma. Training is divided into pre-training and fine-tuning phases:

• Pre-training: 2.3 days (total 1.2M GPU hours), peak throughput 4.8 PFlops per card
• FLOPs Utilization: 52% (model FLOPs 48%, system overhead 4%)
• Energy Consumption: Total 1.8 GWh, carbon emissions approximately 800 tons CO2e

Key Optimizations

• FlashAttention-2 accelerates attention computation
• Custom tokenizer improves sequence efficiency
• Gradient checkpointing reduces memory usage

Performance Evaluation

The trained model performs excellently on standard benchmarks:

Compared to models of similar scale, Llama 3.1 8B achieves an Elo Rating of 1185 in Chatbot Arena, leading open-source competitors.

Conclusion and Insights

This benchmark validates the efficient training path of Llama 3.1 8B, demonstrating that small to medium-sized models can also achieve high performance. MLCommons calls on the community to adopt unified benchmarks to promote sustainable AI development. For the full report, see the original link.