Storage 2 Checkpointing

Introduction

In the era of large-scale AI training, the checkpointing mechanism is a key technology to ensure training stability and fault tolerance. It allows the system to quickly restore model states when training is interrupted, avoiding the huge loss of starting from scratch. The Storage 2 benchmark launched by MLCommons is specifically optimized for checkpointing testing, led by LMSYS Org, aiming to provide standardized evaluation for storage systems.

Storage 2 Benchmark Overview

Storage 2 is the second generation of the MLPerf Storage benchmark, expanding support for modern AI workloads, including Transformer-based model training. The checkpointing test simulates real-world scenarios, such as periodically saving model weights, optimizer states, and random seeds during distributed multi-node training.

• Core Metrics: Checkpoint Save Time, Restore Time, and Total Throughput.
• Test Scale: From single node to clusters with hundreds of GPUs, supporting ResNet-50, BERT, and GPT-like models.

Test Configuration

The benchmark uses a standard hardware stack: NVMe SSD (e.g., Samsung PM1733), InfiniBand network (200Gbps+), and integrates the Slurm scheduler. Scenario types include:

• Synchronous checkpointing: All nodes write simultaneously.
• Asynchronous checkpointing: Nodes save independently to reduce blocking.
• Incremental checkpointing: Only the changed parts are saved to optimize bandwidth.

Key Results and Analysis

Under a typical configuration of 8 nodes × 8 GPUs, top-tier systems save a 1TB checkpoint in less than 5 minutes, with restore times controlled within 2 minutes. Compared to Storage 1, throughput has increased by 30%, thanks to optimizations in parallel file systems such as Lustre and BeeGFS.

Bottleneck analysis shows that network latency and metadata operations are the main challenges, and it is recommended to use RDMA and hierarchical storage.

Conclusion and Outlook

The Storage 2 Checkpointing benchmark provides valuable insights for AI infrastructure vendors, driving the evolution of storage systems toward exascale training. Future versions will integrate more LLM scenarios, and the community is welcome to contribute results to the MLCommons official website.