Training Flux1

Flux.1 is an open-source text-to-image generation model launched by Black Forest Labs, benchmarked in collaboration between LMSYS Org and MLCommons. This report focuses on its training details, revealing the full process from data preparation to deployment optimization.

Model Architecture Overview

Flux.1 is based on the DiT (Diffusion Transformer) architecture, with a total parameter count of 12B. It introduces Rotary Positional Embeddings and parallel attention mechanisms, improving generation quality and efficiency. Compared to Stable Diffusion, Flux.1 performs excellently on images with resolutions up to 2M pixels.

Training Data and Strategy

• Dataset: Uses 10B+ high-quality image-text pairs, strictly filtered and deduplicated to ensure diversity and safety.
• Pre-training Phase: Employs large-scale distributed training, achieving a peak throughput of 1000+ samples/second, with inference accelerated using SGLang.
• Fine-tuning: LoRA adapters require only 1% parameter updates, reducing training time by 80%.

Key Optimization Techniques

• FP8 mixed-precision training reduces memory usage by 50%.
• Gradient checkpointing and activation recomputation, suitable for single-machine 8x H100 configuration.
• Custom noise scheduling improves FID scores to below 2.5.

Benchmark Performance

In the Chatbot Arena-style image generation ranking, Flux.1 [dev] achieves an Elo Rating of 1285, while the [schnell] version is optimized for 4-step inference, delivering a 10x speedup. On MS COCO, mAP@0.5 reaches 0.65, surpassing SD3.

Training Cost and Reproducibility

Complete training requires approximately 100K H100 GPU hours, costing around $2 million. MLCommons provides open-source code and configuration files to support community reproduction. Future plans include integrating more multimodal data.

Flux.1's training paradigm sets a new benchmark for open-source AI, driving image generation toward efficiency and accessibility.