DLRMv3: Generative recommendation benchmark in MLPerf Inference

Introduction

Following the neural scaling laws, computational scaling has greatly reduced the need for manual feature engineering in natural language processing (NLP) and computer vision, shifting to large-scale attention Transformer models that automatically learn rich representations from data [1][2]. This trend is profoundly transforming deep learning recommendation systems, which traditionally relied on multilayer perceptrons (MLPs), graph neural networks (GNNs), and embedding tables [3][4][5]. Recently, large sequential and generative models have been successfully deployed on online content recommendation platforms, significantly improving model quality [5][7][8][9][10][11][12][13]. Given the global scale and importance of recommendation systems [6], incorporating such large-scale sequential recommendation models into the MLPerf Inference benchmark suite helps drive continued infrastructure development.

We introduce DLRMv3, the first sequential recommendation inference benchmark in the MLPerf DLRM series. DLRMv3 builds a ranking model based on the HSTU architecture [5], capturing the dominant computational patterns of modern recommendation workloads: long input sequences, attention-intensive computation, and large embedding tables. Compared to existing DLRM benchmarks (DLRMv2 [14]), DLRMv3 increases model size by 20× (from 50GB to 1TB) and per-candidate computation by 6,500× (from 40M FLOP to 260 GFLOP) in just three years, aligning with contemporary production-grade recommendation loads and highlighting the surge in computational demands. This high-computation regime stems from the scaling behavior reported for HSTU, where higher model computation leads to production quality improvements, facilitating evaluation of real resource burden and accuracy trade-offs.

Task Selection

Modern recommendation systems are typically deployed as multi-stage pipelines, separating candidate retrieval from ranking, sometimes with additional re-ranking or business logic post-processing [15][16]. In typical designs, retrieval models first select a small set of relevant items from a massive corpus, optimizing for high recall, coverage, and strict latency/memory constraints [15][7]. Downstream ranking models then score these candidates using richer features and more expressive architectures, optimizing fine-grained user interaction metrics (e.g., click-through rate CTR, watch time, satisfaction) under slightly looser but still critical production latency/throughput constraints [15][16][18]. This phased design has become the standard for large-scale industrial systems, including web, video, and social content recommendations.

DLRMv3 focuses on the ranking stage of the pipeline. Ranking models typically dominate the overall ML compute budget of production recommendation systems and are the focus of model architecture innovation (e.g., attention-based sequential models and large embedding tables), making them particularly suitable for hardware and system benchmarking. Focusing on ranking also continues the CTR prediction objective of previous MLPerf DLRM benchmarks.

Formally, given user interaction history (e.g., a sequence of previously viewed/interacted items) and a candidate item, the DLRMv3 model predicts the probability of a desired outcome, such as a click, like, or view. This probability prediction task directly aligns with the binary CTR-style outcome modeling of earlier DLRM benchmarks.

Model Selection

Figure 1. Architectures of different DLRM models.

We introduce an HSTU-based architecture as the third-generation deep learning recommendation benchmark (DLRMv3) for MLPerf. In the evolution of DLRM, DLRMv1 consists of MLPs, embedding tables, and simple dot-product feature interactions; DLRMv2 adds a deep cross network (DCN) component for richer explicit feature crossing. DLRMv3 introduces new sequential feature transformation, interaction, and extraction components based on the Hierarchical Sequential Transduction Unit (HSTU) [5], retaining a single large embedding table and a top MLP for final prediction (Figure 1(c)).

The HSTU-style architecture has proven effective in production systems for modeling long user interaction histories, outperforming traditional MLP/DCN models at comparable or higher computational costs and improving recommendation quality. Its computational characteristics—long sequences, attention intensity, and large embedding tables—increasingly dominate modern recommendation inference, making HSTU a representative and forward-looking choice for system-level benchmarking.

The table below compares configurations across DLRM model generations. The 260 GFLOP calculation formula is 2 * layers * (UIH_length * UIH_length * EmbDim / 2 + UIH_length * EmbDim * EmbDim * 4 + UIH_length * EmbDim * EmbDim * 3), covering attention FLOPs and pre-/post-attention GEMMs. This "per-candidate" 260 GFLOP is effectively normalized: in a typical ranking request, the HSTU encoder processes the shared user interaction history (UIH) sequence only once, and its output is reused for scoring a candidate set (2K candidates in DLRMv3), so the UIH encoding cost is amortized rather than repeated 2K times. Additionally, DLRMv3 adopts a streaming time-series setting, allowing reuse of UIH-related KV states across consecutive timestamps for the same user, avoiding recomputation of UIH encoding and reducing redundant dense computation by approximately 80–90% in steady state.

To better align the MLPerf Inference benchmark goals with practical constraints, we introduce two intentional deviations from the original HSTU paper setup. These changes aim for hardware friendliness and broad implementability while capturing the key computational patterns of sequential recommendation models.

Action Embedding Preprocessing: The original HSTU uses context-interleaved action embeddings, where context features and user actions are interleaved into the input sequence to provide richer contextual learning of behavior-item dependencies. However, interleaving doubles the effective sequence length, significantly increasing computational cost. The DLRMv3 benchmark omits action interleaving, using a simplified input sequence that directly combines action embeddings with context embeddings without extending length. Reasons: 1) The synthetic benchmark dataset (detailed in the next section) is used only for performance measurement and lacks sufficiently rich action features to support the added complexity and doubled length; 2) The non-interleaved option provides a more balanced accuracy-efficiency trade-off, suitable for standardized inference benchmarks.

Temporal/Positional Encoding: The original HSTU uses relative position biases (Mask(SiLU(QKT)+bias)V) to capture relative temporal relationships between tokens, improving accuracy. DLRMv3 instead uses absolute temporal/positional encoding, adding position-dependent biases to query, key, and value vectors, with attention computed as Mask(SiLU(QKT))V. Absolute biases are chosen because relative biases introduce kernel optimization challenges and slow down attention computation on many processors, whereas absolute encoding is widely supported, easier to optimize, and yields more predictable performance.

Dataset Selection

Unlike DLRMv1/v2, DLRMv3 formulates recommendation as a sequential transduction task over long user interaction histories with an extremely large item set. To represent modern production workloads, the benchmark dataset must simultaneously satisfy: (1) a reasonably long user interaction history per request (thousands of events) to sufficiently activate sequential models and attention layers; (2) an extremely large item set consistent with DLRMv3's single large embedding table (hash size ~1 billion); (3) a streaming structure where user viewing and preferences evolve over time, allowing inference requests to be replayed chronologically.