DLRMv4: The Next-Generation Recommendation MLPerf Training Benchmark 

Recommender systems have long been an important part of MLPerf Training. Past DLRM benchmarks primarily measured how long a system takes to train a personalized recommendation model to a specified quality target. The current version, DLRMv2, targets scenarios such as e-commerce and streaming media, modeling user behavior and item features and using deep learning to capture interactions between the two, thereby producing more relevant recommendations.

However, with the development of sequence modeling and LLM-related technologies, hyperscale production recommender systems are undergoing an architectural shift: instead of compressing user behavior into a small set of aggregated dense features, they represent a user's recent interaction history as an event-by-event token sequence. Such models continue to improve as compute and parameter scale grow, whereas earlier feature-interaction-based designs tend to plateau in returns.

MLPerf previously introduced an HSTU-based recommendation benchmark in its inference suite, but the training suite still lacked a corresponding workload. DLRMv4 is intended to fill this gap: it replaces the traditional feature-interaction stack with HSTU and introduces the new Yambda-5B dataset, scaling real multi-behavior music interaction data to a production-grade embedding footprint of about 560 GB.

From DLRMv2 to DLRMv4: Why Change the Architecture

DLRMv4 aims to make MLPerf Training more accurately reflect changes in today's production-grade DLRM workloads. On one hand, models need to perform attention over longer user histories, significantly increasing computational complexity; on the other hand, an embedding table of about 560 GB also requires sharding across multiple modern accelerators. This makes DLRMv4 not only a model architecture update but also a new stress test of system training capability.

Model Architecture: HSTU Replaces the Traditional Feature-Interaction Stack

Figure 1: Comparison of the traditional DLRM architecture (a) and the HSTU architecture (b).

Traditional DLRMv2 represents an "encode first, then interact" production recommendation paradigm. Input data comes in various forms: numerical features are processed by a bottom MLP, and categorical IDs are converted into vectors through embedding lookup. These representations are then fed into a dedicated feature-interaction module, such as a cross network, factorization machine, or similar structure, and finally a task head outputs the prediction.

In this design, the core modeling capability is concentrated in the interaction module, but it cannot directly see the raw behavior stream. The model usually processes manually designed and pre-aggregated inputs, such as decayed counters, ratio features, and pooled embeddings.

HSTU removes this division of labor. It first serializes heterogeneous feature spaces into a unified time series: categorical features and slowly changing contextual features are merged into a user's chronologically ordered item and action history, which is then processed end-to-end by a stack of identical transduction layers.

Within each HSTU layer, the three stages that DLRM split apart are folded into a single module: a single projection generates the internal representation needed for that layer; pointwise attention introduces bias based on the positional and temporal distance between two events and aggregates sequence information; and gated transformation allows the attention-aggregated features to interact directly, taking on the role of the traditional cross network and feed-forward block.

This design converges the model's main scaling dimensions to three: depth, width, and sequence length. HSTU-related work shows that between compute and quality, it can maintain a power-law relationship across three orders of magnitude in scale; by contrast, the traditional encode-then-interaction baseline is more likely to saturate over the same range.

How a Training Sample Is Processed by HSTU

Consider scoring a candidate item for a user on a content platform: the system first organizes that user's historical interactions into a chronologically interleaved stream of items and actions. Behavior types can include multiple signals such as actual listening, explicit likes, and skips, and the stream is truncated to a fixed budget of several thousand events.

Each event becomes a token carrying information such as "what the user saw or listened to," "what action the user took," and "when the event occurred." As a result, negative feedback such as a skip is not averaged into some counter, but is preserved in the sequence as a first-class event.

Slowly changing context, such as user identity and precomputed cross features, is compressed into a small number of prefix tokens placed at the beginning of the same sequence; the candidate item to be scored is appended to the end of the sequence. In this way, the model can compare the candidate content with the full history in a single forward pass, without an additional target-aware pooling step.

Finally, the HSTU stack processes this entire causal sequence, and the representation at the candidate position is fed into a small head that outputs an engagement prediction. Jointly modeling history, context, and the candidate target in the same causal block is the key reason HSTU is closer to production recommender systems while also posing a greater challenge to system training capability. This configuration is also consistent with the model category of the MLPerf DLRMv3 inference benchmark.

Dataset: Yambda-5B Brings Real Multi-Behavior Timelines

DLRMv4 uses Yambda-5B for training. Yambda-5B was publicly released by Yandex Music and is one of the largest open recommendation datasets containing real users' multi-behavior interactions. The benchmark uses the full 5B-event version.

Table 1: Core attributes of the Yambda-5B dataset.

Compared with the Criteo 1TB dataset used by DLRMv2, Yambda is better suited to HSTU-style models. The reason is that HSTU requires real per-user timelines: each interaction records the user, item, action type, and time, so each user's behavior history can be reconstructed in chronological order and fed into the transducer as a sequence. The approximately 4.79B interactions are also enough to keep a full training run under sustained load.

Table 2: Comparison of Yambda-5B and Criteo 1TB (DLRMv2).

Feature Expansion: Embedding Table Grows to About 560 GB

The original Yambda contains four types of native sparse features: item, artist, album, and uid. DLRMv4 further adds cross-product hashed tables. Cross features are a common practice in industrial recommender systems, improving ranking quality by explicitly representing feature combinations such as user × artist and item × hour-of-day. Recent research also shows that the more memory allocated to cross features, the more their benefits may continue to scale.

After introducing these features, DLRMv4 is closer to real production models, while at 512-dimensional embeddings and fp32 precision, it pushes the embedding table size to about 560 GiB.

Table 3: Feature expansion of Yambda-5B through cross-product feature tables.

Why Data Distribution Also Matters

Real recommendation traffic is usually highly skewed: a small number of popular items contribute a large share of interactions, while long-tail content occupies a huge item set. If a recommendation benchmark is to represent production systems, it must have similar data distribution characteristics. The original article notes that MLCommons analyzed Yambda-5B to confirm whether it matches the skewed traffic characteristics of such real recommendation scenarios.

Overall, the significance of DLRMv4 is not just replacing a model or dataset. It advances the recommendation workload in MLPerf Training from traditional feature engineering and explicit interaction modules to a new stage focused on long user histories, sequential behavior modeling, and very large embedding sharding. For training systems, this means simultaneously facing more complex attention computation, a larger embedding footprint, and data patterns that are closer to hyperscale production recommender systems.