Introducing the MLPerf End-to-End RAG Inference Benchmark

The MLCommons MLPerf Inference working group has launched the first end-to-end retrieval-augmented generation (RAG) inference benchmark. By generating answers from retrieved documents at query time rather than solely from model weights, RAG significantly reduces hallucinations while leveraging the latest proprietary knowledge, and has become one of the most common deployment approaches for language models.

Production RAG systems typically consist of multiple models, forming a complete pipeline that retrieves documents and reasons to generate answers. This benchmark fully measures this process and includes two workloads.

The Significance of End-to-End RAG

RAG is increasingly popular because it can derive answers from documents retrieved at query time. It reduces hallucinations, keeps answers up to date, and allows general-purpose models to leverage proprietary knowledge unseen during training.

Because RAG is a multi-component system, its performance stems from the synergy of components rather than a single model. Single-model LLM benchmarks cannot capture pipeline-level behavior, nor can they expose opportunities for joint multi-model optimization.

The benchmark also paves the way for agentic AI benchmarks, as its multi-model serving challenges serve as the starting point for future benchmarks.

Optimization opportunities include:

• Model placement: Mapping models of different sizes to different accelerators
• Co-residency: Sharing devices through memory partitioning
• Multi-stage scheduling: Enabling macro and micro batching across heterogeneous accelerators
• Prefix caching: Reusing shared prefixes across multi-hop queries to reduce computation

Dataset and Tasks

The benchmark is based on the FRAMES multi-hop query dataset, containing 824 queries, 2,515 Wikipedia articles, and approximately 107,000 paragraphs of 768 characters.

An example multi-hop query requires chained reasoning across three articles to arrive at an answer.

Figure 1: Example of multi-hop E2E RAG

Ingestion and Question-Answering Pipelines

The benchmark includes two separate pipelines: the ingestion pipeline (e2e-rag-db) builds the vector database in one pass, and the question-answering pipeline (e2e-rag-qna) performs end-to-end scoring.

Figure 2: E2E RAG ingestion and question-answering pipelines

The ingestion pipeline includes parsing, chunking, embedding, and FAISS HNSW indexing. The question-answering pipeline iteratively completes multi-hop reasoning through a query rewriter, embedder, retriever, reranker, document scorer, and sufficiency checker.