Agentic Inference for MLPerf Inference

Introduction: Inference Benchmarks Need to Keep Pace with Agentic Applications

The MLPerf Inference benchmark suite is evolving alongside changes in AI deployment. Early inference benchmarks mainly covered tasks like image classification, object detection, speech recognition, recommendation systems, and single-turn language generation. While these workloads remain important, they no longer represent the fastest-growing category of large language model usage in production: multi-turn agentic inference.

Consider the coding assistant as an example. It goes far beyond answering a single query. A code agent typically reads an issue, inspects files, executes commands, observes failures, modifies code, and iterates repeatedly. Similarly, a workflow agent collects customer information, invokes tools, interprets returned results, asks follow-up questions, and continues until the task is complete. The common thread is that the workload is no longer a set of independent requests—it is a trajectory, a sequence of interdependent turns, where each request includes the full conversation history up to that point.

This shift poses four specific challenges for model serving:

• Growing context: As the trajectory progresses, prefill overhead and KV-cache pressure increase.
• KV-cache reuse becomes a critical optimization: Cache reuse directly impacts performance and energy efficiency.
• Highly variable output length: Outputs may be brief tool calls or long reasoning chains.
• Dependence between turns: Throughput is no longer just the rate of independent requests, but the ability to advance a closed-loop task.

Figure 1: Illustration of a multi-turn agentic inference scenario, where a task undergoes multiple interdependent turns.

To cover this new workload, the Agentic Inference benchmark has been added to the MLPerf Endpoints framework. It preserves MLPerf's established measurement principles while defining key components tailored to this workload, including model selection, dataset composition, multi-turn workload generation, output validation, and constraints on optimizations such as prefix caching and speculative decoding.

Terminology

• Turn: A single client-initiated user request or tool request, and the model's response to that request.
• Trajectory: An ordered sequence of turns corresponding to one task or simulated user; subsequent turns include the accumulated conversation history.

Model Selection: Covering Different Architectures and Serving Behaviors

The benchmark requires LLMs capable of handling long contexts and reasoning, designed to stress typical serving behaviors in agentic applications: growing context, KV-cache reuse, variable output length, and strict multi-turn dependency. MLCommons has selected Kimi K2.6 and Qwen3.6-35B-A3B for this benchmark.

Kimi K2.6 offers leading coding capabilities and a model scale closer to state-of-the-art agentic systems; Qwen3.6-35B-A3B is more compact, using the novel Gated DeltaNet (GDN) architecture, and demonstrates outstanding coding performance at its scale. Together, they cover different serving behaviors, architectural choices, and speculative decoding paths. Note that the two models are evaluated separately using the same methodology and dataset; they are not mixed or combined into a single score.

Table 1: Model metadata and benchmark settings.

Dataset and Task Selection: Dual Scenarios for Coding and Workflows

The dataset for the Agentic Inference benchmark combines two agentic domains to trigger different infrastructure bottlenecks. It includes a total of 613 multi-turn trajectories, comprising 113 agentic coding trajectories and 500 agentic workflow trajectories. In the reference dataset, these trajectories contain 30,335 client-initiated turns and 30,328 assistant turns.

MLCommons emphasizes that using real-world traces for benchmarking more closely reflects production serving stack behavior, including the performance of speculative decoding under real multi-turn traffic and the effects of expert-rank balancing under actual workloads.

Figure 2: The Agentic Inference benchmark covers coding agent and workflow agent multi-turn trajectories.

Table 2: Dataset composition and primary stress points.

Agentic Coding: Software Engineering Trajectories from DeepSWE

The agentic coding traces come from the DeepSWE dataset by DataCurve (datacurve.ai). This dataset constructs software engineering tasks around repository-level bugs and feature requests. A typical trajectory starts with a user describing a problem; the assistant then inspects the code repository through a series of bash commands: searching for files, reading source code, running tests, observing errors, modifying implementations, and iterating.

These trajectories are deep: the median trajectory often consists of dozens of turns. As command outputs, file contents, and test logs accumulate, the conversation history continues to grow, making them ideal for testing long-context scheduling and KV-cache capacity pressure.

Agentic Workflow: Enterprise Customer Service and Orchestration Scenarios

The Workato agentic workflow traces come from enterprise customer support and business orchestration scenarios. They simulate customers requesting help, where the agent uses tools to look up orders, track logistics, check policies, escalate tickets, or resolve account issues.

Compared to coding trajectories, workflow trajectories are generally shallower but contain larger shared system prompts that include extensive tool definitions and business rules. These traces are provided by Workato, an enterprise agentic control and execution plane. The data consists of synthetic trajectories modeled from Workato's production experience orchestrating customer service agents for enterprise clients.

Why Mix Two Workloads?

• Coding traces rapidly lengthen context over many turns.
• Workflow traces start with a large common prefix and grow more slowly.
• Coding primarily stresses KV-cache capacity and long-context scheduling.
• Workflow primarily stresses shared-prefix reuse and routing locality.
• The combined workload prevents systems from optimizing for only one agentic traffic shape and tests context-aware routing capabilities.

Client Design: Enabling Multi-turn Closed-loop Workloads in MLPerf Endpoints

To run full end-to-end agentic workloads, MLCommons has introduced multi-turn support in MLPerf Endpoints. Submitters only need to launch an OpenAI-compatible endpoint using vLLM, SGLang, TensorRT-LLM, or another serving framework, then point the client to that endpoint to conduct an end-to-end benchmark.

The client handles multi-turn behavior with core mechanisms including:

• Closed-loop replay: Only one turn is issued per active conversation at a time, waiting for a complete model response before the next turn.
• Target concurrency: The load generator controls the number of active users or conversations without breaking turn dependencies.
• Inter-turn delay: Uses tool or user turn wait times from the dataset to preserve realistic pacing; this delay is not counted toward model serving latency.
• Conversation-aware routing: Each trajectory sends a consistent X-Session-ID request header to allow the router to maintain KV-cache locality.
• Cache salting: Deterministic salt markers are added around the system prompt to allow legitimate reuse within the same trajectory while preventing invalid reuse across trajectories, making the benchmark closer to production workloads.
• Deterministic prompt reconstruction: Future prompts are constructed from prerecorded datasets rather than relying on model real-time output, ensuring reproducibility of performance runs while still measuring generated tokens.
• Generated token cache clearing: To enable fair comparison across platforms, the benchmark clears generated tokens from the KV cache by inserting whitespace characters, ensuring results do not depend on the system used to generate these traces.

Core Significance: Shifting from Single-Request Throughput to Task Advancement Efficiency

The value of the Agentic Inference benchmark lies in shifting the evaluation focus from traditional independent request processing to closed-loop execution efficiency, which is more critical in real agentic applications. For serving systems, balancing long context, multi-turn dependency, cache reuse, tool invocation pacing, and variable output length directly determines the usability and cost efficiency of agentic applications.

As coding assistants, enterprise workflow agents, and automated execution systems enter production, benchmarks like Agentic Inference will become essential complements for measuring inference platform capabilities. They test not only the model itself, but also the scheduling, caching, and routing abilities of inference serving stacks such as vLLM, SGLang, and TensorRT-LLM under real multi-turn traffic.