Call for Submission: Edge Agentic Inference Benchmark for MLPerf Inference v6.1

The MLCommons Edge LLM Taskforce announced that it will introduce a new Edge Agentic Inference benchmark in the MLPerf Inference v6.1 round. As agentic LLMs such as coding copilots, robot controllers, and private local assistants increasingly run on-device, measuring the accuracy and response speed of model tool calling under real edge budgets is becoming increasingly critical.

This round's benchmark uses the Qwen3.6-27B model (released April 22, 2026) for the first time, running in Q4_K_M GGUF quantization format on a single edge accelerator in the reference implementation. Submitters may also use any allowed quantization scheme that meets the accuracy threshold. The workload consists of two parts: the Berkeley Function Calling Leaderboard v4 (BFCL v4) for deterministic, LLM-judge-free accuracy thresholds, and a recorded agentic-coding replay for single-stream performance evaluation. Together, they capture the real challenges of on-device agents: ever-growing tool-calling conversations that must fit within a fixed context window while remaining sufficiently responsive to a single interactive user.

Submission deadline is July 31, 2026. MLCommons invites hardware vendors, edge device manufacturers, and inference system experts to submit results to help raise the benchmarking standard for on-device agentic inference.

Why an Edge-Side Agentic AI Benchmark Is Needed

Applied AI is shifting from single-turn text generation to multi-turn Agentic workloads. A real agentic session is not "one prompt, one answer" but a continuous trajectory: each turn appends tool outputs and model responses to a growing conversation history; the model switches between short tool calls and longer reasoning; and each turn strictly depends on the previous one.

The data-center version of the MLPerf Agentic benchmark (planned for September) focuses on a different set of issues: large Mixture-of-Experts models, long-horizon trajectories with cumulative multi-turn contexts exceeding 100K tokens, and Pareto frontiers per GPU via concurrent scanning. The edge side is the opposite. This benchmark primarily targets the following constraints:

• Fixed memory and power budget: A single small accelerator must simultaneously hold model weights, KV cache, and activations. Full-precision frontier models typically cannot fit, making quantization a necessity.
• Single in-flight request: Edge inference primarily serves one interactive user, not thousands of concurrent sessions in a data center.
• Fixed served context window: The reference implementation fixes the served context at 32K tokens. This is not a device limit but a controlled benchmark parameter, so long trajectories may exhaust the context window, making context growth and truncation measurable first-order effects.
• Focus on single-user latency rather than total throughput: On a single-slot edge device, throughput and latency are roughly reciprocal, so more important metrics are single-turn TTFT, TPOT, and end-to-end turn latency, rather than aggregate QPS or token throughput optimized for multi-stream data-center servers.

The benchmark inherits the multi-turn methodology from the data-center Agentic specification, including terminology, deterministic replay, JSONL dataset schema, and inline accuracy check, and is specialized for the edge scenario: edge model and quantization scheme, single-stream load pattern, latency-centric metrics, and statistically robust accuracy thresholds.

Model Selection: Qwen3.6-27B and Q4_K_M Quantization

The reference model is Qwen3.6-27B, served via llama.cpp (commit cfff1fc) in Q4_K_M GGUF quantization, with the GGUF file sourced from unsloth/Qwen3.6-27B-GGUF as Qwen3.6-27B-Q4_K_M.gguf.

MLCommons chose Qwen3.6 because it is an open (Apache 2.0) model with strong tool-calling capabilities and a native MTP speculative-decoding head. Although it is a 27B dense model, Alibaba reported it surpasses its previous 397B-MoE flagship on major coding benchmarks and achieves 77.2% on SWE-bench Verified. It also offers good deployment convenience: official weights are released on Hugging Face and ModelScope, and community GGUF versions are directly runnable under llama.cpp.

Quantization is a key choice for edge deployment. Q4_K_M is the "Medium" variant of 4-bit K-quant: weights are stored at 4 bits per value, providing about 4x memory compression over BF16; attention blocks retain higher precision than feed-forward blocks; and importance-weighted (imatrix) calibration preserves the most impactful weights. The result is that the 27B model requires approximately 16.5 GB VRAM to run, compared to about 54 GB for BF16, at the cost of roughly 2% to 5% accuracy loss. This is the difference between "can run on an edge GPU" and "cannot run."

Reference Sampling Parameters

In this tool-calling workload, reasoning is deliberately turned off because it reduces single-turn accuracy while significantly increasing latency. MLCommons states the benchmark has been validated on a range of edge accelerators from multiple vendors, demonstrating the portability of the workload and methodology.

Benchmark Tasks: Dual Datasets for Accuracy and Performance

The benchmark uses two datasets serving different roles.

Accuracy Dataset: BFCL v4

The Berkeley Function Calling Leaderboard v4 (BFCL v4) is used to test whether the model can correctly and deterministically call functions without relying on an LLM judge. It covers single-turn requests—“one prompt to one structured tool call”—and is primarily divided into three categories:

• non_live: Compared via AST match against gold labels.
• live: Also evaluated via AST match.
• hallucination: A binary check—whether the model can refuse to call a function when available tools are irrelevant to the question.

Additionally, BFCL v4 includes optional multi-turn agentic conversations executed via in-process Python simulators. Categories originate from the public gorilla-llm/gorilla-eval-set and are automatically downloaded at runtime.

The scoring threshold for this round uses only the single-turn set, sampled by category to a stable estimate of approximately 995 samples: non_live 72% (~712 samples), live 17% (~171 samples), hallucination 11% (~112 samples). A subset_floor of 25 is set, so any subset with fewer than 25 entries is fully included.

Performance Dataset: Recorded Agentic Coding Replay

The performance dataset is a subset of the MLPerf Agentic benchmark, containing recorded multi-turn agentic coding trajectories. These tasks follow a SWE-bench-like format, sourced from real code repositories (e.g., astropy), and are served as deterministic replays for the performance workload, scaled to suit single-device edge serving.

The reference set includes 20 conversations with 1,007 turns, designed so that all trajectories do not overflow the 32K-token served context, with peak input lengths around 23.5K tokens. Since no context overflow occurs, every turn completes; with a concurrency of one and only one in-flight request at a time, a single edge device can finish a full run in reasonable time, and a valid run requires zero dropped turns.

The dataset itself also serves as ground truth: the tool calls recorded in each trajectory drive a zero-cost inline accuracy check executed concurrently with latency measurement. Accuracy is checked using multiset IOU of executed calls, so correctness and latency come from the same execution.

Request and Response Examples

A single-turn function-calling request is sent to the model's OpenAI-compatible /v1/chat/completions endpoint with available tool schemas. For example, when a user asks for the current temperature in San Francisco in Celsius, the request may include the following tool definition:

{
  "messages": [
    {
      "role": "user",
      "content": "What is the current temperature in San Francisco, in Celsius?"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather for a given location.",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "City and state, e.g. San Francisco, CA"
            },
            "unit": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"]
            }
          },
          "required": ["location", "unit"]
        }
      }
    }
  ]
}

The model should return a structured tool call:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "tool_calls": [
          {
            "type": "function",
            "function": {
              "name": "get_current_weather",
              "arguments": "{\"location\": \"San Francisco, CA\", \"unit\": \"celsius\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ]
}

The BFCL v4 AST checker compares the predicted call against the gold label item by item, including function name and each parameter, ensuring scoring is precise and reproducible.

Significance

This Edge Agentic Inference benchmark shifts evaluation focus from traditional throughput to real on-device agent experience: under limited memory and power, can the model maintain accuracy in long-context, multi-turn tool calls while providing acceptable response latency for a single user? For chip, device, and system vendors seeking to demonstrate their edge AI inference capabilities, MLPerf Inference v6.1 offers a public measurement framework that more closely reflects next-generation on-device AI applications.