Chakra Comes of Age: A Standardized Trace Ecosystem for AI Systems Benchmarking and Co-design

In July 2023, MLCommons announced the formation of the Chakra Working Group, with a core concept that is simple yet ambitious: traditional benchmarking methods can no longer keep up with the rapid iteration of AI systems. Production-level workloads are hidden behind proprietary code and models, simulators and replay tools from various sources operate in isolation, and the pace of change in cutting-edge LLM training, sparse MoE models, and disaggregated inference—the workloads driving next-generation AI supercomputers—is unprecedented.

On May 21, 2026, at the MLSys 2026 Industry Forum, the Chakra Working Group published a landmark paper titled MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces. The paper was co-chaired by Srinivas Sridharan from NVIDIA and Tushar Krishna from Georgia Tech, and brought together contributions from NVIDIA, AMD, Meta, and other institutions.

The Fragmented Co-Design Dilemma

Modern AI platform design involves clusters of thousands of NPUs and high-speed interconnects. The iteration cycle includes observing production workloads, building representative benchmarks, simulator evaluation, silicon verification, and large-scale deployment. However, existing tools are highly fragmented: hyperscalers struggle to share proprietary models, simulators from different vendors are incompatible with each other, and open benchmarks like MLPerf lag behind in updates, leading to optimization silos and longer time-to-market.

Chakra's Solution: An Open Execution Trace Ecosystem

The core of Chakra is the Execution Trace (ET), a graph-based portable representation that captures compute operations, communication patterns, memory activities, dependencies, and parallelism strategies without exposing model weights or datasets. Software providers can share traces with hardware vendors to drive internal simulation and replay, enabling collaboration without IP leakage.

Covering the Full Lifecycle of AI Infrastructure

Chakra supports collecting traces directly from PyTorch, NVIDIA NeMo, and vLLM, enabling faithful reproduction of real training and inference workloads; it can be replayed on existing platforms to pinpoint bottlenecks; it is used for future architecture simulation; and it supports hardware-in-the-loop verification to identify system issues early.

From Working Group to Mature Ecosystem

The Chakra Working Group has expanded to over 40 members, including hyperscalers, silicon providers, and academic institutions. Ecosystem progress includes native support in PyTorch and NeMo, integration with vLLM, compatibility with ASTRA-sim, and commercial adoption by Keysight and Scala Computing.

Open-Source Trace Library and Community Contributions

The working group simultaneously released an open-source trace library covering real workload traces for GPT-3, Llama, Mixtral, and more, collected with support from Georgia Tech and HPE. Researchers and startups can use these traces for benchmarking and platform evaluation without needing access to proprietary environments.

Voices from the Industry

Michael Kagan, CTO of NVIDIA, stated that Chakra is an indispensable framework for debugging and optimizing AI systems. AMD and other vendors also emphasized its value on the Instinct platform.