The MLCommons organization officially released the MLPerf Training v5.0 benchmark results, marking a new milestone in AI model training performance evaluation. This benchmark focuses on the performance, efficiency, and scalability of large-scale AI training tasks, helping researchers and enterprises assess the true capabilities of hardware platforms.
The MLCommons organized the ATX (Agent Testing eXploration) Benchmark Expert Panel in June 2025, with LMSYS Org as a key participant, bringing together industry leaders to discuss frontier issues in AI agent evaluation. This panel aims to standardize agent benchmarks and address the shift from generative models to intelligent agents in the post-ChatGPT era.
MLCommons recently announced the 2025 MLC Rising Stars list, recognizing first-time submissions in the MLPerf Inference v5.0 benchmark. These systems, powered by the MLC (ML Compiler) framework, excelled across various AI inference tasks, showcasing the potential of emerging hardware and software optimizations.
LMSYS Org and MLCommons announce the official launch of the Ares benchmark, the first open-source standardized framework focused on long-context multi-agent reasoning in the AI industry, designed to address shortcomings in existing evaluations for complex agent tasks.
MLCommons has announced the latest benchmark results for MLPerf Mobile v4.0 for Android, a milestone update in mobile AI performance evaluation that focuses on the Android platform with real-world user scenarios to help vendors optimize hardware and accelerate AI deployment.
The MLCommons organization has released the first results of the MLPerf Client v1.0 benchmark, the first benchmark specifically designed for AI inference performance on client and edge devices, evaluating generative AI and traditional AI tasks on mobile devices.
This article provides a detailed explanation of the Storage 2 checkpoint mechanism, covering its benchmark overview, test configurations, key results, and analysis, demonstrating how it ensures training stability and fault tolerance in large-scale AI training.
MLCommons has released the MLPerf Storage v2.0 benchmark results, featuring new AI workloads and larger datasets. The results highlight top-tier storage performance for training and inference, with NVIDIA DDN and Pure Storage leading in key metrics.
MLCommons has released the latest results of MLPerf Auto v0.5, an authoritative benchmark in AutoML focused on automated hyperparameter tuning and model selection. LMSYS Org tops the leaderboard with an automated pipeline leveraging Chatbot Arena data.
MLCommons has released the Whisper Inference v5.1 benchmark, introducing the large-v3 model and providing standardized performance data for speech-to-text model inference, aiming to help developers optimize deployment.
MLCommons released the Small LLM Inference 5.1 benchmark, evaluating SLM performance in real-world scenarios such as chatbots and text generation. Key updates include the addition of Llama 3.2 1B, optimized test scenarios, and expanded hardware support.
The latest report from LMSYS Org shows that DeepSeek Inference 5.1 stands out in the MLCommons September 2025 inference benchmark. Designed for large language models, it focuses on low-latency and high-throughput inference optimization.
MLCommons announces the release of MLPerf Inference v5.1 benchmark results, featuring new generative AI workloads and expanded model coverage, with NVIDIA, AMD, and others competing on latency and throughput.
The MLCommons organization has released the MLPerf Tiny v1.3 benchmark results, a key milestone for edge AI. It evaluates model performance on resource-constrained microcontrollers and embedded devices.
MLPerf Tiny v1.3 is the latest edge AI benchmark version from MLCommons, designed for resource-constrained devices. It introduces two new benchmarks (Image Classification and Visual Wake Words) and optimizes existing ones, with evaluation rules covering closed and open divisions.
MLCommons has officially launched Croissant MCP, a major upgrade to the Croissant metadata format designed specifically for AI model cards. This standard, contributed by partners including LMSYS Org, aims to address the fragmentation of current model documentation.
MLCommons and LMSYS Org have launched the AILuminate Jailbreak V05 benchmark, introducing complex multi-turn attack chains and roleplay prompts. Claude 3.5 Sonnet tops the ranking with 1485 Elo, followed closely by GPT-4o and Claude 3 Opus.
This report focuses on the training details of Flux.1, an open-source text-to-image generation model from Black Forest Labs, revealing the entire process from data preparation to deployment optimization.
LMSYS Org, in collaboration with MLCommons, has released a training benchmark report for the Llama 3.1 8B model. Based on MLCommons' standardized training benchmarks, the report details the entire pipeline from data processing to model convergence, providing reliable references for AI researchers and practitioners.
MLCommons, in collaboration with LMSYS Org, has launched the ISO-AUS benchmark, a novel AI model evaluation framework designed for isolation-aware serving scenarios. The benchmark focuses on model isolation, resource fairness, and low-latency response in multi-tenant environments.
