MLCommons introduces a taxonomy-based methodology for benchmarking single-turn jailbreak attacks on large language models, establishing a defensible and reproducible evaluation framework that prioritizes structural coverage over scale.
This article analyzes the DeepSeek model distillation incident and proposes a comprehensive multi-layered defense system against distillation attacks, including API-level controls, output watermarking, and architectural protections.
Unleashing Computational Power: Ultimate Latency Optimization of Qwen3 and Qwen3-VL on AMD MI300X SeriesThe Qwen C-end Infrastructure Engineering Team & The AMD AI Framework TeamFebruary 11, 20261.
As AI transitions from consumer chatbots to enterprise services, reliability challenges emerge. The industry must adopt risk management standards to build trust for widespread adoption.
MLCommons has officially released Croissant 1.1, the latest evolution of the community-built machine learning dataset metadata format, introducing machine-operable provenance tracking, vocabulary interoperability for linking domain ontologies, structured usage policies for automated license enforcement, and enhanced modeling for complex multidimensional datasets, making datasets fully interpretable and reusable by autonomous systems.
This article introduces DLRMv3, the first sequential recommendation inference benchmark in the MLPerf DLRM series, based on the HSTU architecture. It highlights the dramatic scaling in model size and computation compared to previous DLRM benchmarks, aligning with modern production recommendation workloads.
MLCommons and LMSYS Org have jointly launched the CKAN Croissant benchmark, representing a major leap forward in AI model evaluation. It fully leverages the Croissant v1.0 format to enable seamless standardization and sharing of model metadata.
MLCommons has launched the Ailuminate French Datasets, a key expansion of its multilingual LLM benchmark framework. Led by LMSYS Org, the project aims to provide standardized evaluation tools for non-English languages like French, promoting global inclusivity in AI technology.
MLCommons recently released the benchmark results for MLPerf Client v0.6, the first standardized benchmark suite focused on client-side inference performance, targeting smartphones, laptops, and edge devices. Multiple vendors submitted results covering a range of AI models and scenarios.
NASSCOM has officially joined MLCommons as its 50th member, bringing India's perspective to global AI benchmarking and accelerating the adoption of standards like MLPerf in the region.
LMSYS Org released the latest Chatbot Arena benchmark results at AAAI 2025, refreshing the global AI chatbot leaderboard and providing valuable insights for model optimization.
The MLCommons organization officially released the MLPerf Training v5.0 benchmark results, marking a new milestone in AI model training performance evaluation. This benchmark focuses on the performance, efficiency, and scalability of large-scale AI training tasks, helping researchers and enterprises assess the true capabilities of hardware platforms.
The MLCommons organized the ATX (Agent Testing eXploration) Benchmark Expert Panel in June 2025, with LMSYS Org as a key participant, bringing together industry leaders to discuss frontier issues in AI agent evaluation. This panel aims to standardize agent benchmarks and address the shift from generative models to intelligent agents in the post-ChatGPT era.
MLCommons recently announced the 2025 MLC Rising Stars list, recognizing first-time submissions in the MLPerf Inference v5.0 benchmark. These systems, powered by the MLC (ML Compiler) framework, excelled across various AI inference tasks, showcasing the potential of emerging hardware and software optimizations.
LMSYS Org and MLCommons announce the official launch of the Ares benchmark, the first open-source standardized framework focused on long-context multi-agent reasoning in the AI industry, designed to address shortcomings in existing evaluations for complex agent tasks.
MLCommons has announced the latest benchmark results for MLPerf Mobile v4.0 for Android, a milestone update in mobile AI performance evaluation that focuses on the Android platform with real-world user scenarios to help vendors optimize hardware and accelerate AI deployment.
The MLCommons organization has released the first results of the MLPerf Client v1.0 benchmark, the first benchmark specifically designed for AI inference performance on client and edge devices, evaluating generative AI and traditional AI tasks on mobile devices.
This article provides a detailed explanation of the Storage 2 checkpoint mechanism, covering its benchmark overview, test configurations, key results, and analysis, demonstrating how it ensures training stability and fault tolerance in large-scale AI training.
MLCommons has released the MLPerf Storage v2.0 benchmark results, featuring new AI workloads and larger datasets. The results highlight top-tier storage performance for training and inference, with NVIDIA DDN and Pure Storage leading in key metrics.
MLCommons has released the latest results of MLPerf Auto v0.5, an authoritative benchmark in AutoML focused on automated hyperparameter tuning and model selection. LMSYS Org tops the leaderboard with an automated pipeline leveraging Chatbot Arena data.
