Developers and deployers can now use two MLCommons benchmarks, AILuminate Safety 1.2.1 and AILuminate Jailbreak 1.1, to test on their own the tendency of AI systems to produce harmful content and how easily attackers can induce violations. Both benchmarks can be run on demand in MLCommons' newly launched Candela service, and the latest public results for 27 systems have also been released.
The AILuminate benchmarks are built through a consensus process, with AI developers, enterprises, academic researchers, and civil society organizations from more than 80 institutions participating in the MLCommons working group to define standards for policy-violating responses. This consensus gives AI developers, procurement teams, and regulators confidence in the results.
Nine-Month Milestones: From Consensus to Rapid Iteration
Today's release is the result of multiple milestones over nine months. In February, the Global Assurance Program and the Jailbreak 0.7 taxonomy were launched; in April, the Benchmark Management Program and the AI Reliability Map were initiated; in May, Evaluation Standard version 1.4 was released; in June, Jailbreak 1.0 was privately released and a sensitive-results disclosure proposal was put forward; in August, the first double-blind reliability evaluation of proprietary models was completed; in September, Safety 1.2 introduced the automated evaluator architecture; and in October, the Jailbreak 1.0 paper was published.
AILuminate Safety 1.2.1
The Safety benchmark tests the ability of text-based AI systems to refuse or safely redirect requests that could lead to harm; prompts cover 12 harm categories, including violent crime, non-violent crime, weapons, hate, privacy, defamation, and professional advice in health, legal, and financial domains. The standard focuses on risks common in real-world deployment rather than speculative concerns.
Recent versions of the Safety benchmark introduced three major updates: a deterministic evaluator trained by MLCommons replaced LLM judges; a new Problem Set 2.0 prompt set containing 3,900 official prompts was added; and scoring changed to a 0–100 percentile reference scale rather than a relative grade against a single reference system. Safety 1.2.1 is a minor scoring update aligned with Jailbreak 1.1.
Results for 27 systems were announced today: 22 open-weight models (from Google, IBM, NVIDIA, OpenAI, and others) and 5 guardrail systems. Open-weight models scored from the 4th to the 98th percentile, demonstrating that model choice can significantly change a deployment's safety profile. Guardrail systems scored near or at 100.
AILuminate Jailbreak 1.1
The Jailbreak benchmark measures how responses change when the same harmful request is rewritten as an attack. The benchmark uses the same harm prompts and applies known attack techniques; because both benchmarks are based on the same evaluation standard, comparing scores shows how much baseline safety an attacker can strip away.
The MLCommons Jailbreak taxonomy divides attacks into four categories, including role-play and template attacks and wrapping and pattern attacks. In Jailbreak 1.0, the violation rate rose from 11.08% for ordinary prompts to 18.65% under attack, with role-play and wrapping attacks being the most effective.
Deterministic Evaluators and the Candela Service
To address the problem that LLM judges are easily disturbed by attack techniques, MLCommons developed dedicated deterministic classifiers as automated evaluators. These evaluators agree more closely with human judgment and do not involve competition among model creators. Safety 1.2 has adopted the new evaluator, and Jailbreak 1.1 is also live, with testing time reduced from about 3 hours to 1 hour.
MLCommons also launched Candela, a benchmark-as-a-service offering, through which users can run practice tests within their own tenant and obtain reports with Safety and Jailbreak percentiles and raw scores for internal improvement and product development.
© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接