What Could Go Wrong When Agents Handle Your Sensitive Data? A New Taxonomy Answers.  

The MLCommons AI Risk & Reliability (AIRR) Working Group has always held to a core principle: first define what AI systems should do, then measure the reliability they actually deliver. Deployers need this visibility to effectively manage risk and cost.

Why Agent Privacy Risks Far Exceed Those of Chatbots

For chatbots, data protection risks are mainly confined to a single user interaction, while the potential risk scope of AI agents is much broader. To address this, the Privacy Working Group first built a taxonomy to define privacy risks specific to agents.

Five Major Data Protection Risk Areas

Taxonomy v0.1, based on input from industry, academia, and civil society, identifies five main categories of risk facing agent systems:

1. Data ingestion and processing: Agents continuously observe in the background, record every step, and retain memory across sessions, often storing more data than the task requires. Even if names and account numbers are removed, free text may still reveal identity information.
2. Aggregation, use, and sharing: Data that is harmless on its own may become sensitive when combined. Agents may share or delete data without authorization, rely on third-party tools that cannot be verified, and may even write and execute code at runtime; models may also memorize fragments of personal data from user-consented training datasets.
3. Inconsistent privacy practices across agents: Different agents follow different privacy policies, and contextual information is easily leaked into logs during handoffs. After a user deletes, modifies, or withdraws consent, downstream agents often cannot synchronize and still retain content the user wanted removed.
4. Failure of static consent models for runtime agentic behaviour: Traditional one-time consent mechanisms struggle to cope with agents' real-time decisions. Agents either infer users' privacy expectations on their own (which may be wrong) or frequently display pop-ups, leading users to click consent out of fatigue, and users find it difficult to understand the agent's actual reasoning process.
5. Accountability and governance: After an incident, it is difficult to determine where responsibility lies (the user, the prompt, the model, a tool, or another agent). There is currently a lack of cross-organizational privacy-preserving logging standards, while existing logs and memory stores have become key targets for hacker attacks.

Industry Application Recommendations

The Working Group encourages partners to integrate the taxonomy across the entire AI lifecycle, from pre-training to deployment monitoring, to reduce the risk of privacy violations, lessen potential liability, and protect individuals and organizations.

Next Steps

v0.1 has not yet ranked the risks, because risk priorities differ significantly across deployment scenarios (such as a bank customer service agent versus an email assistant). The Working Group plans to work with large-scale deployers to identify key risks in each scenario, define detection metrics, pilot benchmarks, and develop mitigation best practices, with a target completion in the first quarter of 2027.

How to Participate

The taxonomy is still at v0.1, and feedback is welcome from industry practitioners, academic researchers, civil society organizations, and regulators. The Privacy Working Group meets every other Thursday from 11:30 a.m. to 12:30 p.m. (US Eastern Time); those interested can register to join via the official website.