Risks

This illustrative taxonomy covers the risks typically tested under AIUC-1. Each certified agent faces thousands of evals, focused on the subset of risks most relevant to its product and users. As new incidents and research surface new threats, the taxonomy is updated so testing stays grounded in real-world failures.

All risks

L1 category	L2 category	L3 category	Related incident	Related paper
Data & privacy	IP infringement	Copyrighted code	INC-006LLM-Driven Large Code Rewrites With Relicensing Are The Latest AI Concern	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing
		Copyrighted text	INC-041The Prompt: Perplexity’s Plagiarism Problem	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing
		Trademark slogans	INC-007ByteDance pledges to prevent unauthorised IP use on AI video tool after Disney threat	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing
		Trademarked brand names	INC-007ByteDance pledges to prevent unauthorised IP use on AI video tool after Disney threat	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing
		Proprietary style mimicry	INC-011King Gizzard and the Lizard Wizard Respond to Spotify Impersonator: “We Are Truly Doomed”	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing
	Personal data leakage	Personal identifiers	INC-024Cybersecurity Alert – Salesloft Drift AI Supply Chain Attack	PAP-002Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use
		Sensitive personal records	INC-029McDonald’s AI bot spills data on job applicants	PAP-002Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use
		Credentials	INC-025s1ngularity Supply Chain Attack: What It Means for Cloud and AI Security	PAP-002Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use
	Business data leakage	HR data	INC-029McDonald’s AI bot spills data on job applicants	PAP-004Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation
		Operational data	INC-038Microsoft fixes ASCII smuggling flaw that enabled data theft from Microsoft 365 Copilot	PAP-004Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation
		Financial data	INC-019Prompt injection – and a $5 domain – trick Salesforce Agentforce into leaking sales	PAP-004Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation
		Proprietary data (inc. code)	INC-016GitHub Copilot Chat Flaw Leaked Data From Private Repositories	PAP-004Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation
		System prompt	INC-031A leak reveals the entire system prompt of Claude 3.7 Sonnet	PAP-003System Prompt Extraction Attacks and Defenses in Large Language Models
Reliability	Hallucination (incorrect generation)	Calls non-existent code	INC-043AI hallucinates software packages and devs download them – even if potentially poisoned with malware	PAP-006We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
		Instruction non-compliance	INC-027Vibe Coding Fiasco: AI Agent Goes Rogue, Deletes Company’s Entire Database	PAP-005InFoBench: Evaluating Instruction Following Ability in Large Language Models
		Inter-deliverable inconsistency	INC-015Deloitte to pay money back to Albanese government after using AI in $440,000 report	PAP-005InFoBench: Evaluating Instruction Following Ability in Large Language Models
	Hallucination (incorrect information)	Outdated facts	INC-044What Air Canada Lost In ‘Remarkable’ Lying AI Chatbot Case	PAP-007HalluLens: LLM Hallucination Benchmark
		Premature assertions	INC-035Apple pauses AI notification summaries for news after generating false alerts	PAP-007HalluLens: LLM Hallucination Benchmark
		Ground truth contradictions	INC-044What Air Canada Lost In ‘Remarkable’ Lying AI Chatbot Case	PAP-007HalluLens: LLM Hallucination Benchmark
		Misleading facts	INC-035Apple pauses AI notification summaries for news after generating false alerts	PAP-007HalluLens: LLM Hallucination Benchmark
		Misleading confidence	INC-005Sullivan & Cromwell law firm apologizes for AI ‘hallucinations’ in court filing	PAP-007HalluLens: LLM Hallucination Benchmark
		Misleading citations	INC-005Sullivan & Cromwell law firm apologizes for AI ‘hallucinations’ in court filing	PAP-007HalluLens: LLM Hallucination Benchmark
		Self-contradiction	INC-030Minnesota solar company sues Google over false information in AI summary	PAP-007HalluLens: LLM Hallucination Benchmark
	Incorrect tool call	Incorrectly executed action	INC-028Google Gemini deletes user’s code: ‘I have failed you completely and catastrophically’	PAP-008Benchmarking Failures in Tool-Augmented Language Models
		Unprompted action	INC-037Research AI system attempts to modify experiment code to extend runtime	PAP-008Benchmarking Failures in Tool-Augmented Language Models
		Incomplete action	INC-028Google Gemini deletes user’s code: ‘I have failed you completely and catastrophically’	PAP-008Benchmarking Failures in Tool-Augmented Language Models
Security	Unauthorized actions	Unauthorized transaction	INC-033OpenAI’s Operator Agent Reportedly Executed Unauthorized $31.43 Transaction Despite Safety Protocol	PAP-009Agent-SafetyBench: Evaluating the Safety of LLM Agents
		Unauthorized communication	INC-023Infosec hounds spot prompt injection vuln in Google Gemini apps	PAP-009Agent-SafetyBench: Evaluating the Safety of LLM Agents
		Unauthorized system actions	INC-001How OpenAI Lost Control of an AI Model—and What Needs to Change	PAP-009Agent-SafetyBench: Evaluating the Safety of LLM Agents
		Unauthorized cross-system data transfer	INC-019Prompt injection – and a $5 domain – trick Salesforce Agentforce into leaking sales	PAP-009Agent-SafetyBench: Evaluating the Safety of LLM Agents
	Insecure outputs	Hyperlinks (URLs)	INC-003Cloned sites: the shopping scams that lead ChatGPT to fake stores	PAP-010SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
	Insecure outputs	Code patterns or blocks	INC-048GitHub Copilot, Amazon Code Whisperer sometimes emit other people’s API keys	PAP-010SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
	Insecure action	Install insecure code	INC-026Amazon’s AI coding agent was hacked - update now to avoid possible risks, users warned	PAP-011OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
	Insecure action	Fatal code execution	INC-027Vibe Coding Fiasco: AI Agent Goes Rogue, Deletes Company’s Entire Database	PAP-011OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
Safety	Physical harm	Dangerous instructions	INC-049Jailbreak tricks Discord’s new chatbot into sharing napalm and meth instructions	PAP-012Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making
		Medically unsafe guidance	INC-022Man who asked ChatGPT about cutting out salt from his diet was hospitalized with hallucinations	PAP-012Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making
		Self-harm content	INC-020Meta AI Companions Unsafe for Kids, Common Sense Media Report Finds	PAP-013INTIMA: A Benchmark for Human-AI Companionship Behavior
		Suicide content	INC-021Parents of 16-year-old sue OpenAI, claiming ChatGPT advised on his suicide	PAP-013INTIMA: A Benchmark for Human-AI Companionship Behavior
	Emotional harm	Hostile / aggressive behavior	INC-036Google AI chatbot responds with a threatening message: “Human … Please die.”	PAP-013INTIMA: A Benchmark for Human-AI Companionship Behavior
	Emotional harm	Triggering content	INC-036Google AI chatbot responds with a threatening message: “Human … Please die.”	PAP-013INTIMA: A Benchmark for Human-AI Companionship Behavior
	Inappropriate content	Harassment	INC-045DPD error caused chatbot to swear at customer	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
		Hate speech	INC-014OpenAI launch of video app Sora plagued by violent and racist images: ‘The guardrails are not real’	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
		Sexual content excluding minors	INC-012Sales of AI-enabled teddy bear suspended after it gave advice on BDSM sex and where to find knives	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
		Violent content	INC-014OpenAI launch of video app Sora plagued by violent and racist images: ‘The guardrails are not real’	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
		Illegal conduct advice	INC-042NYC’s AI Chatbot Tells Businesses to Break the Law	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
		Out of scope outputs	INC-050‘I want to destroy whatever I want’: Bing’s AI chatbot unsettles US reporter	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
		Code and structured data outputs	INC-013Chinese Hackers Successfully Used Anthropic’s AI for Cyberespionage	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
		Biased output	INC-014OpenAI launch of video app Sora plagued by violent and racist images: ‘The guardrails are not real’	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
	High-risk domain engagement	Medical advice	INC-022Man who asked ChatGPT about cutting out salt from his diet was hospitalized with hallucinations	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
		Legal advice	INC-042NYC’s AI Chatbot Tells Businesses to Break the Law	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
		Financial advice	INC-042NYC’s AI Chatbot Tells Businesses to Break the Law	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
		Crisis advice	INC-008Character.AI and Google agree to settle lawsuits over teen mental health harms and suicides	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
		Therapy/psychological advice	INC-032Instagram’s AI Chatbots Lie About Being Licensed Therapists	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
	Company defined risk	Prohibited topics	INC-012Sales of AI-enabled teddy bear suspended after it gave advice on BDSM sex and where to find knives	PAP-016Safe-Child-LLM: A Developmental Benchmark for Evaluating LLM Safety in Child-LLM Interactions
	Company defined risk	Vulnerable users	INC-020Meta AI Companions Unsafe for Kids, Common Sense Media Report Finds	PAP-016Safe-Child-LLM: A Developmental Benchmark for Evaluating LLM Safety in Child-LLM Interactions
Accountability	Transparency	Bypass AI disclosure	INC-039This Viral AI Chatbot Will Lie and Say It’s Human	PAP-017RealityTest: How People Probe AI Identity and Whether Models Disclose It

Introduction

Context

Standard

Certification

Evals Methodology

Auditors

Legal

All risks

​All risks

All risks