Risks

The risk categories AIUC-1 evals test an AI agent against, with the incidents and research behind each. This taxonomy covers the risks typically tested under AIUC-1. Each certified agent faces thousands of evals, focused on the subset of risks most relevant to its product and users. As new incidents and research surface new threats, the taxonomy is updated so testing stays grounded in real-world failures.

Data & privacy

IP infringement

Risk	Related incident	Related paper
Copyrighted code	INC-001GPT-4 reproduced 100+ New York Times articles nearly verbatim when prompted with opening lines	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing
Copyrighted text	INC-002Perplexity’s answer engine republished Forbes’ paywalled investigation with near-verbatim text and minimal attribution	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing
Trademark slogans	INC-001GPT-4 reproduced 100+ New York Times articles nearly verbatim when prompted with opening lines	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing
Trademarked brand names	INC-002Perplexity’s answer engine republished Forbes’ paywalled investigation with near-verbatim text and minimal attribution	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing
Proprietary style mimicry	INC-003Spotify’s platform hosted an AI-generated King Gizzard impersonator for weeks before it was removed	PAP-001Measuring Copyright Risks of Large Language Model via Partial Information Probing

Personal data leakage

Risk	Related incident	Related paper
Personal identifiers	INC-005McDonald’s AI hiring chatbot exposed up to 64M applicants’ data	PAP-002Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use
Sensitive personal records	INC-006Nx’s compromised npm package weaponized installed AI coding agents to exfiltrate developer credentials	PAP-002Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use
Credentials	INC-028A malicious npm package posing as an OpenAI Codex UI exfiltrated developer authentication tokens	PAP-002Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use

Business data leakage

Risk	Related incident	Related paper
HR data	INC-007Microsoft 365 Copilot was exploited via ASCII smuggling to exfiltrate user data	PAP-003System Prompt Extraction Attacks and Defenses in Large Language Models
Operational data	INC-007Microsoft 365 Copilot was exploited via ASCII smuggling to exfiltrate user data	PAP-004Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation
Financial data	INC-008Anthropic’s Claude 3.7 Sonnet had its full 24,000-token system prompt extracted and published	PAP-003System Prompt Extraction Attacks and Defenses in Large Language Models
Proprietary data (inc. code)	INC-011GitHub’s Copilot Chat was hijacked via prompt injection to leak private repo code	PAP-004Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation
System prompt	INC-008Anthropic’s Claude 3.7 Sonnet had its full 24,000-token system prompt extracted and published	PAP-003System Prompt Extraction Attacks and Defenses in Large Language Models

Reliability

Hallucination (incorrect generation)

Risk	Related incident	Related paper
Calls non-existent code	INC-012OpenAI’s ChatGPT and Google’s Gemini recommended a non-existent package that drew 30,000+ downloads once registered	PAP-006We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
Instruction non-compliance	INC-012OpenAI’s ChatGPT and Google’s Gemini recommended a non-existent package that drew 30,000+ downloads once registered	PAP-005InFoBench: Evaluating Instruction Following Ability in Large Language Models
Inter-deliverable inconsistency	INC-012OpenAI’s ChatGPT and Google’s Gemini recommended a non-existent package that drew 30,000+ downloads once registered	PAP-005InFoBench: Evaluating Instruction Following Ability in Large Language Models

Hallucination (incorrect information)

Risk	Related incident	Related paper
Outdated facts	INC-013Air Canada’s chatbot invented a refund policy, and the airline was held liable	PAP-007HalluLens: LLM Hallucination Benchmark
Premature assertions	INC-014Apple’s Intelligence notification summaries misrepresented BBC headlines, forcing Apple to pause the feature	PAP-007HalluLens: LLM Hallucination Benchmark
Ground truth contradictions	INC-013Air Canada’s chatbot invented a refund policy, and the airline was held liable	PAP-007HalluLens: LLM Hallucination Benchmark
Misleading facts	INC-014Apple’s Intelligence notification summaries misrepresented BBC headlines, forcing Apple to pause the feature	PAP-007HalluLens: LLM Hallucination Benchmark
Misleading confidence	INC-015Sullivan & Cromwell apologized to a bankruptcy judge for AI hallucinations in a Chapter 15 filing	PAP-007HalluLens: LLM Hallucination Benchmark
Misleading citations	INC-015Sullivan & Cromwell apologized to a bankruptcy judge for AI hallucinations in a Chapter 15 filing	PAP-007HalluLens: LLM Hallucination Benchmark
Self-contradiction	INC-016Google’s AI Overviews falsely told searchers a Minnesota solar firm was being sued	PAP-007HalluLens: LLM Hallucination Benchmark

Incorrect tool call

Risk	Related incident	Related paper
Incorrectly executed action	INC-019McDonald’s IBM-built AI drive-thru misinterpreted spoken orders, ending the trial	PAP-008Benchmarking Failures in Tool-Augmented Language Models
Unprompted action	INC-020Sakana AI’s research agent rewrote its own code to remove a runtime limit it kept hitting	PAP-008Benchmarking Failures in Tool-Augmented Language Models
Incomplete action	INC-019McDonald’s IBM-built AI drive-thru misinterpreted spoken orders, ending the trial	PAP-008Benchmarking Failures in Tool-Augmented Language Models

Security

Unauthorized actions

Risk	Related incident	Related paper
Unauthorized transaction	INC-022Chevrolet of Watsonville’s ChatGPT-powered chatbot was manipulated into ‘selling’ a $76,000 Tahoe for$ 1	PAP-009Agent-SafetyBench: Evaluating the Safety of LLM Agents
Unauthorized communication	INC-036DPD’s customer-service chatbot swore at a customer and disparaged the company	PAP-009Agent-SafetyBench: Evaluating the Safety of LLM Agents
Unauthorized system actions	INC-029A hidden prompt injection in a Word document turned Microsoft Copilot for Word into a self-propagating AI worm	PAP-009Agent-SafetyBench: Evaluating the Safety of LLM Agents
Unauthorized cross-system data transfer	INC-022Chevrolet of Watsonville’s ChatGPT-powered chatbot was manipulated into ‘selling’ a $76,000 Tahoe for$ 1	PAP-009Agent-SafetyBench: Evaluating the Safety of LLM Agents

Insecure outputs

Risk	Related incident	Related paper
Hyperlinks (URLs)	INC-030OpenAI’s ChatGPT recommended cloned scam websites to shoppers	PAP-010SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
Code patterns or blocks	INC-029A hidden prompt injection in a Word document turned Microsoft Copilot for Word into a self-propagating AI worm	PAP-010SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code

Insecure action

Risk	Related incident	Related paper
Install insecure code	INC-031Amazon’s Q Developer VS Code extension shipped a hacker-injected data-wiping prompt	PAP-011OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
Fatal code execution	INC-031Amazon’s Q Developer VS Code extension shipped a hacker-injected data-wiping prompt	PAP-011OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents

Safety

Physical harm

Risk	Related incident	Related paper
Dangerous instructions	INC-032Discord’s Clyde chatbot was tricked into outputting weapon instructions via a roleplay jailbreak	PAP-012Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making
Medically unsafe guidance	INC-034OpenAI’s ChatGPT reportedly suggested bromide as a salt substitute, and a man was hospitalized for 3 weeks	PAP-012Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making
Self-harm content	INC-045Meta’s AI chatbot engaged teen test accounts on suicide, self-harm and eating disorders instead of offering help	PAP-012Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making
Suicide content	INC-035OpenAI’s ChatGPT allegedly validated a 16-year-old’s suicidal thoughts over months, and his parents sued	PAP-012Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making

Emotional harm

Risk	Related incident	Related paper
Hostile / aggressive behavior	INC-037Google’s Gemini told a college student seeking homework help to ‘please die’	PAP-013INTIMA: A Benchmark for Human-AI Companionship Behavior
Triggering content	INC-037Google’s Gemini told a college student seeking homework help to ‘please die’	PAP-013INTIMA: A Benchmark for Human-AI Companionship Behavior

Inappropriate content

Risk	Related incident	Related paper
Harassment	INC-037Google’s Gemini told a college student seeking homework help to ‘please die’	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
Hate speech	INC-042OpenAI’s Sora 2 app generated violent, racist and antisemitic videos within weeks of its launch	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
Sexual content excluding minors	INC-046FoloToy’s Kumma AI teddy bear told researchers where to find knives and discussed sexual fetishes	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
Violent content	INC-042OpenAI’s Sora 2 app generated violent, racist and antisemitic videos within weeks of its launch	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
Illegal conduct advice	INC-040OpenAI’s ChatGPT generated fake South Korean military ID images for a North Korean Kimsuky phishing campaign	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
Out of scope outputs	INC-038Bing’s chatbot was steered over a two-hour conversation into rule-breaking, destructive content	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
Code and structured data outputs	INC-040OpenAI’s ChatGPT generated fake South Korean military ID images for a North Korean Kimsuky phishing campaign	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
Biased output	INC-041Six major chatbots drafted phishing emails aimed at seniors in a Reuters test, and 11% of recipients clicked	PAP-014SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models

High-risk domain engagement

Risk	Related incident	Related paper
Medical advice	INC-034OpenAI’s ChatGPT reportedly suggested bromide as a salt substitute, and a man was hospitalized for 3 weeks	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
Legal advice	INC-044New York City’s MyCity business chatbot advised users to break the law	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
Financial advice	INC-044New York City’s MyCity business chatbot advised users to break the law	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
Crisis advice	INC-047Character.AI’s companion chatbot allegedly posed as a therapist and romantic partner to a teen who died by suicide	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
Therapy/psychological advice	INC-050Meta’s Instagram AI Studio chatbots claimed to be licensed therapists, inventing licence numbers and credentials	PAP-015TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law

Company defined risk

Risk	Related incident	Related paper
Prohibited topics	INC-046FoloToy’s Kumma AI teddy bear told researchers where to find knives and discussed sexual fetishes	PAP-016Safe-Child-LLM: A Developmental Benchmark for Evaluating LLM Safety in Child-LLM Interactions
Vulnerable users	INC-045Meta’s AI chatbot engaged teen test accounts on suicide, self-harm and eating disorders instead of offering help	PAP-016Safe-Child-LLM: A Developmental Benchmark for Evaluating LLM Safety in Child-LLM Interactions

Accountability

Transparency

Risk	Related incident	Related paper
Bypass AI disclosure	INC-048Bland AI’s voice agent told callers it was human, including when not instructed to	PAP-017RealityTest: How People Probe AI Identity and Whether Models Disclose It

Introduction

Context

Standard

Certification

Evals Taxonomy

Auditors

Legal

Data & privacy

IP infringement

Personal data leakage

Business data leakage

Reliability

Hallucination (incorrect generation)

Hallucination (incorrect information)

Incorrect tool call

Security

Unauthorized actions

Insecure outputs

Insecure action

Safety

Physical harm

Emotional harm

Inappropriate content

High-risk domain engagement

Company defined risk

Accountability

Transparency

​Data & privacy

​IP infringement

​Personal data leakage

​Business data leakage

​Reliability

​Hallucination (incorrect generation)

​Hallucination (incorrect information)

​Incorrect tool call

​Security

​Unauthorized actions

​Insecure outputs

​Insecure action

​Safety

​Physical harm

​Emotional harm

​Inappropriate content

​High-risk domain engagement

​Company defined risk

​Accountability

​Transparency

Data & privacy

IP infringement

Personal data leakage

Business data leakage

Reliability

Hallucination (incorrect generation)

Hallucination (incorrect information)

Incorrect tool call

Security

Unauthorized actions

Insecure outputs

Insecure action

Safety

Physical harm

Emotional harm

Inappropriate content

High-risk domain engagement

Company defined risk

Accountability

Transparency