> ## Documentation Index
> Fetch the complete documentation index at: https://standard.aiuc-1.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Risks

export const InlineClock = ({city, timeZone}) => {
  const [time, setTime] = useState('--:--:--');
  useEffect(() => {
    const update = () => setTime(new Intl.DateTimeFormat('en-US', {
      hour: '2-digit',
      minute: '2-digit',
      second: '2-digit',
      hour12: false,
      timeZone
    }).format(new Date()));
    update();
    const id = setInterval(update, 1000);
    return () => clearInterval(id);
  }, [timeZone]);
  return <div className="aiuc-footer-clock">
      <span className="aiuc-footer-clock-city">{city}</span>
      <span className="aiuc-footer-clock-time">{time}</span>
    </div>;
};

The risk categories AIUC-1 evals test an AI agent against, with the incidents and research behind each. This taxonomy covers the risks typically tested under AIUC-1. Each certified agent faces thousands of evals, focused on the subset of risks most relevant to its product and users. As new incidents and research surface new threats, the taxonomy is updated so testing stays grounded in real-world failures.

## Data & privacy

### IP infringement

<div className="aiuc-taxonomy-table" />

| Risk                      | Related incident                                                                                                                                                                                                                                                               | Related paper                                                                                                                                                                                                                                                               |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Copyrighted code          | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-001</span><span className="aiuc-taxonomy-cite-text">GPT-4 reproduced 100+ New York Times articles nearly verbatim when prompted with opening lines</span></div></div>                         | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-001</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2409.13831">Measuring Copyright Risks of Large Language Model via Partial Information Probing</a></div></div> |
| Copyrighted text          | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-002</span><span className="aiuc-taxonomy-cite-text">Perplexity's answer engine republished Forbes' paywalled investigation with near-verbatim text and minimal attribution</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-001</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2409.13831">Measuring Copyright Risks of Large Language Model via Partial Information Probing</a></div></div> |
| Trademark slogans         | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-001</span><span className="aiuc-taxonomy-cite-text">GPT-4 reproduced 100+ New York Times articles nearly verbatim when prompted with opening lines</span></div></div>                         | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-001</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2409.13831">Measuring Copyright Risks of Large Language Model via Partial Information Probing</a></div></div> |
| Trademarked brand names   | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-002</span><span className="aiuc-taxonomy-cite-text">Perplexity's answer engine republished Forbes' paywalled investigation with near-verbatim text and minimal attribution</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-001</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2409.13831">Measuring Copyright Risks of Large Language Model via Partial Information Probing</a></div></div> |
| Proprietary style mimicry | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-003</span><span className="aiuc-taxonomy-cite-text">Spotify's platform hosted an AI-generated King Gizzard impersonator for weeks before it was removed</span></div></div>                    | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-001</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2409.13831">Measuring Copyright Risks of Large Language Model via Partial Information Probing</a></div></div> |

### Personal data leakage

<div className="aiuc-taxonomy-table" />

| Risk                       | Related incident                                                                                                                                                                                                                                               | Related paper                                                                                                                                                                                                                                                          |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Personal identifiers       | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-005</span><span className="aiuc-taxonomy-cite-text">McDonald's AI hiring chatbot exposed up to 64M applicants' data</span></div></div>                                        | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-002</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2604.05432">Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use</a></div></div> |
| Sensitive personal records | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-006</span><span className="aiuc-taxonomy-cite-text">Nx's compromised npm package weaponized installed AI coding agents to exfiltrate developer credentials</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-002</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2604.05432">Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use</a></div></div> |
| Credentials                | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-028</span><span className="aiuc-taxonomy-cite-text">A malicious npm package posing as an OpenAI Codex UI exfiltrated developer authentication tokens</span></div></div>       | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-002</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2604.05432">Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use</a></div></div> |

### Business data leakage

<div className="aiuc-taxonomy-table" />

| Risk                         | Related incident                                                                                                                                                                                                                                      | Related paper                                                                                                                                                                                                                                                      |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| HR data                      | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-007</span><span className="aiuc-taxonomy-cite-text">Microsoft 365 Copilot was exploited via ASCII smuggling to exfiltrate user data</span></div></div>               | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-003</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2505.23817">System Prompt Extraction Attacks and Defenses in Large Language Models</a></div></div>   |
| Operational data             | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-007</span><span className="aiuc-taxonomy-cite-text">Microsoft 365 Copilot was exploited via ASCII smuggling to exfiltrate user data</span></div></div>               | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-004</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2512.16310">Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation</a></div></div> |
| Financial data               | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-008</span><span className="aiuc-taxonomy-cite-text">Anthropic's Claude 3.7 Sonnet had its full 24,000-token system prompt extracted and published</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-003</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2505.23817">System Prompt Extraction Attacks and Defenses in Large Language Models</a></div></div>   |
| Proprietary data (inc. code) | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-011</span><span className="aiuc-taxonomy-cite-text">GitHub's Copilot Chat was hijacked via prompt injection to leak private repo code</span></div></div>             | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-004</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2512.16310">Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation</a></div></div> |
| System prompt                | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-008</span><span className="aiuc-taxonomy-cite-text">Anthropic's Claude 3.7 Sonnet had its full 24,000-token system prompt extracted and published</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-003</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2505.23817">System Prompt Extraction Attacks and Defenses in Large Language Models</a></div></div>   |

## Reliability

### Hallucination (incorrect generation)

<div className="aiuc-taxonomy-table" />

| Risk                            | Related incident                                                                                                                                                                                                                                                            | Related paper                                                                                                                                                                                                                                                                                   |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Calls non-existent code         | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-012</span><span className="aiuc-taxonomy-cite-text">OpenAI's ChatGPT and Google's Gemini recommended a non-existent package that drew 30,000+ downloads once registered</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-006</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2406.10279">We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs</a></div></div> |
| Instruction non-compliance      | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-012</span><span className="aiuc-taxonomy-cite-text">OpenAI's ChatGPT and Google's Gemini recommended a non-existent package that drew 30,000+ downloads once registered</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-005</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2401.03601">InFoBench: Evaluating Instruction Following Ability in Large Language Models</a></div></div>                          |
| Inter-deliverable inconsistency | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-012</span><span className="aiuc-taxonomy-cite-text">OpenAI's ChatGPT and Google's Gemini recommended a non-existent package that drew 30,000+ downloads once registered</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-005</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2401.03601">InFoBench: Evaluating Instruction Following Ability in Large Language Models</a></div></div>                          |

### Hallucination (incorrect information)

<div className="aiuc-taxonomy-table" />

| Risk                        | Related incident                                                                                                                                                                                                                                                     | Related paper                                                                                                                                                                                                                    |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Outdated facts              | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-013</span><span className="aiuc-taxonomy-cite-text">Air Canada's chatbot invented a refund policy, and the airline was held liable</span></div></div>                               | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-007</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2504.17550">HalluLens: LLM Hallucination Benchmark</a></div></div> |
| Premature assertions        | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-014</span><span className="aiuc-taxonomy-cite-text">Apple's Intelligence notification summaries misrepresented BBC headlines, forcing Apple to pause the feature</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-007</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2504.17550">HalluLens: LLM Hallucination Benchmark</a></div></div> |
| Ground truth contradictions | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-013</span><span className="aiuc-taxonomy-cite-text">Air Canada's chatbot invented a refund policy, and the airline was held liable</span></div></div>                               | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-007</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2504.17550">HalluLens: LLM Hallucination Benchmark</a></div></div> |
| Misleading facts            | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-014</span><span className="aiuc-taxonomy-cite-text">Apple's Intelligence notification summaries misrepresented BBC headlines, forcing Apple to pause the feature</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-007</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2504.17550">HalluLens: LLM Hallucination Benchmark</a></div></div> |
| Misleading confidence       | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-015</span><span className="aiuc-taxonomy-cite-text">Sullivan & Cromwell apologized to a bankruptcy judge for AI hallucinations in a Chapter 15 filing</span></div></div>            | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-007</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2504.17550">HalluLens: LLM Hallucination Benchmark</a></div></div> |
| Misleading citations        | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-015</span><span className="aiuc-taxonomy-cite-text">Sullivan & Cromwell apologized to a bankruptcy judge for AI hallucinations in a Chapter 15 filing</span></div></div>            | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-007</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2504.17550">HalluLens: LLM Hallucination Benchmark</a></div></div> |
| Self-contradiction          | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-016</span><span className="aiuc-taxonomy-cite-text">Google's AI Overviews falsely told searchers a Minnesota solar firm was being sued</span></div></div>                           | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-007</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2504.17550">HalluLens: LLM Hallucination Benchmark</a></div></div> |

### Incorrect tool call

<div className="aiuc-taxonomy-table" />

| Risk                        | Related incident                                                                                                                                                                                                                                  | Related paper                                                                                                                                                                                                                                     |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Incorrectly executed action | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-019</span><span className="aiuc-taxonomy-cite-text">McDonald's IBM-built AI drive-thru misinterpreted spoken orders, ending the trial</span></div></div>         | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-008</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2503.14227">Benchmarking Failures in Tool-Augmented Language Models</a></div></div> |
| Unprompted action           | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-020</span><span className="aiuc-taxonomy-cite-text">Sakana AI's research agent rewrote its own code to remove a runtime limit it kept hitting</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-008</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2503.14227">Benchmarking Failures in Tool-Augmented Language Models</a></div></div> |
| Incomplete action           | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-019</span><span className="aiuc-taxonomy-cite-text">McDonald's IBM-built AI drive-thru misinterpreted spoken orders, ending the trial</span></div></div>         | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-008</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2503.14227">Benchmarking Failures in Tool-Augmented Language Models</a></div></div> |

## Security

### Unauthorized actions

<div className="aiuc-taxonomy-table" />

| Risk                                    | Related incident                                                                                                                                                                                                                                                       | Related paper                                                                                                                                                                                                                                    |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Unauthorized transaction                | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-022</span><span className="aiuc-taxonomy-cite-text">Chevrolet of Watsonville's ChatGPT-powered chatbot was manipulated into 'selling' a $76,000 Tahoe for $1</span></div></div>       | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-009</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2412.14470">Agent-SafetyBench: Evaluating the Safety of LLM Agents</a></div></div> |
| Unauthorized communication              | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-036</span><span className="aiuc-taxonomy-cite-text">DPD's customer-service chatbot swore at a customer and disparaged the company</span></div></div>                                  | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-009</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2412.14470">Agent-SafetyBench: Evaluating the Safety of LLM Agents</a></div></div> |
| Unauthorized system actions             | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-029</span><span className="aiuc-taxonomy-cite-text">A hidden prompt injection in a Word document turned Microsoft Copilot for Word into a self-propagating AI worm</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-009</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2412.14470">Agent-SafetyBench: Evaluating the Safety of LLM Agents</a></div></div> |
| Unauthorized cross-system data transfer | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-022</span><span className="aiuc-taxonomy-cite-text">Chevrolet of Watsonville's ChatGPT-powered chatbot was manipulated into 'selling' a $76,000 Tahoe for $1</span></div></div>       | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-009</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2412.14470">Agent-SafetyBench: Evaluating the Safety of LLM Agents</a></div></div> |

### Insecure outputs

<div className="aiuc-taxonomy-table" />

| Risk                    | Related incident                                                                                                                                                                                                                                                       | Related paper                                                                                                                                                                                                                                                                            |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Hyperlinks (URLs)       | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-030</span><span className="aiuc-taxonomy-cite-text">OpenAI's ChatGPT recommended cloned scam websites to shoppers</span></div></div>                                                  | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-010</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2506.05692">SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code</a></div></div> |
| Code patterns or blocks | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-029</span><span className="aiuc-taxonomy-cite-text">A hidden prompt injection in a Word document turned Microsoft Copilot for Word into a self-propagating AI worm</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-010</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2506.05692">SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code</a></div></div> |

### Insecure action

<div className="aiuc-taxonomy-table" />

| Risk                  | Related incident                                                                                                                                                                                                                            | Related paper                                                                                                                                                                                                                                              |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Install insecure code | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-031</span><span className="aiuc-taxonomy-cite-text">Amazon's Q Developer VS Code extension shipped a hacker-injected data-wiping prompt</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-011</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2506.14866">OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents</a></div></div> |
| Fatal code execution  | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-031</span><span className="aiuc-taxonomy-cite-text">Amazon's Q Developer VS Code extension shipped a hacker-injected data-wiping prompt</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-011</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2506.14866">OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents</a></div></div> |

## Safety

### Physical harm

<div className="aiuc-taxonomy-table" />

| Risk                      | Related incident                                                                                                                                                                                                                                                         | Related paper                                                                                                                                                                                                                                                                                              |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Dangerous instructions    | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-032</span><span className="aiuc-taxonomy-cite-text">Discord's Clyde chatbot was tricked into outputting weapon instructions via a roleplay jailbreak</span></div></div>                 | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-012</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2505.19933">Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making</a></div></div> |
| Medically unsafe guidance | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-034</span><span className="aiuc-taxonomy-cite-text">OpenAI's ChatGPT reportedly suggested bromide as a salt substitute, and a man was hospitalized for 3 weeks</span></div></div>       | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-012</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2505.19933">Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making</a></div></div> |
| Self-harm content         | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-045</span><span className="aiuc-taxonomy-cite-text">Meta's AI chatbot engaged teen test accounts on suicide, self-harm and eating disorders instead of offering help</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-012</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2505.19933">Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making</a></div></div> |
| Suicide content           | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-035</span><span className="aiuc-taxonomy-cite-text">OpenAI's ChatGPT allegedly validated a 16-year-old's suicidal thoughts over months, and his parents sued</span></div></div>         | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-012</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2505.19933">Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making</a></div></div> |

### Emotional harm

<div className="aiuc-taxonomy-table" />

| Risk                          | Related incident                                                                                                                                                                                                                     | Related paper                                                                                                                                                                                                                                     |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Hostile / aggressive behavior | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-037</span><span className="aiuc-taxonomy-cite-text">Google's Gemini told a college student seeking homework help to 'please die'</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-013</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2508.09998">INTIMA: A Benchmark for Human-AI Companionship Behavior</a></div></div> |
| Triggering content            | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-037</span><span className="aiuc-taxonomy-cite-text">Google's Gemini told a college student seeking homework help to 'please die'</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-013</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2508.09998">INTIMA: A Benchmark for Human-AI Companionship Behavior</a></div></div> |

### Inappropriate content

<div className="aiuc-taxonomy-table" />

| Risk                             | Related incident                                                                                                                                                                                                                                                     | Related paper                                                                                                                                                                                                                                                                      |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Harassment                       | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-037</span><span className="aiuc-taxonomy-cite-text">Google's Gemini told a college student seeking homework help to 'please die'</span></div></div>                                 | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-014</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2402.05044">SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models</a></div></div> |
| Hate speech                      | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-042</span><span className="aiuc-taxonomy-cite-text">OpenAI's Sora 2 app generated violent, racist and antisemitic videos within weeks of its launch</span></div></div>              | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-014</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2402.05044">SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models</a></div></div> |
| Sexual content excluding minors  | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-046</span><span className="aiuc-taxonomy-cite-text">FoloToy's Kumma AI teddy bear told researchers where to find knives and discussed sexual fetishes</span></div></div>            | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-014</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2402.05044">SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models</a></div></div> |
| Violent content                  | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-042</span><span className="aiuc-taxonomy-cite-text">OpenAI's Sora 2 app generated violent, racist and antisemitic videos within weeks of its launch</span></div></div>              | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-014</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2402.05044">SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models</a></div></div> |
| Illegal conduct advice           | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-040</span><span className="aiuc-taxonomy-cite-text">OpenAI's ChatGPT generated fake South Korean military ID images for a North Korean Kimsuky phishing campaign</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-014</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2402.05044">SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models</a></div></div> |
| Out of scope outputs             | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-038</span><span className="aiuc-taxonomy-cite-text">Bing's chatbot was steered over a two-hour conversation into rule-breaking, destructive content</span></div></div>              | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-014</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2402.05044">SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models</a></div></div> |
| Code and structured data outputs | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-040</span><span className="aiuc-taxonomy-cite-text">OpenAI's ChatGPT generated fake South Korean military ID images for a North Korean Kimsuky phishing campaign</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-014</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2402.05044">SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models</a></div></div> |
| Biased output                    | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-041</span><span className="aiuc-taxonomy-cite-text">Six major chatbots drafted phishing emails aimed at seniors in a Reuters test, and 11% of recipients clicked</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-014</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2402.05044">SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models</a></div></div> |

### High-risk domain engagement

<div className="aiuc-taxonomy-table" />

| Risk                         | Related incident                                                                                                                                                                                                                                                           | Related paper                                                                                                                                                                                                                                            |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Medical advice               | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-034</span><span className="aiuc-taxonomy-cite-text">OpenAI's ChatGPT reportedly suggested bromide as a salt substitute, and a man was hospitalized for 3 weeks</span></div></div>         | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-015</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2507.21134">TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law</a></div></div> |
| Legal advice                 | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-044</span><span className="aiuc-taxonomy-cite-text">New York City's MyCity business chatbot advised users to break the law</span></div></div>                                             | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-015</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2507.21134">TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law</a></div></div> |
| Financial advice             | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-044</span><span className="aiuc-taxonomy-cite-text">New York City's MyCity business chatbot advised users to break the law</span></div></div>                                             | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-015</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2507.21134">TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law</a></div></div> |
| Crisis advice                | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-047</span><span className="aiuc-taxonomy-cite-text">Character.AI's companion chatbot allegedly posed as a therapist and romantic partner to a teen who died by suicide</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-015</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2507.21134">TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law</a></div></div> |
| Therapy/psychological advice | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-050</span><span className="aiuc-taxonomy-cite-text">Meta's Instagram AI Studio chatbots claimed to be licensed therapists, inventing licence numbers and credentials</span></div></div>   | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-015</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2507.21134">TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law</a></div></div> |

### Company defined risk

<div className="aiuc-taxonomy-table" />

| Risk              | Related incident                                                                                                                                                                                                                                                         | Related paper                                                                                                                                                                                                                                                                           |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Prohibited topics | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-046</span><span className="aiuc-taxonomy-cite-text">FoloToy's Kumma AI teddy bear told researchers where to find knives and discussed sexual fetishes</span></div></div>                | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-016</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2506.13510">Safe-Child-LLM: A Developmental Benchmark for Evaluating LLM Safety in Child-LLM Interactions</a></div></div> |
| Vulnerable users  | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-045</span><span className="aiuc-taxonomy-cite-text">Meta's AI chatbot engaged teen test accounts on suicide, self-harm and eating disorders instead of offering help</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-016</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2506.13510">Safe-Child-LLM: A Developmental Benchmark for Evaluating LLM Safety in Child-LLM Interactions</a></div></div> |

## Accountability

### Transparency

<div className="aiuc-taxonomy-table" />

| Risk                 | Related incident                                                                                                                                                                                                                           | Related paper                                                                                                                                                                                                                                                      |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Bypass AI disclosure | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">INC-048</span><span className="aiuc-taxonomy-cite-text">Bland AI's voice agent told callers it was human, including when not instructed to</span></div></div> | <div className="aiuc-taxonomy-cite"><div><span className="aiuc-taxonomy-ref">PAP-017</span><a className="aiuc-taxonomy-cite-link" href="https://arxiv.org/abs/2606.00168">RealityTest: How People Probe AI Identity and Whether Models Disclose It</a></div></div> |

<div className="aiuc-footer">
  <span className="aiuc-footer-corner aiuc-footer-corner-tl">
    <svg fill="none" stroke="currentColor" strokeWidth="1" viewBox="0 0 12 12" width="12" height="12">
      <line x1="0" x2="12" y1="6" y2="6" />

      <line x1="6" x2="6" y1="0" y2="12" />
    </svg>
  </span>

  <span className="aiuc-footer-corner aiuc-footer-corner-tr">
    <svg fill="none" stroke="currentColor" strokeWidth="1" viewBox="0 0 12 12" width="12" height="12">
      <line x1="0" x2="12" y1="6" y2="6" />

      <line x1="6" x2="6" y1="0" y2="12" />
    </svg>
  </span>

  <div className="aiuc-footer-strip">
    <span className="aiuc-footer-mono">37.782274° N -122.392147° W</span>
    <span className="aiuc-footer-strip-center">FIG. A (SITE INDEX)</span>

    <span />
  </div>

  <div className="aiuc-footer-row-main">
    <div className="aiuc-footer-wireframe-cell">
      <svg className="aiuc-footer-wireframe" fill="none" stroke="currentColor" strokeWidth="0.4" viewBox="0 0 200 150">
        <rect height="130" width="180" x="10" y="10" />

        <rect height="40" width="60" x="20" y="20" />

        <rect height="40" width="40" x="90" y="20" />

        <rect height="40" width="40" x="140" y="20" />

        <rect height="60" width="60" x="20" y="70" />

        <rect height="60" width="90" x="90" y="70" />

        <line strokeDasharray="2,2" x1="20" x2="180" y1="65" y2="65" />

        <line strokeDasharray="2,2" x1="85" x2="85" y1="20" y2="60" />

        <circle cx="50" cy="40" r="6" />

        <circle cx="110" cy="40" r="6" />

        <circle cx="160" cy="40" r="6" />
      </svg>
    </div>

    <div className="aiuc-footer-wordmark-cell">
      <div className="aiuc-footer-wordmark">Artificial Intelligence Underwriting Company</div>
    </div>

    <div className="aiuc-footer-clocks">
      <InlineClock city="SFO" timeZone="America/Los_Angeles" />

      <InlineClock city="NYC" timeZone="America/New_York" />

      <InlineClock city="LON" timeZone="Europe/London" />
    </div>
  </div>

  <div className="aiuc-footer-row-sub">
    <div className="aiuc-footer-codeblock-cell">
      <div className="aiuc-footer-codeblock">
        <span className="aiuc-footer-codeblock-header">Code</span>
        <span className="aiuc-footer-codeblock-header">Structural unit</span>
        <span className="aiuc-footer-codeblock-code">a.</span>
        <span className="aiuc-footer-codeblock-text">AIUC-1 requirements for agent data, privacy, security, safety, reliability, accountability, and societal risk.</span>
        <span className="aiuc-footer-codeblock-code">b.</span>
        <span className="aiuc-footer-codeblock-text">Evidence templates for technical implementation, legal policy, operational practice, and third-party evaluation.</span>
        <span className="aiuc-footer-codeblock-code">c.</span>
        <span className="aiuc-footer-codeblock-text">Crosswalks to AI regulations, standards, and security frameworks.</span>
        <span className="aiuc-footer-codeblock-code">d.</span>
        <span className="aiuc-footer-codeblock-text">Quarterly updates shaped by enterprise adoption, risk, regulation, and community input.</span>
      </div>
    </div>

    <div className="aiuc-footer-columns-cell">
      <div className="aiuc-footer-columns">
        <div>
          <div className="aiuc-footer-column-header">I. Standard</div>

          <ul className="aiuc-footer-column-list">
            <li><a className="aiuc-footer-column-link" href="/">Overview</a></li>
            <li><a className="aiuc-footer-column-link" href="/crosswalks">Crosswalks</a></li>
            <li><a className="aiuc-footer-column-link" href="/evidence">Evidence</a></li>
            <li><a className="aiuc-footer-column-link" href="/changelog">Changelog</a></li>
          </ul>
        </div>

        <div>
          <div className="aiuc-footer-column-header">II. Learn</div>

          <ul className="aiuc-footer-column-list">
            <li><a className="aiuc-footer-column-link" href="/learn/about">About AIUC-1</a></li>
            <li><a className="aiuc-footer-column-link" href="/learn/contribute">Contribute</a></li>
            <li><a className="aiuc-footer-column-link" href="/scoping">Scoping</a></li>
            <li><a className="aiuc-footer-column-link" href="/faq">FAQ</a></li>
          </ul>
        </div>

        <div>
          <div className="aiuc-footer-column-header">III. Office</div>

          <ul className="aiuc-footer-column-list">
            <li><a className="aiuc-footer-column-link" href="/consortium">Consortium</a></li>
            <li><a className="aiuc-footer-column-link" href="https://www.aiuc-1.com/contact">Contact</a></li>
            <li><a className="aiuc-footer-column-link" href="/legal/privacy">Privacy policy</a></li>
            <li><a className="aiuc-footer-column-link" href="/legal/terms">Terms of use</a></li>
          </ul>
        </div>
      </div>
    </div>
  </div>

  <div className="aiuc-footer-strip-bottom">
    <span className="aiuc-footer-mono">100</span>
    <span>© AIUC — ALL RIGHTS RESERVED</span>
  </div>

  <span className="aiuc-footer-corner aiuc-footer-corner-bl">
    <svg fill="none" stroke="currentColor" strokeWidth="1" viewBox="0 0 12 12" width="12" height="12">
      <line x1="0" x2="12" y1="6" y2="6" />

      <line x1="6" x2="6" y1="0" y2="12" />
    </svg>
  </span>

  <span className="aiuc-footer-corner aiuc-footer-corner-br">
    <svg fill="none" stroke="currentColor" strokeWidth="1" viewBox="0 0 12 12" width="12" height="12">
      <line x1="0" x2="12" y1="6" y2="6" />

      <line x1="6" x2="6" y1="0" y2="12" />
    </svg>
  </span>
</div>