💻 computer science

A Closer Look at the Existing Risks of Generative AI: Mapping the Who, What, and How of Real-World Incidents

This paper presents a new taxonomy for Generative AI failures by systematically analyzing 499 real-world incidents to reveal that most harms stem from use-related issues affecting stakeholders beyond end-users, thereby distinguishing the Generative AI risk landscape from traditional AI and advocating for non-technical mitigation strategies like public disclosure, education, and regulation.

Original authors: Megan Li, Wendy Bickersteth, Ningjing Tang, Jason Hong, Lorrie Cranor, Hong Shen, Hoda Heidari

Published 2026-04-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Megan Li, Wendy Bickersteth, Ningjing Tang, Jason Hong, Lorrie Cranor, Hong Shen, Hoda Heidari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Generative AI as a super-charged, hyper-creative factory that can instantly produce text, images, audio, and videos. It's like a magical printing press that never sleeps. But just like any powerful factory, it doesn't just produce goods; it also produces "accidents" and "messes."

This paper is essentially a forensic investigation into 499 real-world accidents caused by this factory. The researchers (from Carnegie Mellon University) didn't just guess what could go wrong; they looked at a massive pile of police reports, news stories, and public complaints to map out exactly what happened, who got hurt, and why.

Here is the breakdown of their findings in simple terms:

1. The Big Surprise: The "Bystander" Problem

In most safety stories, we assume the person who gets hurt is the one using the tool. If you drive a car recklessly, you are the one who crashes.

But with Generative AI, the researchers found the opposite is often true.

The Analogy: Imagine you are at a party. You decide to use a magic paintbrush to draw a funny picture of your friend. You (the user) think it's hilarious. But the person in the picture (the bystander) is now being mocked by the whole town, or their reputation is ruined.
The Finding: The study found that most of the harm didn't happen to the person using the AI. It happened to people who had nothing to do with it—strangers, communities, or society at large. The "users" often got the benefits (fun, productivity), while the "non-users" got the bruises (scams, identity theft, confusion).

2. The Two Main Culprits: "The Glitch" vs. "The Villain"

The researchers categorized the reasons these accidents happened into two main buckets:

The Glitch (Technical Failures): Sometimes the AI just gets it wrong. It hallucinates (makes up facts), gets confused, or fails to follow rules.
- Who gets hurt? Usually the person using the AI. For example, a lawyer uses AI to write a court brief, the AI makes up a fake case, and the lawyer gets in trouble.
The Villain (Malicious Use): This was the biggest surprise. The most common cause of harm wasn't the AI being broken; it was people using a working AI to do bad things on purpose.
- Who gets hurt? Almost always the bystanders.
- Examples: Someone using AI to create a fake deepfake video of a politician to cause a riot, or creating fake nude images of a classmate to bully them. The AI worked exactly as designed; the human behind it was the problem.

3. The "Who, What, and How" Map

The paper built a new map (a taxonomy) to organize these incidents:

What happened (The Harm): The most common types of harm were to people's Autonomy (being tricked or impersonated), Politics/Economy (fake news swaying elections or markets), and Reputation (defamation).
How it happened (The Failure Mode): The most frequent cause was Malicious Use (intentional bad actors). The second most common was Undisclosed Use (using AI when you're supposed to be doing human work, like a student cheating or a magazine getting flooded with AI spam).
Who got hurt: As mentioned, the "non-interacting" people (the public, victims of deepfakes) bore the brunt of the damage.

4. Why Current Rules Aren't Working

The paper argues that our current way of thinking about safety is broken because it assumes the person using the tool is the only one who needs protection.

The Analogy: It's like having a safety manual for a hammer that only tells you how not to hit your own thumb, but says nothing about not hitting your neighbor with it.
The Reality: Because the people using the AI often don't suffer the consequences (the victim does), they have no motivation to be careful. The "benefit" goes to the user, but the "risk" goes to the stranger.

5. What Should We Do?

The authors suggest three main things to fix this mess, focusing on people and rules rather than just better code:

Teach People (AI Literacy): We need to teach the general public how this "factory" works. If people understand that AI can lie (hallucinate) or be used for scams, they won't blindly trust it. This stops some "accidents" caused by ignorance.
Regulate the "Open Source" Wild West: The paper notes that many of the worst tools (like apps that create fake nude images) are built on "open source" models that anyone can download and tweak. The current rules let these tools exist with few restrictions. The authors suggest we need stricter rules here to stop the "villains" from building their weapons.
Track the "Paper Trail" (Provenance): We need better ways to tell what is real and what is AI. If we can tag AI content (like a "Made by AI" label) and detect fakes, it becomes harder for scammers to spread lies or for deepfakes to ruin reputations.

The Bottom Line

Generative AI isn't just a tool that breaks occasionally; it's a tool that is frequently weaponized by humans to hurt people who didn't even ask for it. The solution isn't just making the AI "smarter"; it's about changing how we regulate it, teaching the public how to spot the tricks, and realizing that the people using the tech aren't the only ones who need protection.

1. Problem Statement

While significant attention is paid to the existential risks of Artificial Intelligence (AI), there is a critical gap in understanding the existing risks—those already observed in the real world causing harm to individuals, communities, and society.

The Gap: Existing risk taxonomies often lack grounding in empirical evidence, failing to specify who is affected, how the harm arises (the failure mode), and the specific relationship between technical failures and downstream societal impacts.
The Specific Challenge: Generative AI (GenAI) differs from traditional AI in its general-purpose nature and output modalities (text, image, audio, video). Current frameworks do not adequately capture the unique landscape of GenAI failures, particularly the disconnect between the users who benefit from the technology and the third parties who suffer the harms.
Goal: To construct a data-driven taxonomy of GenAI failures and harms, mapping the "Who" (stakeholders), "What" (harm types), and "How" (sociotechnical failure modes) based on real-world incidents.

2. Methodology

The authors conducted a systematic empirical analysis of 499 publicly reported Generative AI incidents.

Data Sources: The dataset was curated from two major repositories: the AI Incident Database (AIID) and the AI, Algorithmic and Automation Incidents and Controversies (AIAAIC). The OECD AI Incident Monitor was excluded due to its automated curation and broader definition of "incident."
Selection Criteria: Only incidents where a Generative AI system was the primary cause of the event were included.
Coding Framework: Each incident was treated as a case study and coded for four attributes:
1. Modality: Output type (text, image, audio, video, multimodal).
2. Affected Stakeholders: Categorized into Interacting (end users, developers) and Non-Interacting (third-party individuals, organizations, communities, society).
3. Harm: The nature of the negative outcome.
4. Sociotechnical Failure Mode: The proximal cause (technical, human, or systemic).
Taxonomy Development:
- Harms: Built upon existing taxonomies (Weidinger et al., Abercrombie et al.) but revised to include a new category, "Overburdening ecosystems" (flooding human-curated spaces with AI content). The final taxonomy contains 41 specific harms across 12 categories.
- Failure Modes: Developed a novel taxonomy of 14 failure modes organized into 4 categories:
  1. Design-related (e.g., problematic data collection).
  2. Development/Evaluation-related (e.g., hallucinations, lack of robustness).
  3. Release-related (e.g., lack of transparency).
  4. Use-related (e.g., malicious use, improper use).
Analysis Process: Two authors independently coded the data, resolving disagreements through consensus. They analyzed prevalence, co-occurrence of failure modes, and the mapping between failure modes, harms, and stakeholders.

3. Key Contributions

Empirical Taxonomy of GenAI Harms: A validated, quantitative mapping of 41 specific harms observed in the wild, moving beyond theoretical risk assessment.
Novel Taxonomy of Sociotechnical Failure Modes: A structured framework distinguishing between design, development, release, and use-related failures, specifically tailored to GenAI.
The "Who-What-How" Mapping: A comprehensive analysis linking specific failure modes to specific harms and, crucially, identifying the disparity between beneficiaries and victims.
Differentiation from Traditional AI: Evidence demonstrating that the landscape of GenAI harms is distinct from traditional AI, particularly regarding the prevalence of malicious use and the impact on non-interacting stakeholders.

4. Key Results & Findings

A. Who is Harmed? (The Stakeholder Disparity)

Non-Interacting Stakeholders are Primary Victims: The majority of incidents (approx. 80%+) caused harm to parties who did not directly interact with the AI system (e.g., individuals targeted by deepfakes, organizations receiving AI spam, society facing misinformation).
Interacting Stakeholders are Rarely Harmed: End users and developers rarely suffer the direct consequences of the harms they generate.
Accountability Breakdown: This creates a misalignment where those who reap the benefits (users/developers) assume little risk, while those who bear the risks (society/third parties) have no direct recourse.

B. What Harms Occur? (Prevalence)

Top Harm Categories:
1. Autonomy/Agency Loss (n=178): Often linked to impersonation and identity theft.
2. Political & Economic (n=106): Linked to manipulation and instability.
3. Informational (n=84): Pollution of the information ecosystem.
Specific High-Frequency Harms: Impersonation/identity theft (n=108), Political manipulation (n=95), and Loss of confidence/trust (n=58).

C. How do Harms Arise? (Failure Modes)

Use-Related Failures Dominate: Use-related failure modes account for the vast majority of incidents (n=301), significantly outnumbering design or development failures.
Malicious Use is the Primary Driver: Malicious use (n=198) is the single most prevalent failure mode, driving harms related to autonomy, political manipulation, psychological distress, and financial fraud.
Hallucinations: While less frequent than malicious use overall, hallucinations (n=56) are the primary failure mode affecting interacting stakeholders (e.g., users relying on false legal advice).

D. Correlations and Modalities

Failure-Harm Pairing:
- Malicious Use $\rightarrow$ Autonomy, Political, Psychological, and Financial harms (often via deepfakes).
- Hallucination $\rightarrow$ Human Rights/Civil Liberties harms (e.g., false evidence in court).
- Problematic Data Collection $\rightarrow$ Privacy & Security harms.
Modality Impact: Text-based systems are most associated with hallucinations. Image, video, and audio systems are overwhelmingly associated with Malicious Use (e.g., non-consensual deepfake pornography, voice scams).
Open-Source Models: The study corroborates that tools enabling non-consensual intimate imagery (AIG-NCII) are predominantly built on open-source models (e.g., Stable Diffusion), highlighting a regulatory gap.

5. Significance and Recommendations

Significance

The paper challenges the prevailing narrative that AI risks are primarily technical or future-oriented. It proves that current GenAI risks are largely sociotechnical, driven by human intent (malicious use) and systemic design flaws (data collection), and that the burden of these risks falls disproportionately on innocent third parties. This shifts the focus from purely "model safety" to "ecosystem safety."

Recommendations

Prioritize Non-Interacting Stakeholders: Policy and governance must shift focus to protect third parties who do not interact with the system but suffer its consequences.
Public Education & AI Literacy: Since "Improper Use" is a major failure mode, the public needs better education on GenAI capabilities and limitations (e.g., understanding hallucinations) to reduce accidental harm.
Regulatory Stance on Open-Source: The current regulatory environment (e.g., EU AI Act) incentivizes open-source development but fails to mitigate the specific risks of misuse (like AIG-NCII). The authors call for evidence-based regulation that balances innovation with the prevention of abusive tools.
Provenance and Detection: Implementing data tracking and synthetic content detection is necessary to mitigate harms like misinformation and deepfake fraud, though these must be paired with broader accountability measures.
Shift from Technical to Non-Technical Mitigation: The authors argue that technical fixes (guardrails) are insufficient. Effective mitigation requires public disclosures, education, and careful regulatory stances to address the human and systemic layers of the problem.