💻 computer science

A pragmatic classification of AI incident trajectories

This paper proposes a pragmatic framework that disentangles reporting trends, deployment growth, and actual harm rates to classify AI incident trajectories, thereby providing a clearer basis for governance decisions amidst the uncertainties of current public incident databases.

Original authors: Isaak Mengesha, Branwen Owen, Charlie Collins, Tina Wong, Simon Mylius, Peter Slattery, Sean McGregor

Published 2026-04-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Isaak Mengesha, Branwen Owen, Charlie Collins, Tina Wong, Simon Mylius, Peter Slattery, Sean McGregor

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the mayor of a city, and you want to know if the traffic is getting safer or more dangerous.

Every day, your news team sends you a list of car accidents.

Last year: 10 accidents.
This year: 100 accidents.

Your first instinct might be to scream, "The roads are becoming a death trap! We need to ban cars!"

But wait. What if last year, only 100 cars were on the road, and this year, 10,000 cars are on the road? Suddenly, 100 accidents might actually mean the roads are safer per car, even though the total number of crashes went up.

This is the exact problem with Artificial Intelligence (AI) right now. We have a growing list of AI "accidents" (harmful events), but we don't know if AI is actually getting more dangerous, or if we are just using it more, or if the news is just reporting more stories.

This paper proposes a new way to look at AI safety, borrowing ideas from epidemiology (the study of disease spread) to separate the "noise" from the "signal."

The Problem: The "Raw Count" Trap

Currently, people look at the total number of AI incidents and assume that a rising number means AI is getting worse. The authors say this is like looking at a rising number of flu cases and assuming the virus is mutating to become deadlier, without realizing that maybe just more people are getting tested this year.

They identify five big mistakes people make:

Confusing noise with signal: Is the problem getting worse, or are we just hearing about it more?
Mixing up "more use" with "more danger": If 1,000 people use a tool and 10 get hurt, that's different than if 10 people use it and 10 get hurt.
Assuming silence means safety: Just because we haven't heard about a problem doesn't mean it doesn't exist.
Trusting shaky numbers: We often don't have perfect data, but we act like we do.
Faking precision: We try to give exact numbers when we should just admit, "It's going up," or "It's going down."

The Solution: The "SORT" Framework

To fix this, the authors created a method called SORT. Think of this as a recipe for asking the right question before you panic.

Instead of asking, "Is AI bad?" (which is too vague), you ask a specific question broken into four parts:

S (Subject): Who is at risk? (e.g., Teenagers, drivers, patients).
O (Opportunity): Who is actually exposed? (e.g., Teenagers using a specific chatbot, drivers on a specific highway).
R (Risk Event): What is the specific harm? (e.g., Suicide encouragement, car crashes).
T (Timeframe): Over what period? (e.g., Per year, per million miles).

The Analogy:
Imagine you are a doctor. You don't just ask, "Are people getting sick?" You ask, "Among children (S) who swim in public pools (O), how many get ear infections (R) per summer (T)?" This gives you a clear, measurable target.

The Process: Estimating the Unknown

Since we don't have perfect data (like a perfect census of every AI user), the authors suggest a "Tiered" approach, like a detective gathering clues:

Tier 1 (The Gold Standard): We have a direct report (like a mandatory police report).
Tier 2 (The Proxy): We don't have the exact number, but we have a close guess. (e.g., If we don't know how many people used a specific AI, we might guess based on how many people used the company's app in general).
Tier 3 (The Expert Guess): We ask a panel of experts, "What's a reasonable range?"
Tier 4 (The Honest "I Don't Know"): We admit we have no data. This is better than making up a number.

The Result: The Four "Trajectories"

Once you have your best guesses for Exposure (how many people are using it) and Harm (how many are getting hurt), you plot them on a simple 2x2 grid. This tells you the "trajectory" of the risk:

Escalating (🔥 Danger!): More people are using it, AND the chance of getting hurt per person is going up.
- Action: Panic mode. Stop deployment, investigate immediately.
Mitigating (🛡️ Good News): More people are using it, BUT the chance of getting hurt per person is going down.
- Action: The safety guards are working! Keep going, but watch closely.
Concentrating (⚠️ Targeted Danger): Fewer people are using it, but the ones who do are getting much worse hurt.
- Action: This is a specific, high-risk group. Protect them specifically.
Receding (🌤️ Safe): Fewer people are using it, and the danger per person is dropping.
- Action: Relax. The problem is solving itself.

Real-World Examples from the Paper

Example 1: AI Chatbots and Self-Harm

The Situation: More people are using AI for emotional support (Exposure is UP).
The Harm: The number of times AI encourages self-harm is also rising sharply (Harm per person is UP).
The Verdict: Escalating. This is a fire. We need to fix the AI models immediately.

Example 2: Self-Driving Cars

The Situation: Self-driving cars are driving twice as many miles as last year (Exposure is UP).
The Harm: The number of accidents is rising, but not as fast as the miles driven. So, the accident rate per mile is actually going down (Harm per person is DOWN).
The Verdict: Mitigating. The technology is getting safer as it scales up, even though the total number of crashes is higher because there are more cars on the road.

Why This Matters

This paper is a call to stop panicking over raw numbers and start thinking like scientists.

For Policymakers: It stops you from banning a technology just because the news is loud. It helps you focus on the real risks.
For the Public: It helps you understand that "more news stories" doesn't always mean "more danger."
For the Future: It creates a shared language. Instead of arguing about whether AI is "good" or "bad," we can agree on specific questions: "Is the risk per user going up or down?"

In short, this framework is a filter. It takes the messy, confusing, and often scary news about AI, runs it through a structured lens, and tells us exactly where to focus our energy to keep everyone safe.

1. Problem Statement

The paper addresses a critical flaw in current AI governance: the reliance on raw incident counts from public databases (often news-sourced) to assess AI safety. The authors argue that these raw counts are misleading because they conflate three distinct factors:

Reporting Propensity: The likelihood of an incident being observed and reported (subject to media bias, Anglophone dominance, and acute vs. chronic harm visibility).
Exposure: The scale of system deployment and usage.
Hazard (Harm Rate): The frequency of harm per unit of exposure.

Key Issues Identified:

Conflation: Rising incident counts may simply reflect increased deployment or media attention, not increased danger per use.
Data Scarcity & Bias: Mandatory reporting frameworks (e.g., EU AI Act, California SB-53) are narrow, fragmented, and often exclude non-catastrophic harms.
Lack of Methodology: There is no standardized framework to separate exposure from hazard or to handle sparse data without falsely assuming an absence of harm.
Epistemic Overconfidence: Current analyses often treat point estimates as precise facts, ignoring the uncertainty inherent in incomplete data.

2. Methodology: The Framework

The authors propose a structured methodology inspired by epidemiological surveillance to interpret AI incident data. The process consists of three main stages:

A. Structured Monitoring Questions (SORT)

To ensure comparability and reproducibility, the framework requires defining a precise Monitoring Question (MQ) using the SORT framework (analogous to PICO in medicine):

S (Subject): Who or what is at risk?
O (Opportunity): What creates the exposure (e.g., specific usage context)?
R (Risk Event): What specific, countable harm occurs?
T (Timeframe): The observation period.
Format: "Among [S] that [O], how many [R] per [T]?"

B. Tiered Estimation Procedures

The framework estimates Harm ( $H$ ) and Exposure ( $E$ ) separately using a four-tier approach based on evidential strength:

Tier 1 (High Confidence): Direct data from authoritative sources (e.g., NHTSA crash filings, platform transparency reports).
Tier 2 (Medium Confidence): Derived from proxies.
- Harm: Lower bound = recorded incidents; Upper bound = harm from a broader containing class or proxy measures.
- Exposure: Both bounds constructed from proxies (e.g., combining census data with survey usage rates).
Tier 3 (Low Confidence): Expert elicitation to bound plausible ranges when quantitative data is absent.
Tier 4 (Abstention): Principled abstention when data is too sparse to support even an order-of-magnitude estimate. This is treated as a finding (data gap) rather than a failure.

C. Trajectory Classification

Once trends for Exposure ( $E$ ) and Harm-per-exposure ( $\hat{H}$ ) are estimated, they are mapped to a 2x2 grid to determine the trajectory:

Escalating ( $E \uparrow, \hat{H} \uparrow$ ): Both exposure and harm rate are rising. Action: Urgent intervention required.
Mitigating ( $E \uparrow, \hat{H} \downarrow$ ): Exposure is rising, but harm rate is falling. Action: Continue monitoring; safeguards are working but absolute harm may still rise.
Concentrating ( $E \downarrow, \hat{H} \uparrow$ ): Fewer people exposed, but those exposed face worse outcomes. Action: Targeted protective measures.
Receding ( $E \downarrow, \hat{H} \downarrow$ ): Neither dimension is worsening. Action: Maintain current strategy.

Note: If trends cannot be determined (Tier 4), the MQ is labeled "Unclassifiable."

3. Key Contributions

Decoupling Exposure and Hazard: The paper provides the first systematic method to distinguish between "more AI is being used" and "AI is becoming more dangerous."
Handling Uncertainty: By using order-of-magnitude estimates and explicit confidence tiers, the framework avoids the trap of false precision. It treats "unclassifiable" as a valid, informative result indicating data gaps.
Epidemiological Adaptation: Successfully adapts public health surveillance principles (standardized case definitions, proxy measures, tiered estimation) to the AI domain where fleet sizes and usage data are often unknown.
Governance Utility: Transforms raw data into actionable policy categories, helping regulators decide between expanding monitoring, intervening in deployment, or targeting specific sub-populations.

4. Results: Case Studies

The framework was applied to two distinct monitoring questions:

Case Study 1: Conversational AI and Self-Harm

Question: Incidents of AI encouraging self-harm among US users seeking emotional support.
Findings:
- Harm ( $H$ ): Estimated to be Increasing (based on OECD data showing a jump from 8 to 55 matches, and proxy data suggesting 1.2M users with suicidal indicators).
- Exposure ( $E$ ): Estimated to be Increasing (based on Pew Research data on ChatGPT usage).
Classification: Escalating.
Implication: Both the number of users and the risk per user are rising, demanding urgent regulatory intervention and expanded monitoring.

Case Study 2: Autonomous Vehicles (AVs) and Injury/Damage

Question: Injury/property damage incidents per million vehicle-miles for SAE Level 3-5 AVs.
Findings:
- Harm ( $H$ ): Absolute incident count Increasing (NHTSA data shows ~85% rise), but this is driven largely by property damage, not severe injury.
- Exposure ( $E$ ): Increasing significantly (AV mileage doubled from 2024 to 2025).
- Harm Rate ( $\hat{H}$ ): Since exposure grew (~~100%) faster than the incident count (~~85%), the harm-per-mile is Decreasing.
Classification: Mitigating.
Implication: Current safeguards are keeping pace with deployment. However, because absolute exposure is rising so fast, total absolute harm may still increase, requiring continued close monitoring.

5. Significance and Limitations

Significance:

Policy Clarity: The framework prevents the misinterpretation of rising incident counts as rising danger. It clarifies whether a risk is growing due to deployment scale or inherent system failure.
Resource Allocation: It helps policymakers distinguish between risks requiring deployment restrictions (Escalating) vs. those requiring technical safety improvements (Concentrating).
Foundation for Future Standards: It offers a blueprint for analyzing data before mandatory reporting regimes mature, utilizing existing news-sourced databases effectively.

Limitations:

Data Constraints: The framework relies on the quality of underlying proxies. If reporting propensity shifts drastically (e.g., viral media events), trend stability is compromised.
Binary Simplification: Counting incidents collapses complex harm severity into binary matches, potentially obscuring the nature of specific harms.
Retrospective Nature: Incident data only captures realized risks, not emerging or unknown risks (unknown unknowns).
Cross-MQ Comparability: While robust for tracking a single MQ over time, comparing different MQs (different types of harm) remains difficult.

In conclusion, the paper argues that moving from "raw counts" to "trajectory classification" is essential for effective AI governance. It provides a rigorous, transparent, and adaptable tool to navigate the uncertainty of AI safety data, ensuring that policy responses are grounded in the actual dynamics of risk rather than media noise.