Designing escalation criteria for international AI incident response: criteria, triggers, and thresholds

Technical Summary: Stress-testing AI Incident Escalation

Problem Statement
As regulatory frameworks for AI incident reporting emerge (e.g., the EU AI Act, California's SB 53), a critical gap exists in evaluating whether the escalation criteria within these frameworks reliably detect the incidents they are designed to catch. Current literature focuses on incident database design and systemic harm detection but lacks an evaluation of whether the upstream criteria—definitions, thresholds, and triggers—function effectively when applied to real-world scenarios. Poorly designed criteria risk producing systematic blind spots, failing to escalate incidents that warrant attention.

Methodology
The authors employ a three-stage sequential design process:

1. Framework Development: An eight-criterion escalation framework was constructed by reviewing cross-domain precedents (e.g., WHO International Health Regulations, EU's DORA, Basel II) and existing AI regulations.
2. Stress-Testing: The framework was subjected to a "walkthrough" methodology, inspired by cybersecurity tabletop exercises and early DORA implementation experiences. Ten documented AI incidents and structured variants were analyzed across four high-risk domains: cyber, CBRN (Chemical, Biological, Radiological, Nuclear), psychological harm, and multi-agent risk.
3. Counterfactual Analysis: For each incident, the authors determined if the framework would trigger escalation, at which decision gate, and based on what evidence available at specific time points. Structured variations were introduced to identify conditions under which classifications would change.

The Escalation Framework
The proposed framework classifies incidents into three outcomes: Escalate (international coordination required), Alert (capacity support/information sharing), or Discard. It utilizes a sequential flowchart with eight gated criteria:

1. AI Causation: Was AI a causal factor?
2. Scope: Domain-based exclusions (e.g., military).
3. Immediate Escalation: Pre-harm triggers (e.g., CBRN assistance, loss of developer control) regardless of confirmed harm.
4. Pattern Detection: Aggregation of incidents sharing a root cause (technical, capability, or contextual).
5. Harm Category & Severity: Assessment against recognized taxonomies (e.g., MIT) and severity levels (Level 4+).
6. Cross-border Propagation: Need for international coordination.
7. Irreversible Cross-border Harm: Permanent damage with downstream effects.
8. Near Miss: Averted incidents revealing failure modes affecting other systems.

Key Results
The stress-testing revealed that the framework performs reliably for discrete-onset, individually severe incidents where AI's causal role is clear. However, three categories of incidents proved structurally challenging, leading to systematic under-detection in current regulatory frameworks:

• Cumulative, Individually Sub-threshold Harms: Incidents like non-consensual deepfakes and psychological harm from human-AI interaction involve millions of low-severity interactions. Current frameworks assessing incidents individually fail to capture the aggregate severity, which only becomes visible through pattern detection and aggregation.
• Near-Miss and Potential Harm: Incidents with high potential severity but low realized harm (e.g., agentic platform vulnerabilities) are often missed unless specific near-miss criteria are applied.
• Severity Scale Gaps: Diffuse harms (psychological distress, dignity violations) do not map well to existing quantitative severity descriptors. Current frameworks lack mechanisms for cross-category aggregation, causing incidents to fall below escalation thresholds despite regulatory treatment as severe.

Findings: The Dependency Chain and Design Patterns
The authors identify a three-layer dependency chain explaining these failures:

1. Layer 1 (Data Infrastructure): What can be detected.
2. Layer 2 (Definitional Foundations): How incidents and harm are categorized and scored.
3. Layer 3 (Escalation Triggers): Thresholds and decision logic.

Gaps in upstream layers propagate downstream. The paper identifies four specific design patterns in current frameworks (EU AI Act, SB 53) that drive under-detection:

1. Triggers Requiring Confirmed Harm for Pre-harm Risks: Requiring actual harm outcomes before reporting prevents the escalation of high-risk events (e.g., model providing CBRN assistance) where the developer sees the trigger but not the downstream harm.
2. Individual-Only Assessment Missing Cumulative Harms: Lack of aggregation provisions means population-level effects (e.g., 500,000 users exhibiting risk factors) are invisible when assessed as discrete events.
3. Legally-Defined Thresholds That Are Not Practically Testable: Definitions relying on "infringement of fundamental rights" or specific quantitative thresholds (e.g., 50 casualties) are either unmeasurable by developers or miss moderate but real severity incidents.
4. Discrete-Event Architecture Applied to Standing Conditions: Applying event-based logic to continuous, standing conditions (e.g., ongoing psychological harm, gradual information integrity erosion) renders these risks invisible, as there is no discrete onset or resolution point.

Proposed Solution
To address "standing conditions," the authors propose tolerance-based monitoring, drawing on financial services precedents (e.g., Basel II, Bank of England). This involves:

1. Standardizing harm categorizations across providers.
2. Operationalizing continuous monitoring (moving from point-in-time snapshots to continuous feeds).
3. Setting tolerances where reportable events are defined by abnormal changes in the rate, severity, or distribution of harm (e.g., prevalence spikes above baseline) rather than the mere presence of harm.

Significance and Claims
The paper claims that designing better triggers alone is insufficient. Definitional work (Layer 2) and data infrastructure (Layer 1) are preconditions for operational escalation criteria. Without addressing these upstream layers, frameworks risk being formally complete but operationally unimplementable, particularly for risk domains where under-detection has severe consequences.

The work contributes to the "Operationalization" capacity of technical AI governance by:

• Providing an eight-criterion framework grounded in cross-domain precedents.
• Demonstrating through stress-testing that current regulatory designs contain structural features leading to systematic blind spots.
• Proposing a dependency structure that explains why certain risks (cumulative, psychological, standing conditions) are currently invisible to escalation mechanisms.
• Offering a constructive proposal for tolerance-based monitoring to address ongoing, population-level harms.

The authors note limitations, including the sample size of ten incidents and the reliance on judgment calls in the walkthrough methodology, suggesting future work should involve multi-stakeholder validation and domain-specific trigger refinement.