A Structured Approach to Safety Case Construction for AI Systems

This paper addresses the inadequacy of traditional safety-case frameworks for modern AI systems by introducing comprehensive taxonomies of claims, arguments, and evidence, along with a reusable, structured template designed to construct credible and adaptive safety cases for generative and agentic AI.

The Gist

Imagine you are building a new, incredibly smart robot assistant. In the old days of engineering (like building airplanes or nuclear plants), you could write a manual that said, "If you press button A, the plane goes up. If you press button B, it goes down." You could test every single button, prove it works, and then say, "This plane is safe."

But modern AI is different. It's not built button-by-button; it's "taught" by reading millions of books and watching millions of videos. It learns things we didn't explicitly tell it to do. It's like a child who learns to speak by listening to the whole world, rather than being taught a strict dictionary. Because of this, we can't just write a manual and say, "It's safe." We don't even know all the things it can do yet.

This paper is a guide on how to build a "Safety Case" for these unpredictable AI systems. Think of a Safety Case not as a rulebook, but as a convincing legal argument or a portfolio of proof that says, "We believe this AI is safe enough to use, and here is the evidence to prove it."

Here is the paper broken down into simple concepts and analogies:

1. The Problem: The "Black Box" vs. The "Blueprint"

• Old Way (Airplanes): You have a blueprint. You know every part. You test every part. If the blueprint says "safe," it's safe.
• New Way (AI): The AI is a "black box." You feed it data, and it figures out how to solve problems on its own. Sometimes, it discovers new abilities (or new ways to break things) that the creators didn't even know were possible.
• The Challenge: You can't write a safety argument based on a blueprint you don't have. You have to argue safety based on what the AI actually does when you poke and prod it.

2. The Solution: The "Safety Case" Toolkit

The authors created a new toolkit to help people build these arguments. They call it a CAE system: Claims, Arguments, and Evidence.

Think of it like building a house:

• The Claim (The Roof): This is the big statement you want to prove.
  • Example: "This AI is safe to use for sorting government documents."
• The Argument (The Beams): This is the logic connecting your roof to the ground. It explains why the claim is true.
  • Example: "We know it's safe because we tested it against a human expert, and it made fewer mistakes."
• The Evidence (The Bricks): This is the actual proof.
  • Example: "Here are the test results showing the AI made 2.8% errors, while the human made 3.0%."

3. The New "Taxonomy" (The Filing System)

The paper realizes that old filing systems don't work for AI. So, they created a new way to categorize the pieces of the puzzle:

• New Types of Claims:

  • Old: "This machine will never fail." (Too rigid for AI).
  • New: "This machine is safe only if we keep it inside this specific room and don't let it talk to the internet." (Context-aware).
  • New: "This machine is safe because we built a cage around it so it can't do bad things." (Capability-limited).

• New Types of Arguments:

  • Comparative: "It's not perfect, but it's no worse than the human we replaced."
  • Discovery-based: "We didn't know it could do X, but we tested it, found it could, and then put a guardrail on it."

• New Types of Evidence:

  • Instead of just "test results," they use things like "Red Teaming" (hiring hackers to try to break the AI), "Expert Opinions," and "Real-world logs" (watching how it behaves in the wild).

4. The "Patterns" (Recipe Cards)

The authors noticed that people keep facing the same four hard problems when trying to prove AI is safe. They created four "Recipe Cards" (Patterns) to solve them:

1. The "Discovery" Pattern:
   • Problem: We don't know what the AI can do yet.
   • Recipe: Keep testing it constantly. Every time you find a new trick it learned, update your safety argument. It's like a living document that grows as you learn more about the AI.
2. The "No Perfect Answer" Pattern:
   • Problem: Sometimes there is no "correct" answer (e.g., judging a creative essay). How do you know the AI is safe?
   • Recipe: Compare it to a human. If the AI is "good enough" compared to a human, that's good enough. It's about being "no worse than" the baseline.
3. The "Living Update" Pattern:
   • Problem: The AI changes every week. It gets new data, new tools, or new updates.
   • Recipe: Your safety case must be a "living artifact." When the AI updates, the safety case updates automatically. It's like a car that re-inspects its own brakes every time you drive it.
4. The "Threshold" Pattern:
   • Problem: How much risk is too much?
   • Recipe: Set a number. "If the error rate is under 5%, we are good." If it hits 5.1%, the system stops. It turns safety into a clear, measurable line.

5. The Real-World Test: The Government Tender

To prove their ideas work, the authors tried it on a real government project: Using AI to help evaluate job bids (tenders).

• The Situation: A government wants to use AI to help humans decide which company gets a contract.
• The Problem: There is no "correct" answer. Different humans might pick different winners. How do you prove the AI isn't biased or dangerous?
• The Solution: They used the "No Perfect Answer" Pattern.
  • They didn't try to prove the AI was "perfect."
  • They proved the AI was "no worse than" the old human-only system.
  • They ran 200 fake bids. The humans disagreed with each other 3% of the time. The AI + Human team disagreed only 2.8% of the time.
  • The Result: The Safety Case said, "Look, the AI is actually slightly more consistent than the humans. It's safe to use."

Summary

This paper is essentially saying: "Stop trying to prove AI is perfect like a machine. Start proving it is safe like a partner."

It gives us a structured way to say: "We know this AI is weird and changes often, but here is our evidence, our logic, and our guardrails that prove it is safe enough for the job." It turns safety from a static checklist into a dynamic, living conversation between the developers, the regulators, and the AI itself.

Technical Summary

Here is a detailed technical summary of the paper "A Structured Approach to Safety Case Construction for AI Systems" by Lee et al.

1. Problem Statement

Traditional safety cases, widely used in aviation, nuclear, and rail engineering, rely on deterministic system boundaries, stable architectures, and known failure modes. These classical approaches fail when applied to modern AI systems (generative, agentic, and frontier AI) due to several fundamental mismatches:

• Emergent Capabilities: AI capabilities are discovered post-training rather than engineered line-by-line, making pre-deployment hazard enumeration impossible.
• Absence of Ground Truth: Many AI systems operate in domains where "correct" outputs are undefined, making absolute safety verification impossible.
• Dynamic Evolution: Models are frequently updated, fine-tuned, or retrained outside the system owner's direct control, invalidating static evidence.
• Context Dependence: Safety properties are heavily influenced by context engineering (prompts, tools, retrieval), making isolated component testing insufficient.
• Epistemic Uncertainty: Assurance must shift from deductive certainty to discovery-driven, inductive, and abductive reasoning to handle probabilistic behaviors and unknown failure modes.

Current literature lacks a coherent, reusable framework that integrates claims, arguments, and evidence (CAE) specifically tailored to these AI dynamics. Existing approaches are often domain-specific, isolated, or lack a unified taxonomy for AI-specific reasoning.

2. Methodology

The authors employed a Systematic Literature Review (SLR) following established guidelines to synthesize existing knowledge and derive new structures.

• Search Strategy: A multi-stage process involving keyword-based searches across major databases (IEEE, ACM, Springer, ScienceDirect) and snowballing (forward and backward) from 9 seed papers.
• Selection Criteria: From an initial pool of ~2,000 papers, 112 primary studies were selected based on strict inclusion/exclusion criteria (e.g., explicit linkage to AI safety cases, structured arguments, and CAE).
• Quality Assessment: Selected studies were scored on a 1–5 scale across 11 criteria (relevance, methodology, evidence quality, etc.), with a threshold of 2.0 for inclusion.
• Synthesis: The authors performed a mixed-methods synthesis, using descriptive statistics to identify trends and thematic analysis to derive taxonomies, templates, and patterns. This was supplemented by a real-world case study (AI tender evaluation) to validate the proposed framework.

3. Key Contributions

A. Comprehensive AI Safety Case Taxonomy

The paper introduces a unified, non-exclusive taxonomy for the three pillars of safety cases, adapted for AI:

• Claim Types:
  • Assertion-based: Absolute safety or marginal safety (relative to a baseline).
  • Constrained-based: Safety conditional on operating envelopes, data regimes, or modalities.
  • Capability-based: Safety ensured by limiting system abilities (e.g., tool-use limits, intrinsic refusal).
• Argument Types:
  • Demonstrative: Layered controls or barriers (Deductive).
  • Comparative: "No worse than" a baseline (Inductive/Statistical).
  • Risk-based: Quantitative risk estimation or guideline compliance (Statistical/Deductive).
  • Causal/Explanatory: Root-cause analysis of failures (Abductive).
  • Capability-oriented: Proving inability to perform harmful acts.
  • Normative: Adherence to standards or governance.
• Evidence Families:
  • Includes empirical (testing, red-teaming), comparative (benchmarks), model-based (simulations), expert-derived, formal methods, operational/field data, and mechanistic (interpretability) evidence.

B. Reusable Safety Case Templates

The authors propose a meta-model for constructing safety cases that moves beyond static documents. These templates are designed to be composable, allowing practitioners to stitch together multiple CAE chains (e.g., one for architectural robustness, another for misuse safeguards). The templates explicitly define:

• Conditional propositions incorporating data and context assumptions.
• Multi-modal reasoning structures combining deductive, inductive, abductive, and statistical logic.
• Dynamic update triggers for evidence re-validation.

C. End-to-End Safety Case Patterns

Four specific patterns are defined to address recurring AI assurance challenges:

1. Discovery-Driven Evaluation: For justifying safety when new capabilities/risks are found post-deployment. Uses abductive reasoning to explain anomalies and statistical reasoning for progressive improvement.
2. Marginal-Risk (No Ground Truth): For systems where absolute correctness is unknown. Relies on comparative arguments (AI vs. Human baseline) and proxy metrics (predictability, consistency) to prove "no worse than" performance.
3. Continuous-Evolution: For managing dynamic updates. Treats the safety case as a "living artifact" linked to live metrics (SPIs) and version control, requiring re-validation upon model changes.
4. Threshold-Comparator: For quantitative decision-making. Maps multiple quantitative thresholds (e.g., harm probability, compute limits) to deployment decisions using statistical aggregation and normative justification.

D. Ecosystem Pipeline

The paper outlines a governance pipeline involving three roles:

• Builder: Developers/owners constructing versioned safety cases.
• Validator: Regulators/auditors verifying compliance and evidence integrity.
• Registry: A trusted repository for storing and comparing validated cases.

4. Results and Validation

• Taxonomy Mapping: The study mapped 112 studies to the new taxonomy, revealing that while "conditional claims" and "empirical evidence" are common, there is a significant gap in "post-deployment assurance" and standardized "evidence quality criteria."
• Case Study (AI Tender Evaluation): The authors applied the Marginal-Risk Pattern to a government tender system where an AI assisted human reviewers.
  • Challenge: No ground truth for "correct" tender scores.
  • Solution: Compared the AI+Human process against a Human+Human baseline.
  • Outcome: The AI+Human system showed a marginal risk difference of -0.2% (slightly better consistency) and an inconsistency rate of 2.8%, well below the 5% acceptance threshold. This validated the "no worse than" claim using comparative and statistical arguments.
• Pattern Integration: The study demonstrated that a single safety case can be a composite of multiple patterns (e.g., using Discovery-Driven for red-teaming results and Continuous-Evolution for model updates).

5. Significance

This paper provides a critical bridge between traditional safety engineering and the unique challenges of modern AI.

• Standardization: It offers the first comprehensive, reusable taxonomy and template library specifically for AI, addressing the fragmentation in current literature.
• Shift in Logic: It formally codifies the shift from design-time determinism to evaluation-time discovery, legitimizing inductive and abductive reasoning as core components of safety assurance.
• Dynamic Assurance: It moves the field toward "living" safety cases that evolve with the system, essential for frontier AI where static certification is impossible.
• Practical Utility: By providing concrete patterns and a case study, it gives practitioners a actionable roadmap for constructing credible, auditable, and regulatory-compliant safety arguments for complex AI systems.

The work establishes a foundation for composable assurance, enabling the construction of safety cases that remain credible despite the uncertainty, rapid evolution, and emergent behaviors inherent to generative and agentic AI.