🤖 AI

Towards AI epidemiology: a measurement standardisation framework for prospective risk detection

This concept paper proposes a measurement standardisation framework to enable "AI epidemiology" for prospective risk detection in deployed AI systems by defining a grammar and statistical protocol to validate the reliability of expert-AI interaction assessments and establish a basis for future governance and epidemiological studies.

Original authors: Kit Tempest-Walters

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kit Tempest-Walters

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a busy hospital, a bank, or a law firm. You've hired a team of incredibly smart, fast, but invisible assistants (AI models) to help your experts make decisions. The problem is, these assistants are "black boxes." You can see what they say, but you have no idea how they think inside. Sometimes they give great advice; sometimes they give dangerous or rule-breaking advice, and you don't know why until it's too late.

This paper proposes a new way to manage these invisible assistants without needing to open up their "brains." It suggests a system called AI Epidemiology.

Here is the breakdown of the paper's ideas using simple analogies:

1. The Problem: We Can't See Inside the Machine

Currently, scientists try to understand AI by looking at its internal code (like trying to understand a car engine by taking it apart). The paper calls this "correspondence-based interpretability." But as AI gets bigger, this becomes impossible. The engine is too complex, and the parts are too tangled.

The Paper's Solution: Instead of trying to see how the engine works, let's just watch what the car does and how the driver reacts to it. We don't need to know the mechanics to know if the car is driving safely.

2. The Core Idea: The "Traffic Cop" System

The authors propose a framework that acts like a Traffic Cop standing next to every interaction between a human expert and the AI.

Every time an expert asks the AI a question and gets an answer, the system automatically records three things in a standardized "report card":

The Mission: What was the question? (e.g., "Should I approve this loan?")
The Conclusion: What did the AI say? (e.g., "Reject the loan.")
The Justification: Why did the AI say that? (e.g., "The credit score is too low.")

The system then gives this report card two scores:

Policy Alignment: Does this answer follow the company's rules?
Evidential Alignment: Is the AI's reasoning based on real facts, or is it making things up?

3. The "Judge" Who is Also an AI (The Circular Problem)

You might ask: "Who grades the report card?"
The paper admits a funny problem: They use another AI (a Large Language Model) to grade the first AI's answers. It's like hiring a second invisible assistant to check the work of the first one.

How they fix this: They don't just let the second AI guess. They give it a strict Rubric (a checklist), force it to look at the official rulebooks (Reference Documents), and make it think step-by-step before giving a score. They also run tests to make sure the "Judge AI" isn't just being a "yes-man" (agreeing with everything) or liking long answers just because they are long.

4. The "Epidemiology" Analogy

This is the most creative part of the paper. The authors compare their system to Public Health Epidemiology.

The Old Way (Mechanism): In the past, doctors didn't know why smoking caused cancer. They didn't know the molecular biology. But they noticed a pattern: Everyone who smoked got sick. They didn't need to understand the biology to tell people to stop smoking.
The New Way (Correlation): Similarly, this framework doesn't need to know how the AI makes a mistake. It just needs to notice patterns.
- Example: "Every time the AI talks about 'post-bankruptcy loans,' it gets a low score on facts."
- Result: The institution knows, "Hey, our AI is bad at this specific topic," and they can fix it or have humans double-check those specific cases.

5. What Happens in Real Life?

The paper describes a workflow for the human expert:

The expert asks the AI a question.
The AI answers.
Instantly, the system shows the expert a "Traffic Light" signal:
- Green: High scores on rules and facts. Go ahead.
- Yellow/Red: Low scores. The system flags it.
If the light is red, the expert is prompted to hit an "Override" button. They can then correct the AI's answer.
The system quietly saves all these "Traffic Light" data points.

6. The Big Picture: Three Stages of Proof

The paper doesn't claim to have solved everything yet. It outlines a three-stage research plan to prove this works:

Stage 1 (Reliability): Prove that the "Judge AI" can consistently grade the answers correctly, just like a human would.
Stage 2 (Governance): Prove that these grades actually help experts and managers spot trouble before it happens.
Stage 3 (Outcome): Prove that when the AI gets a "Red Light," it actually leads to bad real-world results (like a bad loan or a medical error), confirming that the system is detecting real risks.

Summary

This paper proposes a standardized reporting system for AI. Instead of trying to understand the AI's secret thoughts, it treats AI interactions like disease symptoms. By collecting thousands of these "symptoms" (standardized reports) and grading them, institutions can spot dangerous patterns and fix them before they cause harm, all without needing to see inside the AI's black box.

Important Note: The paper explicitly states that this is a proposal and a framework. It does not claim to have already solved AI safety or to have results from real-world hospitals or banks yet. It is setting the rules for how we will test this in the future.

Technical Summary: Towards AI Epidemiology

Problem Statement

Modern AI systems, particularly Large Language Models (LLMs), are increasingly deployed in high-stakes regulated domains (healthcare, finance, legal) where their internal computations remain opaque ("black-box"). Current governance approaches rely heavily on correspondence-based interpretability (e.g., SHAP, LIME, mechanistic tracing), which attempts to map model outputs back to internal causal mechanisms. However, as models scale, these methods face computational intractability and epistemic unreliability; feature attributions become unstable, and models often generate plausible-sounding but unfaithful explanations.

Consequently, there is a critical governance gap: a lack of scalable, non-mechanistic methods to detect AI output failures and policy misalignments in real-time deployment without access to model internals. Existing incident reporting systems are retrospective and selective, while algorithmic auditing often lacks standardized units of analysis for routine interactions.

Methodology

The paper proposes a Measurement Standardisation Framework designed to compress expert-AI interactions into structured, comparable fields to enable prospective risk detection. This approach shifts the locus of risk detection from internal model processes to observable patterns of behavior, drawing an analogy to epidemiology (e.g., John Snow's water source identification or Bradford Hill's smoking-cancer link), where population-level correlations justify intervention despite mechanistic uncertainty.

Core Components: The Grammar

The framework defines a structured protocol comprising eight fields to capture interactions:

Mission: The task extracted from the expert's input (task type + subject).
Conclusion: The AI's recommended action, categorized rather than verbatim.
Justification: The reasons provided by the AI, categorized.
Risk Level: A stratification variable (High/Medium/Low) based on the potential severity of harm if the conclusion is incorrect.
Policy Alignment Score: A three-level ordinal score (High/Medium/Low) assessing consistency with institutional/regulatory policy.
Evidential Alignment Score: A three-level ordinal score assessing whether the factual claims in the justification are supported by the domain evidence base.
Override: A binary flag indicating if an expert explicitly contested the AI's conclusion.
Corrective Option: The revised conclusion and justification exported by the expert after an override.

The LLM-as-Judge Mechanism

To achieve scale, the framework employs an LLM-as-Judge to automatically score Policy and Evidential Alignment. To address the structural circularity of using a black-box model to judge another, the framework imposes bounded conditions:

Explicit Rubrics & Anchored Scales: Using a three-level anchored scale to improve agreement.
Structured Chain-of-Thought: The judge reasons step-by-step against rubrics before assigning a score.
RAG Grounding: Judges are grounded in specific policy and evidence documents to reduce sycophancy and self-preference.
Low-Temperature Decoding: To ensure score stability and repeatability.

A Reliability Verification Procedure is mandated before deployment, requiring:

Human-Judge Agreement: Measured via linearly weighted Cohen's $\kappa$ (target $\ge 0.61$ ).
Consistency: Measured via Intraclass Correlation Coefficient (ICC) for test-retest stability (target $\ge 0.75$ ).
Bias Diagnostics: Targeted tests for sycophancy, verbosity, and self-preference.

Empirical Evaluation Protocol

The paper specifies a protocol to test whether the compressed grammar fields retain sufficient predictive power for risk detection compared to full conversational text.

Design: A non-inferiority test comparing a classifier trained on grammar fields ( $X$ ) against a "whole-input" classifier.
Metric: Area Under the Curve (AUC) for separating misaligned from aligned interactions.
Margin: A pre-specified non-inferiority margin of $\delta = 0.05$ .
Inference: Paired bootstrap inference (10,000 resamples) with DeLong's test for sensitivity and Holm–Bonferroni correction for multiple testing.
Refinement: An iterative grammar refinement process (splitting categories, adjusting rubrics) guided by the AUC gap until non-inferiority is met on a frozen test set.

Key Contributions

Conceptual Shift: Introduces "correspondence-based interpretability" as a distinct category from the proposed "correlation-based" approach. It argues that for immediate governance, observing output patterns is more viable than waiting for mechanistic transparency.
Measurement Standardisation: Defines a specific grammar and scoring protocol that transforms unstructured expert-AI dialogues into standardized, comparable data units suitable for population-level analysis.
Governance Signals: Proposes three actionable institutional signals derived from the data:
- Aggregate Alignment Distributions: Identifying domains where AI outputs are consistently misaligned.
- Override Patterns: Highlighting where experts systematically contest AI recommendations.
- Corrective Option Patterns: Revealing the specific grounds and revised conclusions experts prefer, offering a "corrective" dataset for future model tuning or policy updates.
Three-Stage Empirical Programme: Outlines a staged research path:
- Stage 1: Establish measurement standardisation (reliability of LLM judges).
- Stage 2: Validate governance signals (usefulness for experts and institutions).
- Stage 3: Establish outcome associations (linking alignment scores to downstream adverse events).

Results and Claims

As a concept paper, no empirical results are presented. The paper explicitly states that population-level claims are the subject of a staged research programme, not results claimed herein.

The paper makes three specific, testable claims:

Reliability Claim: Under bounded conditions (rubrics, RAG, low temperature), LLMs can produce reliable, standardised assessments of alignment comparable to human raters.
Governance Claim: Alignment scores provide immediate signals to experts during deployment and allow institutions to monitor alignment patterns across models and domains.
Epidemiological Claim: Once measurement is standardised, aggregate alignment scores can be used to study associations with downstream outcomes, enabling "AI epidemiology" that detects risk via correlated variables rather than mechanistic analysis.

Significance

The paper's significance lies in providing a pragmatic, scalable pathway for AI governance in the absence of perfect interpretability. By treating AI risk detection as an epidemiological problem, it offers a method to:

Detect risks prospectively (before incidents occur) rather than retrospectively.
Operate at scale without requiring access to model weights or internal states.
Generate audit trails and corrective data that can inform both immediate expert decisions and long-term institutional policy.

The framework acknowledges its limitations, including the "exposure-instrument disanalogy" (using an LLM to judge an LLM) and the potential for observer effects, but argues that the proposed verification procedures and iterative refinement can mitigate these threats to validity. The ultimate goal is to establish a foundation where "AI epidemiology" can function as a robust, non-mechanistic safety layer for deployed AI systems.