🤖 machine learning

Who Analyses the Analyser? Self-Validating LLM Hazard Analysis with Constitutional Meta-STPA

This paper addresses the critical gap of analyzing the AI tools used for safety analysis by introducing "Constitutional Meta-STPA," a self-validating framework that applies Systems-Theoretic Process Analysis (STPA) to itself to derive and enforce a governance constitution, thereby ensuring that the LLM-assisted analyser is rigorously audited for hallucinations and unverifiable constraints.

Original authors: Samuel Tetteh, Udip Shrestha, Joshua R. Waite, Cody Fleming

Published 2026-07-10

📖 6 min read🧠 Deep dive

CC0 1.0

Original authors: Samuel Tetteh, Udip Shrestha, Joshua R. Waite, Cody Fleming

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot assistant whose job is to inspect other machines for safety flaws. It's great at finding broken gears in cars or leaky valves in pumps. But here's the catch: nobody ever asked, "Who checks the robot?"

This paper asks exactly that question: Who analyses the analyser?

The authors realized that while everyone trusts these AI tools to write safety reports, the tools themselves can be glitchy. They might invent fake safety rules, forget to log their work, or give a confident answer that's actually nonsense. It's like hiring a detective who never writes down their clues and sometimes makes up evidence.

The Self-Inspecting Robot

To fix this, the team built a special version of their safety tool that turns the microscope on itself. They used a method called STPA (a rigorous way of finding safety risks) to analyze the AI tool's own design.

Think of it like a robot that builds a map of its own brain, finds the weak spots, and then writes a rulebook for itself based on those findings. They didn't just guess the rules; they let the tool's own "safety analysis" of itself generate the rules.

The Two-Part Rulebook

The tool ended up with a "Constitution" (a rulebook) made of two distinct layers:

The "Tool Principles" (The Behavior Layer): These are 21 rules about how the AI should act when it's doing its job. For example, "Don't make up safety standards," "Be specific about what you can't do," and "Always check your work."
The "Meta-Safety Principles" (The Governance Layer): These are 8 rules about how the machine running the AI should behave. These include things like "Keep a permanent log of every question and answer," "Lock in the specific version of the brain you're using so it doesn't change mid-task," and "Don't let the AI export a report unless it passes a strict checklist."

The Big Surprise: It's Not Magic, It's Muscle

The authors tested this by asking different AI models to analyze the tool itself. Here is what they found:

The Strong Models: When they used the most advanced AI models (a "frontier ensemble" of two top-tier models), the tool successfully found 18 out of 21 of the behavior rules and all 8 of the governance rules just by looking at its own design.
The Weak Models: When they used weaker, cheaper models, they only found 12 out of 21 behavior rules and just 3 out of 8 governance rules.

The Takeaway: The rules themselves weren't the problem; the brain doing the analyzing was. The "constitution" works, but only if the AI is smart enough to find the holes in itself.

What Actually Makes the AI Safer?

The team ran a tricky test with 20 "adversarial probes"—tricky questions designed to trick the AI into being unsafe or lying. They tested the AI with different versions of the rulebook:

No Rules: The AI scored an average safety of 1.03.
Generic Rules: Adding a generic "be helpful and honest" list didn't help at all (score: 1.05).
The 21 Tool Principles: When they added the specific 21 behavior rules, the safety score jumped to 1.85. That's a 79% improvement, and it was a huge, statistically significant win.
The 8 Governance Rules: Adding the 8 governance rules (the logging and version-locking stuff) didn't change the score on these specific trick questions.

The Lesson: The specific behavior rules (the 21 Tool Principles) are what actually stop the AI from making mistakes in its answers. The governance rules (the 8 Meta-Safety Principles) are crucial for the system (like keeping an audit trail), but they don't magically make the AI's text answers safer on their own.

What About "Coverage"?

The authors also looked at a common idea in AI safety: "If we just make the AI check more boxes, it's safer." They measured how many rules the AI "covered" in its analysis.

They found something weird: More rules in the prompt didn't always mean more safety coverage.

When they tested the AI on a standard hardware machine (like a car brake), the "Meta-Safety" rules (about logs and versions) scored 0 out of 8. The AI didn't mention them because the car doesn't have an audit log.
When the AI analyzed itself, the score jumped to 6 out of 8.

This proved that "coverage" isn't a magic dial you can turn up to make things safer. It's a detector. It tells you what kind of system you are looking at. If the AI is talking about audit logs, it's analyzing an AI tool. If it's silent on logs, it's analyzing a machine. You can't just force the AI to say "I checked the logs" to make it safer; the logs have to actually exist.

The "Oops" Moment

The authors were also very honest about a mistake they made. Earlier, they thought that adding more rules would make the AI find more safety issues in a straight line (like a dose of medicine). But when they ran the test again with strict, fixed settings, that "dose-response" idea disappeared. They found that the number of issues found actually went down or stayed flat as they added more rules. They published this "failed" result openly, showing that sometimes the AI just gets more focused and stops rambling, rather than finding more problems.

The Bottom Line

This paper doesn't claim to have "solved" AI safety. Instead, it built a tool that:

Writes its own rulebook by analyzing its own design.
Proves that specific behavior rules (the 21 Tool Principles) make the AI's answers much safer (a 79% boost).
Shows that generic "be nice" rules don't work.
Releases everything: the code, the rulebook, and the logs, so anyone can replay the exact same experiments.

The authors conclude that if you are building an AI safety tool, you need the specific behavior rules to stop the AI from hallucinating, and you need the governance rules to keep a paper trail. But you also need a really smart AI model to do the work in the first place. Without a capable brain, even the best rulebook won't save you.

Technical Summary: Constitutional Meta-STPA

1. Problem Statement

Large Language Models (LLMs) are increasingly utilized to automate the generation of safety analysis artifacts (losses, hazards, Unsafe Control Actions [UCAs], and constraints) within rigorous frameworks like Systems-Theoretic Process Analysis (STPA). However, a critical blind spot exists: while the target systems (e.g., aircraft, medical devices) are analyzed, the LLM-assisted tools performing the analysis are rarely scrutinized. These tools are themselves safety-relevant systems capable of hallucinating standards, emitting unverifiable constraints, and failing to maintain audit trails. The paper asks the fundamental question: "Who analyses the analyser?" Current literature lacks a method to formally analyze the LLM-assisted safety tool itself, leaving it vulnerable to the very unsafe control actions STPA is designed to prevent.

2. Methodology: Constitutional Meta-STPA

The authors propose Constitutional Meta-STPA, a closed-loop methodology that applies STPA to the class of AI-assisted safety tools to derive their own governance constitution.

2.1. The Meta-STPA Process

Instead of analyzing a physical system, the tool treats the AI safety pipeline as the system under analysis.

Controllers: The LLM, validator layer, voting harness, export gate, and the on-call engineer.
Control Actions: Generating losses/hazards/UCAs, flagging errors, reporting inter-model disagreement, and exporting reports.
Derivation: By walking the four UCA types over these control actions, the system identifies hazards unique to AI tools (e.g., unlogged LLM calls, silent version drift, tampered descriptions).
Constraint Generation: Each identified hazard yields a safety constraint, forming a governance constitution.

2.2. The Constitution Structure

The derived constitution consists of 29 principles bound to specific code enforcement points:

21 Tool Principles (TP): A behavioral layer shaping the content of the analysis (e.g., systematic enumeration, no fabrication, measurable constraints).
8 Meta-Safety Principles (MSP): A governance layer governing the tool's machinery (e.g., audit logging, run manifests, model version pinning, semantic-matching voting, validator-gated export).

2.3. The Pipeline Implementation

The tool operates as a five-step pipeline ( $\Pi$ ) mapping a system description ( $S$ ) and constitution ( $C$ ) to an artifact ( $A$ ):

LLM Generation: Free-form generation of losses, hazards, and control structures.
Deterministic Enumeration: The engine creates fixed slots for every (controller, action, UCA type) triple. The LLM must fill these with a concrete UCA or an explicit "not applicable" justification. This ensures structural completeness independent of model diligence.
Constraint Generation: One constraint slot is created per non-N/A UCA.
Validation: Eight deterministic regex/structure validators check for completeness, vague language, overconfidence, and missing evidence.
Gating: A "Validator-Gated Export" (MSP-07) blocks redistributable artifacts (PDFs, emails) if validation errors exist, unless an engineer provides a recorded override.

2.4. Reproducibility and Auditing

Manifests: Every run generates a run manifest.json pinning the tool version, model, seed, temperature, and SHA-256 hashes of the system description and constitution.
Audit Logs: Every LLM call is logged with prompt/response SHA-256 hashes, token counts, and latency.
Semantic Voting: To handle model disagreement, the tool uses a token-Jaccard similarity matcher (threshold $\tau=0.4$ ) rather than strict ID equality, allowing for synonym handling while maintaining determinism.

3. Key Contributions

Meta-STPA Method: A method that derives a safety tool's governance constitution by applying STPA to the tool itself, resulting in a published 29-principle constitution.
Controlled Behavioral Evaluation: An experiment isolating the impact of constitutional layers, demonstrating that task-specific Tool Principles significantly improve safety behavior, while generic "helpful/harmless" preambles do not.
Coverage Framework: A formal definition of "constitution-marginal coverage" with a soundness lemma. The paper reports a negative result: lexical coverage is not monotone with constitution strength on a fixed system but acts as a system discriminator (distinguishing AI tools from hardware).
Full Reproducibility Artifact: Release of code, constitutions, manifests, and seeded calls allowing every result to be replayed from a single command line.

4. Experimental Results

4.1. Self-Derivation (Can the tool analyze itself?)

Strong Frontier Models: A voting ensemble of claude-opus-4.8 and claude-sonnet-4 recovered 18/21 canonical Tool Principles and all 8/8 Meta-Safety Principles from the tool's own design.
Weak Models: A weaker pair (gpt-4o-mini + gpt-4o) recovered only 12/21 canonical and 3/8 meta principles.
Conclusion: The meta-layer is model-limited, not constitution-limited. The principles are recoverable from the design by capable models, not just asserted.
Generalization: When applied to a second, independently authored AI tool (a code-review assistant), the strong ensemble recovered all 8/8 governance principles, confirming these are properties of the class of AI tools, not artifacts of the specific tool design.

4.2. Behavioral Impact (Does the constitution change behavior?)

Setup: 20 adversarial probes were tested against five constitution variants (None, Generic, Tool Principles, Tool+Generic, Full).
Findings:
- Adding the 21 Tool Principles (cv2) raised the mean safety score from 1.03 to 1.85 (+79%, $p<0.001$ , $d=1.29$ ).
- Generic "helpful/harmless" preambles (cv1) showed no significant improvement over no constitution.
- Adding the 8 Meta-Safety Principles (MSP) to the Tool Principles did not further improve single-turn adversarial scores, as MSPs govern analysis-time machinery (logging, gating) not the text generation itself.

4.3. Coverage and Construct Validity

Non-Monotonicity: On a fixed system, increasing constitution strength did not linearly increase lexical meta-coverage. In fact, the empty control (cv0) often surfaced more meta-hazards than the full constitution because these hazards are intrinsic to the AI tool's structure.
System Discrimination: Meta-coverage was 0/8 for hardware systems (AEB, infusion pump, UAV) but rose to 6/8 for the AI self-system. This confirms coverage is a discriminator between AI tools and hardware, not a "dose" metric to be maximized.
Reproducibility: Under fixed seeds, the tool showed high stability across vendors (gemini-2.5-flash was exactly reproducible; claude-sonnet-4 showed low variance).

5. Significance and Claims

The paper claims to answer "Who analyses the analyser?" by demonstrating that the analyser can analyze itself.

Grounded Constitution: The governance principles are not hand-crafted assertions but are derived from the tool's own design via STPA.
Operational Safety: The constitution is enforced via code (validators, gating, manifests) rather than just prompt instructions.
Model Capability as a Bottleneck: The ability to recover governance principles depends on the capability of the underlying LLM, not the wording of the constitution.
Transparency: The authors openly report a failure to replicate an earlier exploratory result (a monotone rise in coverage), reframing the coverage metric as a system discriminator rather than a dose-response curve.

The paper concludes that for LLM-assisted safety tools to be trusted, they must include a task-specific behavioral layer (Tool Principles) and a governance substrate (Meta-Safety Principles) that is hard-coded into the tool's execution flow, ensuring that the tool's own safety constraints are auditable and enforceable.