🤖 AI

The Two Boundaries: Why Behavioral AI Governance Fails Structurally

This paper argues that behavioral AI governance structurally fails because it is undecidable to verify arbitrary program effects against policies (per Rice's theorem), and proposes "coterminous governance"—an architectural separation of computation from effects where governance is embedded in the execution pipeline—as the only solution to eliminate the inevitable risks and inefficiencies caused by misaligned capability and policy boundaries.

Original authors: Alan L. McCann

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Alan L. McCann

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Two Boundaries" Mismatch

Imagine you have a very powerful robot that can do almost anything: send emails, move money, or change database records. To keep it safe, you put up a fence (governance) to tell it what it is allowed to do.

The paper argues that in almost every AI system built today, there are two different fences that don't match up:

The Capability Fence (What the robot can do): This is determined by the robot's tools and code. If the robot has a tool to send emails, it can send emails.
The Rule Fence (What the robot is allowed to do): This is determined by the safety rules and filters you wrote.

The Problem: These two fences are almost never the same size or shape.

The "Risk Zone" (Ungoverned Capability): Sometimes the robot has a tool it can use, but your rules didn't cover that specific tool. It's like the robot has a secret back door that your security guard doesn't know about. The robot slips through, and bad things happen.
The "Theater Zone" (Governance Theater): Sometimes your rules are very strict about things the robot cannot do. It's like having a security guard checking for "flying" when the robot has no wings. You are wasting energy policing things that aren't possible, while the real dangers (the back doors) remain open.

The paper says that trying to fix this by adding more guards or more rules just makes the "Theater" bigger and the "Risk" slightly smaller, but it never fixes the gap.

Why You Can't Just "Watch" the Robot (The Math Problem)

You might think, "Why don't we just build a super-smart monitor that watches the robot and stops it if it does something bad?"

The paper uses a famous math rule called Rice's Theorem to explain why this is impossible for smart, flexible AI.

The Analogy: Imagine you are trying to predict the ending of a movie just by looking at the script. If the movie is short and simple, you can guess the ending. But if the movie is an infinite, complex story that changes based on the audience, no algorithm can predict the ending with 100% certainty before the movie plays.
The Result: Because AI is so flexible (it can write its own code and combine tools in new ways), you cannot mathematically prove in advance that a specific action will be safe. You can only guess. And because you can only guess, there will always be a chance the robot does something bad that your monitor didn't see.

The Solution: The "Order Taker" vs. The "Chef"

The paper proposes a structural fix. Instead of trying to watch the robot and stop it from doing bad things, you change how the robot is built.

The Old Way (Behavioral Governance):
The robot is a Chef who has direct access to the stove, the knives, and the gas. You stand next to them with a clipboard, yelling "Stop!" if they reach for the wrong ingredient.

Flaw: The Chef might grab the gas before you can yell, or they might find a new way to cook that you didn't write down on your clipboard.

The New Way (Structural Governance / Coterminous Governance):
You separate the Chef from the Kitchen.

The Chef (Computation): The AI is only allowed to think and write down a recipe (a "directive"). It cannot touch the stove, the knives, or the gas. It has no physical hands.
The Order Taker (Governance Boundary): A separate, strict system receives the recipe. It checks: "Does this recipe have the right ingredients? Is it safe?"
- If yes, the Order Taker goes into the kitchen and cooks the meal.
- If no, the Order Taker throws the recipe away.

Why this works:

The AI cannot do anything bad because it literally doesn't have the tools to touch the world.
Every single action must go through the Order Taker.
There is no "Risk Zone" because the AI has no secret back doors.
There is no "Theater Zone" because the Order Taker only checks recipes that can actually be cooked.

Why "Monitoring" Isn't Enough

The paper also explains why simply "logging" or "recording" what the AI does doesn't count as governance.

The Analogy: Imagine a security camera in a bank. If the camera sees a robber, it records the crime. But the camera didn't stop the robbery.
The Math: If a camera catches 99% of robberies, that sounds great. But if the bank has 1,000 transactions a day, the chance that at least one robbery slips through the 1% gap is nearly 100%.
The Point: You need to stop the action before it happens, not just record it after.

The Bottom Line

The paper concludes that to truly govern AI, you cannot just add more filters or rules on top of existing systems. You have to rebuild the system so that the AI can only ask for things, and a separate, strict gatekeeper decides if those things happen.

If the boundaries match: The AI is safe by design (Structural Governance).
If the boundaries don't match: You will always have some risk and some wasted effort, no matter how many rules you add (Behavioral Governance).

The authors have proven this mathematically using computer code (Coq) to show that this "Order Taker" approach is the only way to guarantee that every action is governed.

1. Problem Statement

The paper addresses the fundamental structural failure in current AI governance frameworks regarding AI effects (actions performed in the world, such as API calls, database writes, and tool invocations).

The Two Boundaries: Every effectful system possesses two distinct boundaries:
1. Expressiveness Boundary: Defined by the system's architecture (languages, APIs, tools), representing everything the system can do.
2. Governance Boundary: Defined by policies and monitoring, representing everything the system is allowed to do.
The Divergence: In current deployments, these boundaries are defined independently, creating three regions:
1. Governed Capability (Overlap): The only functional region where capabilities are covered by policy.
2. Ungoverned Capability (Risk): Capabilities the system can perform but which lack governance coverage (e.g., data exfiltration via unmonitored side channels).
3. Governance Theater: Policies addressing capabilities the system does not possess (e.g., filtering credit card numbers in emails when the system has no access to credit card data).
The Core Issue: Behavioral governance (content filters, RLHF, monitoring) attempts to close this gap by adding layers after the fact. However, because AI agents have dynamic, Turing-complete expressiveness, the governance boundary can never structurally match the expressiveness boundary through behavioral means alone.

2. Methodology and Theoretical Foundation

The paper employs formal methods and theoretical computer science to analyze the governance problem.

Rice's Theorem (1953): The central theoretical tool. Rice's theorem states that for any non-trivial semantic property of a Turing-complete program, it is undecidable whether an arbitrary program possesses that property.
- Application: The property "does this program's effect comply with policy P?" is a non-trivial semantic property. Therefore, no algorithm can generally decide if an arbitrary Turing-complete AI agent will violate a governance policy.
Formal Verification: The claims are mechanized in Coq (454 theorems across 36 modules, zero admitted lemmas). This provides mathematical certainty regarding the structural properties of the proposed solution.
Architectural Analysis: The paper contrasts "behavioral" approaches (analyzing outputs) with "structural" approaches (architectural separation of concerns).

3. Key Contributions

A. The Impossibility of Behavioral Effect Governance

The paper proves that behavioral governance is structurally insufficient for effect control in Turing-complete systems.

Content filters, RLHF, and Constitutional AI operate on semantic properties (what the model says or intends).
Because the space of possible actions is combinatorial and Turing-complete, behavioral approaches can only produce approximations (false positives/negatives).
Monitoring is insufficient because it is a separate system; it observes but does not control, leading to inevitable coverage gaps that compound exponentially with the number of actions.

B. The Structural Alternative: Separation of Computation and Effect

The paper proposes a radical architectural shift: Separate Computation from Effect.

Computation Layer: Pure code that produces directives (structured data describing intended effects) but cannot perform I/O.
Governance Boundary: A single, mandatory layer that receives directives, validates them against policy (syntactic check), and executes the effect.
Result: Governance becomes a function of the directive (syntax), not the program (semantics). Since directives are structured data, their properties are decidable by construction.

C. Coterminous Governance

The paper defines Coterminous Governance as the state where the Expressiveness Boundary equals the Governance Boundary.

Definition: A system satisfies coterminous governance if (1) every expressible effect passes through a governance boundary (Safety), and (2) the primitives are expressive enough to target the desired computations (Sufficiency).
Properties:
- No Ungoverned Region: Nothing escapes governance by construction.
- No Theater Region: Every policy corresponds to a real capability.
- Compositional Reasoning: If components A and B are governed, their composition is automatically governed.
- Inherent Intelligibility: The governance record is the execution record itself.

D. Efficiency Consequences

Behavioral Governance: Additive cost. Each layer (filter, monitor) adds latency and compute proportional to throughput ( $k \times$ throughput).
Structural Governance: Subsumes infrastructure. Governance checks are part of the execution pipeline. The cost is linear with throughput only, with no separate system to maintain.
Empirical Data: On Apple Silicon, governed execution overhead was measured at 0.23ms median (vs. 0.24ms for ungoverned), proving that structural governance is indistinguishable from native execution speed.

4. Results

Theoretical Proof: Rice's theorem confirms that behavioral governance of Turing-complete agents is undecidable in the general case.
Formal Verification: The Coq proofs demonstrate that the proposed "Separation Principle" yields a system where the Safety and Sufficiency theorems hold, ensuring coterminous boundaries.
Performance: Structural governance introduces negligible latency compared to behavioral layers, which scale poorly as the number of governance layers increases.
Failure Modes: The paper demonstrates that without structural separation, "Risk" (ungoverned capabilities) and "Theater" (policies for non-existent capabilities) are structurally inevitable.

5. Significance and Implications

Paradigm Shift: The paper argues that AI safety cannot be achieved by "patching" existing architectures with more filters or monitors. It requires a fundamental re-architecture where effects are not direct capabilities of the code but mediated through a governed boundary.
Distinction of Levels: It clarifies that Content Governance (shaping what LLMs say) and Effect Governance (controlling what agents do) are distinct problems requiring different mechanisms. Content governance is behavioral; Effect governance must be structural.
Policy vs. Enforcement: The paper emphasizes that structural governance guarantees enforcement of a policy with mathematical certainty, but it does not guarantee the correctness of the policy itself. Policy design remains a human judgment problem.
Practical Limitations: The approach requires a complete re-architecture of existing agent frameworks (which currently mix I/O and logic). It cannot be retrofitted as a library. It also imposes a discipline on developers (pure computation vs. directives) that may impact prototyping speed.

Conclusion

"The Two Boundaries" concludes that for AI systems with agency, structural safety is the only path to zero-residual risk. Behavioral approaches can reduce risk but cannot eliminate the "ungoverned region" due to the undecidability of semantic properties in Turing-complete systems. Coterminous Governance, achieved by separating computation from effect, is the necessary architectural criterion to ensure that every action an AI takes is provably governed.