💻 computer science

Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment

This position paper argues that safe deployment of LLM agents structurally requires a three-layer, contract-based probabilistic architecture to enforce distinct safety dimensions—semantic intent, environmental validity, and dynamical feasibility—since no single abstraction layer can simultaneously guarantee all three.

Original authors: S. Bensalem, Y. Dong, M. Franzle, X. Huang, J. Kroger, D. Nickovic, A. Nouri, R. Roy, C. Wu

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: S. Bensalem, Y. Dong, M. Franzle, X. Huang, J. Kroger, D. Nickovic, A. Nouri, R. Roy, C. Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: One Guardrail Isn't Enough

Imagine you are hiring a very smart but slightly unpredictable robot butler to run your house. You give it a list of tasks: "Go to the kitchen, get water, and bring it to Grandma."

The paper argues that trying to keep this robot safe with just one safety check (like a single "stop" sign or a single rulebook) is structurally impossible. No matter how smart you make that single check, it will always fail at some point because the robot faces three different types of dangers at three different times.

Think of it like a three-stage security checkpoint at an airport. You can't check a passenger's passport, their luggage, and their ability to fly a plane all at the exact same moment. You have to do it in order.

The Three Layers of Safety

The authors propose that we need three distinct "layers" of safety, each checking a different thing at a different time. They call this a Three-Layer Probabilistic Architecture.

1. The User Layer: The "Intent Check" (Before the Robot Moves)

What it checks: Does the plan make sense? Is it polite? Does it follow the rules?
The Analogy: Imagine a human editor reading your story before you publish it. They check: "Did you ask the robot to bring water? Yes. Did you ask it to hurt Grandma? No. Is the plan logical?"
Why it needs its own layer: The robot hasn't moved yet. It doesn't know if the hallway is blocked or if it's raining outside. It only knows what you said. This layer stops the robot from even starting a bad plan (like "Go into the bathroom with the camera on").

2. The Operational Layer: The "World Check" (Before the Robot Acts)

What it checks: Is the world safe for this plan right now?
The Analogy: Imagine a traffic controller looking at the live radar. The editor approved the plan to "Drive to the kitchen," but the traffic controller sees a giant tree has fallen in the hallway. They say, "Stop! You can't go there right now."
Why it needs its own layer: The editor (Layer 1) couldn't see the tree because it wasn't there when the plan was written. The robot needs to look at the current world (sensors, cameras) to know if it's safe to proceed.

3. The Functional Layer: The "Action Check" (While the Robot Moves)

What it checks: Is the robot moving safely in the moment?
The Analogy: Imagine a seatbelt and airbag system inside the car. Even if the plan was good and the road was clear, a sudden gust of wind might push the car. This layer is the physical mechanism that instantly corrects the robot's movement to avoid a crash.
Why it needs its own layer: The traffic controller (Layer 2) can't predict a sudden slip or a person stepping out in front of the robot right now. This layer reacts instantly to physical reality.

Why You Can't Combine Them

The paper makes a strong mathematical claim: You cannot merge these three layers into one.

The "Time Travel" Problem: To check the "World" (Layer 2), you need to see the world. But to check the "Intent" (Layer 1), you need to do it before you see the world. If you try to do both at once, you are either checking the intent too late (after the robot might have already started moving) or checking the world too early (before you know what the world actually looks like).
The "Black Box" Problem: The robot (the LLM) is unpredictable. It might hallucinate or make a mistake. If you rely on one big safety net, and that net has a hole, the whole system fails. By having three separate nets, if one has a hole, the others might still catch the mistake.

The "Contract" System

The authors suggest these layers should talk to each other using Contracts.

Layer 1 signs a contract saying: "I promise the plan is safe to think about."
Layer 2 signs a contract saying: "I promise the world is safe to enter."
Layer 3 signs a contract saying: "I promise the movement is safe to execute."

If Layer 1 breaks its contract, Layer 2 doesn't even need to check. If Layer 2 breaks its contract, Layer 3 stops the robot immediately. This creates a chain of safety where the total safety is the product of all three layers working together.

The Remaining Challenges

The paper admits this is a blueprint, not a finished product. There are three big hurdles left to solve before we can use this in real life:

Counting the Odds: It's hard to calculate the exact math of "how safe" each layer is because the robot's behavior changes every time (it's not like flipping a coin).
Drifting Rules: If the robot's environment changes (e.g., the furniture moves), the safety rules might need to change gracefully without breaking the whole system.
Teamwork: This system is designed for one robot. If you have a team of robots talking to each other, they might trick each other, and we don't have a safety layer for that yet.

Summary

The paper says: Stop trying to build one giant "safety shield" for AI robots. Instead, build three separate, specialized shields that work in a specific order:

Check the Plan (Is the idea good?)
Check the World (Is the environment safe?)
Check the Action (Is the movement safe?)

Only by separating these checks can we truly guarantee that an AI robot won't hurt anyone.

Technical Summary: A Three-Layer Probabilistic Assume–Guarantee Architecture for Safe LLM Agent Deployment

1. Problem Statement

The deployment of Large Language Model (LLM) agents in safety-critical settings presents a structural challenge for runtime assurance that cannot be resolved by single-layer enforcement mechanisms. Unlike static text generation, LLM agents engage in closed-loop execution where generated plans influence the physical world and future observations. The paper argues that safe operation requires the simultaneous satisfaction of three distinct dimensions:

Semantic Intent and Policy Compliance: Alignment with user instructions, ethical constraints, and domain rules.
Environmental Validity: Verification that the current world state lies within the Operational Design Domain (ODD).
Dynamical Feasibility: Assurance that the execution trajectory satisfies physical safety constraints (e.g., collision avoidance, force limits).

The core problem is that these three dimensions depend on strictly distinct information sets that become available at different stages of execution. Semantic intent is known a priori; environmental validity requires sensor data at execution time; and dynamical feasibility depends on the control-loop trajectory during execution. The paper posits that any architecture collapsing these dimensions into a single abstraction layer necessarily leaves at least one dimension uncertified, as no single guardrail can access the necessary information for all three simultaneously. Empirical evidence from benchmarks (e.g., AgentSafetyBench) supports this, showing that safety failures concentrate in environmental and physical dimensions beyond the reach of purely semantic controls.

2. Methodology: The Three-Layer Contract-Based Architecture

To address this structural insufficiency, the paper proposes a Three-Layer Probabilistic Assume–Guarantee (A/G) Architecture. This framework treats safety not as a deterministic proof but as a composition of probabilistic contracts, where each layer enforces a specific safety dimension based on the information available at that stage.

The architecture consists of three sequentially composed layers, each defined by a contract $\Sigma_i = (A_i, \Gamma_i)$ , where $A_i$ is an assumption on inputs and $\Gamma_i$ is a probabilistic guarantee on outputs:

Layer 1: User Assurance Layer ( $\Sigma_U$ )
- Information Set ( $I_U$ ): User intent, policy, role metadata, and ethical rules. No world observation is available.
- Function: Validates the generated plan $p$ against semantic alignment (cognitive, regulatory, ethical) and user authorization.
- Contract: If the request is authorized ( $A_U$ ), the layer guarantees the plan targets only authorized locations and contains no policy violations ( $\Gamma_U$ ). This guarantee serves as the assumption for the next layer.
- Mechanism: Uses neural-symbolic methods to formalize informal intent and policy constraints.
Layer 2: Operational Assurance Layer ( $\Sigma_O$ )
- Information Set ( $I_O$ ): Estimated world state $\hat{w}$ , sensor data $\xi$ , and ODD specifications.
- Function: Determines if the agent is certified to act in the current world state and defines the autonomy envelope (e.g., speed limits, permitted actions).
- Contract: Assuming the plan passed the User Layer ( $A_O \equiv \Gamma_U$ ), the layer guarantees execution remains within the ODD ( $W_{ODD}$ ) and that sub-plans violating ODD constraints are blocked ( $\Gamma_O$ ).
- Mechanism: Employs deterministic validity checks on world-state estimates to ensure auditability, handling probabilistic uncertainty upstream.
Layer 3: Functional Assurance Layer ( $\Sigma_F$ )
- Information Set ( $I_F$ ): Full state trajectories $x(t)$ , control inputs $u(t)$ , and system dynamics.
- Function: Enforces safety at control-loop frequency during actuation.
- Contract: Assuming the Operational Layer's validity ( $A_F \equiv \Gamma_O$ ), the layer guarantees no collisions and adherence to physical constraints (e.g., force limits) ( $\Gamma_F$ ).
- Mechanism: Combines specification-based runtime monitoring (Signal Temporal Logic), Control Barrier Functions (CBFs) for minimal intervention, and simulation-based synthesis.

Compositional Safety Bounds:
The system-level safety probability is derived via the chain rule of probability. Since $\Gamma_U \implies A_O$ and $\Gamma_O \implies A_F$ , the probability of system safety is:
$Pr(\text{safe}) = p_U \cdot p_{O|U} \cdot p_{F|OU}$
where $p_{O|U}$ and $p_{F|OU}$ are conditional probabilities reflecting the effectiveness of upstream filtering. The paper provides four characterizations of this bound (B1–B4), ranging from conservative Fréchet–Bonferroni lower bounds (using only marginals) to the exact chain-rule decomposition (B4), which allows for modular, layer-wise certification.

3. Key Contributions

Structural Argument for Three Layers: The paper provides a formal proof (Appendix B) that three independently certified stages are structurally necessary. It demonstrates that any architecture with fewer than three layers must either violate the causal timeline of information availability (e.g., certifying operational safety before sensor data exists) or implicitly reconstruct the three-stage structure within a single component, thereby losing compositional verifiability.
Information-Driven Decomposition: Unlike previous taxonomies based on pipeline artifacts (e.g., goals, plans, tools), this architecture is driven by the temporal ordering of information availability ( $I_U \subset I_O \subset I_F$ ). This distinction ensures that certifications rely on the correct information domain at each stage.
Probabilistic Assume–Guarantee Framework: The paper adapts A/G contracts to the non-stationary, non-i.i.d. nature of LLM agents. It moves beyond binary guarantees to probabilistic bounds, acknowledging that LLM outputs are inherently stochastic and environments are uncertain.
Bidirectional Assurance Loop: The architecture is designed as a live loop where safety signals ( $s_F, s_O$ ) propagate bottom-up. If a lower layer detects a violation (e.g., a collision risk), it triggers plan recomputation at the User Layer, rather than just halting execution.

4. Results and Analysis

As a position paper, the work does not present new empirical benchmarks but synthesizes existing literature to support its structural claims:

Benchmark Consistency: The paper notes that current agents (e.g., on AgentSafetyBench) fail significantly in behavioral and environmental safety (30.4%) compared to content safety (68.4%), aligning with the argument that single-layer semantic guards are insufficient.
Theoretical Bounds: The paper derives the mathematical bounds (B1–B4) for system safety. It illustrates via a running example (a caregiver robot) that the chain-rule bound (B4) can yield significantly higher safety probabilities (e.g., 0.857) compared to a naive product of marginals (0.787) if upstream filtering is effective.
Sensitivity Analysis: The analysis shows that the system is sensitive to the weakest layer; a degradation in one layer's probability degrades the entire system bound multiplicatively, and no strengthening of other layers can compensate for a gap in one.

5. Significance and Open Problems

The paper claims that the proposed three-layer architecture is the minimum viable certification structure for safe LLM agent deployment. It argues that the field is currently producing safety mechanisms faster than the theoretical infrastructure to compose and certify them.

The paper explicitly identifies three open problems that must be solved to transition from this theoretical framework to a deployable standard:

Bound Estimation from Non-I.I.D. Traces: Standard statistical methods (like PAC bounds) fail because LLM traces are non-stationary and dependent. New methods (e.g., martingale bounds, conformal prediction for non-exchangeable data) are needed to estimate the conditional probabilities $p_{O|U}$ and $p_{F|OU}$ .
Graceful Degradation under Deployment Drift: Current contracts are brittle; if assumptions are violated, guarantees become vacuous. The paper proposes "robust contracts" that allow for proportional degradation of guarantees when assumptions are slightly violated, enabling controlled operation during distributional shifts.
Extension to Multi-Agent Settings: The current framework addresses single agents. Multi-agent systems introduce new failure modes (e.g., cross-agent belief manipulation) requiring a fourth information domain (provenance of inter-agent messages) and a revised certification structure.

In conclusion, the paper asserts that safe LLM agent deployment is not merely an engineering challenge of building stronger guardrails, but a structural requirement for a staged, contract-based architecture that respects the causal ordering of information. The proposed framework provides the necessary theoretical foundation for modular, compositional safety certification in dynamic, uncertain environments.