🤖 AI

From Agent Failure Paths to Quantified Residual Risk: A Compositional Framework for Resilient Agentic AI

This paper introduces CPSAINT and FRIESA-K, a compositional framework that bridges the gap between detailed failure mechanisms and quantified residual risk for agentic AI by mapping seven-layer integrity decompositions to state-dynamic-based risk estimates, enabling consistent cross-domain reasoning and formalized trust.

Original authors: Hassan Karim, Sai Sitharaman, Deepti Gupta, Danda B. Rawat

Published 2026-07-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hassan Karim, Sai Sitharaman, Deepti Gupta, Danda B. Rawat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a robot that can think for itself, like a digital chef that not only reads a recipe but also decides which ingredients to buy, drives to the store, and cooks the meal. This is the world of "Agentic AI." Unlike a simple chatbot that just answers questions, these agents take action, make plans, and touch the real world. But here's the tricky part: when things go wrong, they don't just fail in one spot. A mistake in the robot's "brain" (its code) might cause a glitch in its "eyes" (sensors), which leads to a bad decision, which finally makes the robot crash into a wall.

For a long time, experts have had two different ways to look at these failures. One group uses maps to trace the exact path of the crash, showing every step of the mistake, but they can't tell you how bad the crash will be. The other group uses math to guess how bad the crash might be, but they treat the robot's brain as a mysterious black box, ignoring how the mistake actually happened. The big question is: Can we combine the map of the mistake with the math of the danger to get a clear picture of the risk?

This paper, titled "From Agent Failure Paths to Quantified Residual Risk," tries to solve that puzzle. The authors, a team of researchers from various universities and tech companies, propose a new framework called CPSAINT and FRIESA-K. Think of this as a new set of tools that lets us trace a robot's failure step-by-step and then instantly calculate exactly how much danger that specific failure creates.

Here is how their idea works, using a simple story. Imagine a warehouse robot trying to stack boxes without hitting a human. The authors break the robot's world into seven layers, like a seven-story building: the physical body, the sensors, the data, the computer brain, the actions, the environment, and time. If the robot fails, it's usually because a problem started on one floor and traveled up or down to another. Maybe the "sensors" (floor 2) got corrupted data, which confused the "brain" (floor 4), causing a "delay" in thinking, which led to a "bad action" (floor 6) just as "time" (floor 7) ran out.

The paper introduces a special formula, FRIESA-K, to measure the risk of this specific journey. It looks at how often the problem happens, how easy it is to exploit, how bad the final crash would be, and how much the robot's safety controls can stop it. Crucially, instead of just guessing how good the safety controls are, the authors use a mathematical model (a type of "absorbing Markov model") to simulate the robot's state. They ask: "If we have this specific safety control, how much does it lower the chance of a catastrophe within a specific time?" This gives them a real, calculated number for how effective the safety net is.

They tested this idea on two very different robots. The first was a warehouse robot that had to stop in a split second (0.75 seconds) to avoid a physical collision. The second was a banking agent that processes loan documents, where the risk isn't a crash, but a loss of trust or a broken audit trail. In both cases, the same seven-layer map and the same math formula worked perfectly.

For the warehouse robot, the simulation showed that if the safety controls were strong, the risk dropped dramatically. The math predicted a residual risk score of about 2,563,283 (a comparative number, not a dollar amount) with a specific set of controls. When they ran 5,000 simulated crash scenarios, the average risk was 2,553,038, proving the math was consistent.

For the banking agent, the story was different. The immediate physical risk was lower, but the risk of losing "governance observability" (the ability to see what went wrong later) was high. The authors added a special "penalty" to the math to account for this. Even though the time window was short (0.75 seconds), the framework showed that the banking agent had a residual risk score of 966,412. The key finding here is that the framework could separate the "operational risk" (did the robot crash?) from the "governance risk" (can we explain why it failed?), which is something older methods couldn't do well.

The authors are careful to say this is a simulation and a theoretical framework, not a magic fix that has been tested in every real-world factory or bank yet. They showed that their method works mathematically and in their specific simulations, but they admit that real-world calibration is still needed. They also noted that for very short time windows (like 0.75 seconds), the "governance penalty" is small, but it would become huge if the time window were longer (like a 12-hour shift).

In short, this paper doesn't just say "robots are risky." It gives us a way to draw the exact path of a robot's failure and then use a calculator to tell us exactly how much that specific failure path matters. It bridges the gap between "here is how it broke" and "here is how bad it is," offering a new way to build safer, more trustworthy AI agents that can work alongside us in the real world.

Technical Summary: From Agent Failure Paths to Quantified Residual Risk

Problem Statement
Agentic AI systems, which autonomously pursue multi-step goals through planning and environmental interaction, are operating across trust boundaries faster than current risk models can represent. Existing approaches suffer from a dichotomy: structural safety methods (e.g., STAMP, STPA) provide causal failure pathways but lack transferable residual-risk estimates, while quantitative risk methods (e.g., CVaR, adversarial risk analysis) provide risk estimators but treat the internal failure path as a black box. Consequently, no existing framework simultaneously answers two critical questions: (1) through what specific mechanism does integrity fail, and (2) how much residual risk results from that specific path? This gap prevents cross-domain reasoning and the quantitative optimization of controls for path-dependent failure modes in agentic and cyber-physical systems.

Methodology
The paper proposes a compositional framework coupling a structural decomposition with a quantitative risk functional.

CPSAINT (Structural Decomposition): The authors define a seven-layer integrity decomposition to model failure propagation:
- Layers: Physical state ( $P$ ), Sensors ( $S$ ), Data ( $D$ ), Compute ( $C$ ), Actuators ( $A$ ), Environment ( $E$ ), and Time ( $T$ ).
- Failure Modes: The system utilizes a five-mode alphabet: corruption, delay, omission, replay, and coupling abuse.
- Path Semantics: A valid agent failure path ( $\pi$ ) is an ordered sequence of layer-mode pairs that respects the propagation relation of the instantiated system. This allows diverse domains (e.g., robotics and banking) to share a common grammar while preserving domain-specific path content.
FRIESA-K (Quantitative Risk Functional): The framework maps each failure path to a quantified risk instance using the function:
$R(\pi, \tau, u) = \frac{F(\pi) R_i(\pi) E(\pi) S(\pi) A(\pi)}{K(\pi, \tau, u)}$
- Numerator: Represents threat frequency ( $F$ ), reachability ( $R_i$ ), exploitability ( $E$ ), severity ( $S$ ), and amplification ( $A$ ). Severity is defined by terminal consequences, while amplification captures cross-layer impact broadening.
- Denominator ( $K$ ): Represents dynamic control effectiveness. Unlike static expert scores, $K$ is derived from a controlled absorbing Markov model. It is defined as the ratio of the baseline catastrophe probability to the controlled catastrophe probability by a finite horizon $\tau$ . This grounds control effectiveness in state dynamics rather than arbitrary assignment.
Governance Observability: To address governance without redefining the base risk equation, the authors introduce a separate additive penalty term, $R_{gov}$ . This term quantifies the dwell time in "warning states" (degraded but non-catastrophic) that lack observability. The final reported score is $R_{res} = R + w_{gov} R_{gov}$ .
Composability: The paper formalizes a composition theorem stating that a valid CPSAINT path induces a well-defined FRIESA-K risk instance without altering the functional form. This ensures that the same layer grammar and variable semantics apply across different domains.

Key Contributions

CPSAINT Definition: A seven-layer structural decomposition with explicit path semantics that unifies failure analysis across physical and software-intensive agents.
FRIESA-K Definition: A risk functional with locked variable boundaries and a dynamic resistance term ( $K$ ) constructed from controlled stochastic dynamics (Markov chains).
Composition Theorem: A formal proof that valid structural paths induce well-defined risk instances, preserving functional form across domains.
Governance Integration: A mechanism to report governance-observability degradation via an additive penalty, separating operational risk from assurance/accountability risks.
Empirical Validation: Demonstration of the framework on two contrasting scenarios: a hard real-time warehouse robot and a governance-instrumented financial-services agent.

Results
The framework was evaluated using Gillespie SSA fault-injection simulations and deterministic sensitivity analysis on two cases:

Warehouse Robot: A path involving sensor corruption, data corruption, compute delay, and actuation coupling abuse. The results showed that the residual risk score dropped sharply with improved controls due to high containment and recovery elasticities. The operational term dominated the risk score.
Financial Services Agent: A path involving data corruption, compute corruption, timing delays, and external coupling abuse. While the operational risk was lower, the framework successfully modeled the structural governance mechanism. The results demonstrated that while the governance penalty ( $R_{gov}$ ) was numerically small at short horizons (0.75s), the structural separation allows it to become material at longer evaluation windows (e.g., hours), distinguishing between immediate operational safety and long-term assurance degradation.

Sensitivity analysis confirmed that the risk score is linear in failure frequency ( $F$ ) and inversely proportional to control effectiveness ( $K$ ). The framework successfully distinguished between the risk-reduction value of data-layer detection versus actuation-layer containment.

Significance and Claims
The paper claims to provide a "concise mechanism-to-magnitude pipeline" for resilient agentic and embodied AI. Its primary significance lies in:

Unifying Views: It bridges the gap between structural failure causality and quantitative risk estimation, avoiding the "black box" approach of purely statistical methods and the lack of quantification in purely structural methods.
Cross-Domain Reasoning: By maintaining a fixed layer grammar and variable semantics, the framework supports reasoning across materially different domains (e.g., robotics and finance) without redefining the risk functional.
Explicit Assumptions: It forces explicit assumptions about failure paths and control effectiveness, moving away from monolithic, brittle risk scores.
Dynamic Control Interpretation: It derives control effectiveness from measurable changes in catastrophe probability via state dynamics, rather than relying on informal expert scoring.

The authors note that the framework is a "compact kernel" for composable trust. They acknowledge limitations, including the need for domain-specific calibration of parameters and the current assumption of independence among numerator terms, but assert that the framework successfully establishes a formal linkage between structural failure mechanisms and quantified residual risk.

Technical Summary: From Agent Failure Paths to Quantified Residual Risk

More like this