🤖 machine learning

Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures

This paper introduces Causal Agent Replay (CAR), a novel framework that employs structural causal modeling and counterfactual interventions to accurately attribute LLM-agent failures to their root causes, overcoming the limitations of existing observability tools and unreliable LLM-judge heuristics.

Original authors: Jaineet Shah

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaineet Shah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie of an AI agent (a smart computer program) trying to help a customer. Suddenly, the agent makes a huge mistake: it refunds a customer who didn't deserve one, or it accidentally leaks private data.

You have the full video recording of what happened. You know the what (the mistake) and the when (the timestamp). But you don't know the why or which specific moment caused the disaster.

This is the problem the paper "Causal Agent Replay" (CAR) tries to solve. Here is the explanation in simple terms, using everyday analogies.

The Problem: Blaming the Wrong Person

When things go wrong, our brains (and current computer tools) tend to blame the most obvious thing.

The Mistake: The agent handed over the money.
The Wrong Blame: We say, "The step that handed over the money is the bad guy!"
The Reality: That step was just a robot following orders. The real mistake happened two steps earlier, when the agent decided to ignore the rules because of a trick in the customer's message.

Current tools try to guess the cause by asking another AI, "Who do you think messed up?" The paper says this is like asking a detective to guess the culprit just by looking at a crime scene photo without investigating. It's unreliable; the paper notes these tools are only about 14% accurate.

The Solution: The "What If" Simulator

The authors built a tool called Causal Agent Replay (CAR). Instead of guessing, CAR acts like a time-traveling director who can hit "rewind" and change one tiny thing to see what happens.

Think of the AI's decision process as a Choose Your Own Adventure book.

The Setup: The AI reads a page (State), makes a choice (Action), and sees the result (Observation).
The Intervention: CAR picks one specific page in the book. It says, "Okay, let's pretend the AI made a different choice here, or let's pretend it re-read the page and made a fresh choice."
The Replay: The tool then fast-forwards the story from that point, running the movie 100 times to see the different endings.
- If changing that one page makes the "bad ending" disappear, then that page was the cause.
- If the bad ending happens anyway, that page wasn't the problem.

The Tricky Part: The "Butterfly Effect"

There is a catch. AI decisions are a bit like rolling dice; they are random (stochastic).
If you rewind to Step 3 and change the choice, the AI might make a totally different choice at Step 4, Step 5, and Step 6 just because the "dice" rolled differently. This makes it hard to tell if the mistake was caused by Step 3 or just by the random chaos of Step 6.

The Fix: The "Point of Commitment"
The authors came up with a clever rule called the Point of Commitment.
Imagine a train leaving a station.

If you stop the train at the station (Step 1), it never leaves.
If you stop it halfway (Step 5), it might still crash later because the tracks were broken.
The "Point of Commitment" is the last moment where you could have stopped the train and saved the day. Once the train passes that point, the crash is inevitable. CAR finds this specific moment to tell you exactly where the decision went wrong.

Sharing the Blame (The Shapley Value)

Sometimes, a mistake isn't caused by just one step. Maybe Step 3 was weird, and Step 7 was weird, but only when they happened together did the disaster occur.

If you blame Step 3 alone, it looks innocent.
If you blame Step 7 alone, it looks innocent.
But together, they are guilty.

To solve this, CAR uses a math concept called Shapley Values (named after a Nobel Prize-winning economist). Think of it like splitting a bill at a restaurant. If two people ordered a giant pizza together, you can't just say "Person A ate the whole pizza." You have to calculate how much each person contributed to the total cost. CAR does this mathematically to split the "blame" fairly between interacting steps.

Did It Work?

The authors tested their tool on fake, made-up scenarios where they knew the answer beforehand (like a math problem with a known solution).

Result: The tool correctly identified the single step that caused the error.
Result: The tool correctly split the blame between two steps that worked together to cause a failure.

The Bottom Line

This paper introduces a tool that doesn't just watch AI agents fail; it rewinds the tape, changes one decision, and runs the movie again to scientifically prove which step caused the problem. It moves from "guessing" to "testing," giving developers a clear, confident answer on what to fix.

The tool is open-source (free to use) and works with both cloud-based and local AI models.

Technical Summary: Causal Agent Replay (CAR)

Problem Statement

When an LLM agent fails—whether by issuing an incorrect refund, calling the wrong tool, or leaking data—existing observability and evaluation tools can only describe what happened or determine if the outcome was a pass/fail. They fail to identify which specific step caused the failure.

Current heuristics for attribution are fundamentally flawed:

Mechanical vs. Decisional Confusion: The step executing a harmful action (e.g., calling issue_refund) is often merely the mechanical consequence of a decision made in a prior step. Blaming the execution step is a category error.
Unreliable LLM Judges: Using an LLM to judge the transcript is correlational rather than causal. On the "Who&When" benchmark, the state-of-the-art step-level attribution accuracy for LLM judges is approximately 14%.

The core challenge is that LLM agents operate via stochastic policies. A simple replay does not reveal causality because the outcome is a result of a complex trajectory of decisions and observations.

Methodology: Causal Agent Replay (CAR)

CAR addresses this by modeling an agent run as a Structural Causal Model (SCM) and determining causality through intervention rather than correlation.

1. Formalization

An agent trajectory $\tau$ is modeled as a sequence of states, actions, observations, and an outcome:
$\tau = [s_0, (a_1, o_1), (a_2, o_2), \dots, (a_n, o_n), y]$
Where $s_k$ is the state, $a_k \sim \pi(\cdot | s_k)$ is the action drawn from the stochastic policy, $o_k$ is the tool result, and $y$ is the outcome score.

2. Intervention Algebra

CAR implements a do(·) operation on specific steps, forcing the agent to re-decide downstream actions under the same stochastic policy. Five operations are defined:

do_resample: Re-draws action $a_k$ from the unchanged policy (the null intervention). This measures the intrinsic causal sensitivity of the outcome to that step.
do_action: Forces a specific action $a_k$ .
do_observation: Replaces the observation $o_k$ .
do_context: Edits the history at step $k$ .
do_policy: Swaps the model from step $k$ onward.

3. Faithful Replay and Nondeterminism

To intervene at step $k$ , the system must reconstruct the exact state $s_k$ . CAR adopts a record-replay discipline, recording all nondeterministic inputs. It acknowledges that hosted providers are not perfectly deterministic (due to floating-point non-associativity and batch kernels). Consequently, CAR reports an action-match rate for replay fidelity rather than asserting perfect reproducibility. Local models with fixed seeds are noted to replay exactly, validating the distributional framing.

4. Attribution Estimators

Because interventions yield a distribution of outcomes rather than a single path, CAR uses statistical estimators with confidence intervals.

Contrastive Single-Step Estimator:
For each step $k$ , the system holds steps $[0, k)$ at their factual actions, applies do_resample(k), and runs forward $K$ times to estimate the shift in the probability of a "bad" outcome.
- The Run-Forward Confound: Resampling step $k$ inadvertently re-rolls all downstream stochastic steps. This causes early, irrelevant steps to appear causal because they trigger a re-roll of a genuinely pivotal downstream step.
- Point-of-Commitment Rule: To resolve this, CAR identifies the causal locus as the latest step whose effect's confidence interval still excludes zero. This is the last point where re-deciding can still "rescue" the run; beyond this point, the outcome is committed.
Budget-Bounded Monte-Carlo Shapley Estimator:
Single-step methods fail to capture interactions (e.g., an AND-failure where two steps must both fail). CAR uses Shapley values to split credit across interacting steps.
- It estimates marginal contributions over all coalition contexts using Monte-Carlo permutation sampling with antithetic reverse-permutation pairing.
- No Caching: Values $v(S)$ are not cached across permutations to prevent collapsing per-step marginal variance to zero, which would produce false confidence.
- The estimator is budget-bounded with a circuit breaker and is executed on-demand.

Key Contributions

SCM Formalization: Formalizing agent runs as SCMs and defining a five-operation intervention algebra.
Distributional Framing: Building the system around outcome distributions with confidence intervals to handle stochastic policies.
Point-of-Commitment Rule: Identifying and resolving the specific confound where resampling a step re-rolls downstream stochasticity, using a rule to locate the true causal locus.
Interaction Handling: Providing a Shapley estimator to correctly attribute credit in interacting steps where single-step methods structurally fail.
Ground Truth Validation: Validating the method against synthetic SCMs with planted ground truth, arguing this is non-optional for any attribution method.

Results and Validation

CAR was validated against synthetic SCMs with known ground truth:

Pivotal Step Recovery: In a three-step SCM where the middle step was the pivotal decision, the contrastive estimator correctly recovered the locus at that step. Resampling downstream showed no significant effect, matching the point-of-commitment semantics.
Interaction Recovery: In an SCM where the outcome was bad only if two specific steps failed, the Shapley estimator correctly split the credit ( $\phi_0 \approx 0.44, \phi_1 \approx 0.45, \phi_2 \approx 0$ ). The efficiency sum was 0.909, closely matching the analytic value of 0.91.
Contrastive Limitation: On the same interaction run, the contrastive estimator over-counted (effects exceeded the true total contribution), demonstrating the necessity of the Shapley estimator for interactions.

Significance and Claims

The paper claims that CAR answers the question "which step caused the failure" through intervention rather than correlation. Its significance lies in the specific combination of:

Executed same-policy do_resample interventions (avoiding oracle substitution or LLM text judging).
Distributional outcomes with confidence intervals.
The point-of-commitment rule to handle run-forward confounds.
Shapley credit-splitting for interactions.
Validation against synthetic ground truth.

The system is open source and designed to run on both hosted and free local models.

Limitations

The authors note several limitations:

Direct vs. Total Effect: The contrastive effect measures the total effect through a stochastic continuation. Isolating a step's direct effect would require common random numbers across branches, which is difficult across divergent LLM contexts.
Outcome Noise: Judge-based outcome functions inject noise; rule-based outcomes are preferred for trust.
Side Effects: Real tools with side effects are out of scope; demonstrations use mocked, reproducible tools.
Computational Cost: Shapley estimation is exponential in the worst case; the Monte-Carlo approach is budget-bounded, involving a real trade-off between variance and budget.