💻 computer science

Did It Happen? Counterfactual Evaluation of LLM Agent Recovery from Ambiguous Tool Outcomes

This paper introduces a counterfactual benchmark demonstrating that while ambiguous tool timeouts impose a 50% success ceiling on LLM agent recovery, implementing stable idempotency contracts enables perfect recovery, whereas relying solely on status information yields only partial improvement.

Original authors: Shengyao Sun

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shengyao Sun

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a digital assistant that can not only answer questions but also perform tasks in the real world: creating accounts, moving data, or deploying software updates. These are known as AI agents. For years, researchers have focused on whether these agents can choose the right tools and follow instructions. But as these systems move from chat windows into critical infrastructure, a new, more dangerous problem has emerged. It is not about whether the agent knows what to do, but whether it knows what has happened. In the messy reality of computer networks, a tool might fail to start, or it might start successfully and then lose its connection before sending back a "success" message. To the agent, both scenarios look exactly the same: a silence, or a timeout. This creates a blind spot. If the agent guesses wrong and tries again, it might accidentally create two accounts instead of one. If it guesses to stop, it might leave a task half-finished. The core question for the future of reliable automation is how to navigate this silence without causing chaos.

A researcher at Shanghai Jiao Tong University set out to measure exactly how well current AI models handle this specific type of confusion. They built a controlled testing ground designed to mimic the worst-case scenario: a moment where a computer system stops responding, leaving the AI unsure if the action it requested actually took place. The researcher did not simply ask the AI to guess; they created a rigorous experiment where every single test case was a pair of hidden realities. In one version of the test, the action never happened. In the other, the action happened perfectly, but the confirmation was lost. Crucially, the AI saw the exact same "timeout" message in both versions. The only difference was the hidden truth of what the computer system had actually done. The goal was to see if the AI could recover correctly in both worlds, or if it was doomed to fail in one of them.

The study tested three different ways of helping the AI recover from this silence. The first approach was simply to give the AI a prompt asking it to be careful and reliable. The second approach gave the AI a tool to check the status of the system, allowing it to see if the action had actually occurred. The third approach changed the rules of the tool itself, making it safe to repeat the action without causing duplicates, a concept known as idempotency. The researcher ran these tests across eighty-one different software engineering scenarios, ranging from creating a single file to managing complex chains of resources. They used a specific AI model, qwen-plus, as their primary test subject, running the experiment hundreds of times to ensure the results were not just luck.

The results were stark and revealing. When the AI was only given a polite reminder to be careful, it performed no better than a coin flip. It succeeded in about half of the cases, which is the theoretical maximum when you have no information about what happened. The AI could not distinguish between a failed attempt and a lost confirmation, so it either repeated an action that had already succeeded or gave up on one that had failed. When the researcher gave the AI a way to check the status of the system, performance improved significantly, reaching about eighty percent success. However, this was not a perfect solution. In certain complex workflows involving a sequence of steps, the AI would successfully check the status but still fail to choose the correct next step, showing that having information does not guarantee the ability to use it correctly.

The most effective solution was the third one: changing the tool itself to be safe against repetition. When the tool was designed so that repeating the action with the same identifier would simply be ignored if it had already run, the AI achieved a perfect success rate. It could retry the action as many times as needed without fear of duplication, and the system would always end up in the correct state. This finding suggests that the most reliable path forward is not to rely on the AI's ability to reason its way out of a blind spot, but to build the safety mechanisms directly into the tools the AI uses. The researcher also noted that even when the AI got the final state of the system right, it sometimes failed to follow the strict formatting rules required by the software, proving that a correct outcome and a correct report are two different things.

The study concludes that the problem of ambiguous tool outcomes is not a prompting issue that can be solved by better instructions. It is a structural issue that requires either clear information or built-in safety. The researcher found that without a way to see the hidden state or a tool that prevents duplicates, the AI is fundamentally limited to a fifty percent success rate in these specific scenarios. While giving the AI a status check helps, it is not a silver bullet, as the AI can still make mistakes in interpreting what it sees. The only method that guaranteed a perfect outcome in their simulations was to design the tools so that retrying is harmless. This work provides a clear map for engineers building the next generation of AI agents: if you want your system to be reliable, you must either give it eyes to see what has happened or build it so that it cannot hurt itself by trying again.

Technical Summary: "Did It Happen? Counterfactual Evaluation of LLM Agent Recovery from Ambiguous Tool Outcomes"

Problem Statement

As Large Language Model (LLM) agents increasingly perform side-effecting actions in external systems (e.g., creating issues, deploying resources, updating states), a critical failure mode emerges: ambiguous tool outcomes. Specifically, when an agent receives a timeout after requesting a non-idempotent action, the observation is consistent with two mutually exclusive hidden histories:

H=0 (Not Executed): The request never reached the tool or failed before execution.
H=1 (Executed, Response Lost): The action executed successfully, but the response was lost in transit.

These two states require opposite recovery strategies: H=0 requires a retry, while H=1 requires stopping to avoid duplication. However, because the agent observes only the identical timeout, it cannot distinguish between them. Blindly retrying fixes H=0 but duplicates H=1; blindly stopping fixes H=1 but leaves H=0 incomplete. Existing benchmarks often fail to isolate this ambiguity, either testing only pre-execution failures (rewarding blind retries) or failing to control for model stochasticity when comparing recovery strategies.

Methodology

Counterfactual Benchmark Design

The paper introduces a paired counterfactual benchmark where every experimental unit consists of two hidden worlds (H=0 and H=1) that emit the exact same public transcript (a timeout) until a visible tool response diverges.

Balanced Exactly-Once Success Rate (BEOSR): The primary metric is defined as $B = (Y_0 + Y_1) / 2$ , where $Y_h$ is 1 if the hidden world $h$ reaches the exact target state. This metric evaluates whether a single policy can succeed in both hidden histories simultaneously.
Theoretical Ceiling: The author proves that without an information-bearing affordance (like a status check) or a stable idempotency contract, any shared policy has a theoretical maximum BEOSR of 0.5. This is because a policy must choose either "stop" or "retry," which succeeds in only one of the two hidden worlds.

Experimental Setup

Task Suite: 81 software-engineering templates derived from 27 real-world contexts (GitHub, Kubernetes, Stripe, AWS EC2) instantiated across three state topologies:
1. Create-once: Risk of duplicate entities.
2. Ordered-transition: Risk of overshooting stages or stalling.
3. Reserve-then-link: Risk of overconsumption, orphaned resources, or unlinked allocations.
Interface Conditions: The study evaluates four conditions:
- C0/C1 (Baseline/Prompt Only): No explicit recovery affordance; relies on prompt advice.
- C2 (Status Query): A read-only tool allows the agent to query the hidden state (observable but requires correct interpretation).
- C3 (Stable Idempotency Key): The tool contract deduplicates retries automatically (safe by construction).
Serving Cells: The main evaluation uses Alibaba Cloud's qwen-plus (756 pairs). Directional replications and exploratory cells use GLM-5.2, DeepSeek-v4-pro, MiniMax, and SJTU-hosted models.
Evaluation: Scoring is based on the exact final simulator state, not natural language claims. A "state-only" score checks the system state, while a "joint" score checks both state and strict protocol adherence.

Key Results

1. The Information Ceiling and Recovery

Prompt-Only (C1): Achieved a BEOSR of 0.4979, confirming the theoretical 0.5 ceiling. Agents could not distinguish hidden states and failed to achieve "exactly-once" success in both worlds simultaneously.
Status Query (C2): Achieved a BEOSR of 0.8066. While the information barrier was removed, success was not guaranteed. The primary contrast ( $\Delta$ ) between C2/C3 and C1 was 0.4053, exceeding the predeclared threshold of 0.10.
Stable Idempotency (C3): Achieved a BEOSR of 1.0000. By moving the deduplication logic to the tool interface, the system guaranteed safe recovery regardless of the hidden state.

2. Topology-Dependent Failures

While C3 succeeded universally, C2 performance varied significantly by topology:

Create-once: 0.8889 BEOSR.
Reserve-then-link: 0.9877 BEOSR.
Ordered-transition: 0.5432 BEOSR.
- Analysis: In the ordered-transition topology, agents queried the status in every evaluated case (100% utilization) yet achieved only 0.5432 BEOSR. The failure occurred at the decision stage: in 74 out of 81 cases where the status query correctly returned the "old stage," the agent failed to translate this information into the required "advance" action, incorrectly choosing to "stop" instead. This demonstrates that information availability does not guarantee correct information use.

3. State vs. Protocol Correctness

The study distinguishes between achieving the correct system state and adhering to strict output protocols.

In the main cell, state-only success was high, but strict protocol adherence varied.
In some replication cells (e.g., SJTU MiniMax), a positive state-only contrast coexisted with a negative joint (state + protocol) contrast. This indicates that an agent can recover the external state correctly while still violating orchestration contracts (e.g., output formatting), highlighting that safe state recovery and production-compatible output are distinct qualities.

4. Replication

Directional replications across different models (GLM, DeepSeek, MiniMax) and serving environments preserved the positive direction of the state-only contrast (all $\Delta > 0.35$ ), though magnitudes varied.

Significance and Claims

The paper claims to provide a controlled behavioral evaluation that separates information availability, information use, and runtime enforcement as distinct layers of reliability.

Benchmark Validity: The paired counterfactual design solves a validity problem in agent evaluation by ensuring that the "information limit" is exact per pair, preventing model stochasticity from confounding recovery comparisons.
Engineering Implications:
- Prompting is insufficient: Without an information-bearing affordance, recovery is bounded by a 0.5 success ceiling.
- Observability is necessary but not sufficient: Providing status endpoints (C2) improves success but requires agents to correctly interpret and act on the data, a failure point identified in ordered-transition workflows.
- Enforcement is optimal: Moving duplicate suppression to the tool/runtime boundary via stable idempotency keys (C3) guarantees recovery without relying on agent reasoning.
Scope Limitations: The author explicitly states that the study estimates effects under controlled ambiguous outcomes. It does not claim to measure production incidence, internal model mechanisms, hardware effects, or universal guarantees across all software systems. The results are specific to the fixed suite of 81 templates and the tested serving configurations.

In conclusion, the study argues that agent evaluation must move beyond "final answer" correctness to assess the reliability of the process that produced the state, specifically testing whether agents can navigate the gap between ambiguous observations and safe recovery actions.