🤖 AI

Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responses

This qualitative pilot study demonstrates that agentic AI models exhibit distinct, channel-specific signatures in their log-probabilities when encountering inconsistent versus incomplete tool responses, revealing that robustness to these different fault types is asymmetric and requires tailored detection mechanisms rather than a universal approach.

Original authors: Jiachen Xu, Torben Bach Pedersen, Zhongming Yao, Xiaoyu Zhang, Yushuai Li

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiachen Xu, Torben Bach Pedersen, Zhongming Yao, Xiaoyu Zhang, Yushuai Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern landscape of artificial intelligence, a new kind of digital worker has emerged: the language agent. Unlike earlier programs that simply answered questions based on a static database, these agents are designed to act. They can browse the web, query databases, and use software tools to solve complex problems, much like a human assistant who picks up the phone to make a reservation or checks a spreadsheet for inventory. However, these digital workers are not infallible. When they reach out to a tool, they expect a clean, accurate response. In the real world, tools sometimes fail, returning error messages, or they return data that looks perfect on the surface but contains a subtle lie. The central challenge for researchers is not just building agents that work when everything goes right, but building ones that can recognize when something has gone wrong and respond appropriately. If an agent cannot tell the difference between a broken tool and a misleading one, it will apply the wrong fix, potentially wasting time or making the situation worse.

A team of researchers at Aalborg University and Zhejiang University set out to investigate this exact moment of decision. They wanted to know if an artificial intelligence could instantly distinguish between a tool that has failed to return any data and a tool that has returned a well-formed but false answer. To test this, they created a controlled environment simulating a retail customer-service scenario. In this digital world, an agent acts as a support representative, using a set of sixteen different tools to look up order details, check payment methods, and update user information. The researchers took a standard, successful sequence of actions and, at a specific point, swapped the tool's normal response with a faulty one. They created three types of faults: a complete failure where the tool returned an error message, a falsified order status that changed the rules of what the agent was allowed to do next, and a falsified payment identifier that looked correct but didn't belong to the customer.

The researchers did not simply watch the agent try to finish the task and see if it succeeded. Instead, they paused the process the instant the faulty data arrived. They examined the internal "thought process" of the model, specifically looking at how likely it was to generate the next word in its response. They measured two distinct things. First, they checked how well the returned data fit the expected format of the tool itself. Second, they checked how well that data fit with everything the agent had already seen in the conversation, including the user's original request and previous tool results. They also looked at the agent's plan for the very next step, measuring how confident it was about what action to take and whether that action involved checking the state of the system again or moving forward with a change.

The results revealed a sharp and immediate difference in how the agent reacted to these different types of errors. When the tool returned a broken error message, the agent immediately recognized that the data did not fit the tool's basic format. This was a clear signal that the tool had failed. Consequently, the agent's plan shifted dramatically; it became highly likely to choose an action that would re-read the state of the system, essentially deciding to try again or check the facts. In contrast, when the tool returned a well-formed but false answer, the agent did not see any problem with the format. The data looked perfect. However, the agent's internal comparison of the new data against the history of the conversation revealed a conflict. The false data clashed with what the user had previously stated or what the system policy dictated.

Crucially, the researchers found that the agent's reaction to these false answers depended entirely on the nature of the lie. When the falsified information was a simple mismatch, like a payment ID that didn't belong to the user, the agent's plan remained largely unchanged. It did not become confused or uncertain; it simply accepted the false data and moved on, because the lie did not alter the fundamental rules of the interaction. But when the falsified information was an order status that changed the rules of what actions were permitted, the agent's behavior shifted. It became uncertain about what to do next, and its plan changed to reflect the new, incorrect reality. This showed that the agent could distinguish between a broken tool and a lying tool, but it could not always distinguish between a harmless lie and a consequential one that would derail the task.

The study also looked at what happened after the initial mistake. The researchers found that the type of error left a lasting trace in the agent's future behavior. The broken tool caused the agent to keep trying to re-read the system state in subsequent steps, a sign that it was stuck in a loop of verification. The falsified order status, however, sent the agent down a different path entirely, leading it to take actions that changed the system state based on the false information. The falsified payment ID, which the agent had initially accepted without issue, continued to be accepted in later steps, with no sign of the agent realizing it had been misled. This demonstrated that the agent's ability to recognize an error was not a single, universal alarm. Instead, it was a collection of different signals. One signal told the agent the tool was broken, another told it the data was inconsistent with the past, and a third told it the data was inconsistent with the rules. No single signal could catch every type of error.

The researchers concluded that robustness in these agents is not about having a single metric that flags all bad data. Instead, it requires reading multiple channels of information simultaneously. An agent must be able to see when data fails to match a tool's shape, when it contradicts the history of the conversation, and when it contradicts the rules of the domain. The study showed that these different types of errors produce distinct signatures in the agent's internal calculations. A broken tool is immediately obvious because it breaks the format. A lying tool is harder to catch because it fits the format but breaks the story. The most dangerous lies are those that change the rules of the game, as they can send the agent down a completely wrong path without triggering any confusion. By understanding these specific signatures, developers can build agents that are better equipped to diagnose their own mistakes and choose the correct remedy, whether that means retrying a tool, cross-checking a fact, or stopping to ask for clarification.

Technical Summary: Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responses

Problem Statement
Robustness in tool-using language agents is defined not by the avoidance of bad tool returns, but by the ability to respond appropriately to the specific nature of the failure. The paper posits that a failed tool call (incomplete return) and a well-formed but factually incorrect return (inconsistent return) constitute distinct problems requiring different remedies (e.g., retrying vs. cross-checking). Current diagnostic metrics, such as step-wise reasoning entropy, often fail to distinguish between these fault families at the moment a return arrives, merely indicating that "something has gone wrong" without specifying what. The core research question is whether an agent can qualitatively distinguish between incomplete and inconsistent returns immediately upon their arrival, and if so, through which observable channels in the model's internal state.

Methodology
The authors conduct a qualitative pilot study within the retail domain of $\tau^2$ -bench, utilizing a ReAct-style loop with 16 tools. The experimental design employs "teacher forcing" to score decision points rather than running agents to completion, thereby isolating the immediate reaction to a tool return.

Fault Injection: The study injects controlled faults into the first call to get_order_details (position 2 or later) in a trajectory. Four conditions are established:
1. Clean (C): The true return.
2. Failed (F): An incomplete return (error string) replacing the payload.
3. Inconsistent Status ( $N_s$ ): A schema-valid payload where the order status is falsified to a legal value that contradicts the domain policy or user intent.
4. Inconsistent Identifier ( $N_m$ ): A schema-valid payload where a payment method identifier is falsified to a well-formed value not belonging to the user.
Measurement Channels: Two primary channels are read from the model's log-probabilities at the decision point following the injection:
1. Likelihood Gap ( $G$ ): The difference between the surprisal of the returned string under the whole trajectory context ( $S_1$ ) and under the tool schema alone ( $S_0$ ). $G = S_1 - S_0$ . This measures how much the trajectory context adds to the predictability of the return.
2. Action Distribution: The probability distribution over the 17 legal actions (16 tools + reply). Metrics include the entropy of this distribution ( $H$ ), the leading action ( $a^\star$ ), and the mass assigned to specific action groups, particularly tools that re-read state ( $\pi(V)$ ).
Control: Every measurement is paired against a clean baseline for the same task, ensuring that differences ( $\Delta X$ ) are attributable solely to the injected fault.

Key Results
The study reveals an asymmetry in how different fault types manifest across the measurement channels:

Likelihood Channel ( $G$ ):
- Incomplete Returns (F): These are immediately legible. They are improbable under the tool schema alone ( $S_0$ ), causing a sharp positive shift in $\Delta S_0$ . This places them in a region of the feature space no other condition enters.
- Inconsistent Returns ( $N_s, N_m$ ): These leave the schema channel ( $S_0$ ) untouched. They only register in the gap ( $G$ ) if the injected value collides with a token already present verbatim in the prefix (e.g., $N_m$ where the true ID was explicitly stated). If the contradiction requires reasoning through domain policy (e.g., $N_s$ where the user described the state in prose), the gap remains flat.
Action Distribution Channel:
- Incomplete Returns (F): These consistently shift the agent's probability mass toward tools that re-read state ( $\pi(V)$ increases), effectively signaling a need to retry or verify. The distribution often sharpens and redirects.
- Inconsistent Returns: These do not exhibit a unified direction. $N_s$ (falsified status) creates uncertainty (increasing entropy) and shifts the leading action toward state-changing calls, whereas $N_m$ (falsified identifier) leaves the distribution largely unchanged.
Long-term Trace: When the same injected return is carried through subsequent calls (where the payload bytes are identical across arms but the context differs), the asymmetry persists. The likelihood gap ( $G$ ) remains elevated only for the failed return (F), while the inconsistent returns ( $N_s, N_m$ ) show no such signal. However, the leading action for the falsified status ( $N_s$ ) diverges from the clean baseline toward state-changing tools, whereas the failed call (F) continues to drive the agent toward re-reading state.

Key Contributions

Qualitative Distinction of Fault Families: The paper demonstrates that incomplete and inconsistent returns are legible at the moment of arrival but require different diagnostic channels. No single quantity captures all fault types.
Channel-Specific Signatures: It establishes that schema-level likelihood ( $S_0$ ) is the primary detector for incompleteness, while trajectory-level likelihood ( $S_1$ ) and action distribution shifts are necessary to detect specific types of inconsistencies.
Action Distribution as a Policy Proxy: The study shows that the action distribution orders faults by their impact on the next action (e.g., whether the error necessitates re-reading state or changing state) rather than by the fault family itself.
Methodological Framework: It proposes a paired, teacher-forced scoring methodology that isolates the immediate model response from the compounding effects of agent drift, allowing for precise attribution of signals to the injected context.

Significance and Claims
The paper claims that robustness relies on the agent's ability to distinguish fault types immediately, but that current single-curve metrics (like overall entropy) are insufficient for this. The findings suggest that "recognition is asymmetric": each condition is legible in some channel, but no single channel is legible for all conditions.

The authors modestly frame this as a pilot study. They acknowledge limitations, including the use of a single domain, a single model, and template-based (non-adversarial) noise. They do not claim to have solved the problem of robustness but rather to have mapped the "blind spots" of current diagnostic approaches. The study concludes that separating the "merely contradictory" from the "consequential" requires a quantity that jointly considers the observation and the policy, a gap their current channels do not fully fill. The work serves as a foundational step in understanding how agents process noisy tool feedback, suggesting that future robustness mechanisms must monitor multiple, distinct channels simultaneously to apply the correct remedy.

Technical Summary: Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responses

More like this