💬 NLP

Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback

This paper introduces the concept of "cognitive poisoning" in LLM agents, where malicious tools build trust through benign feedback before executing harmful actions, and proposes the TRUST-Bench benchmark and VISTA-Guard defense framework to effectively detect and mitigate these trajectory-based risks by scoring the final executable action rather than relying on prompt-level heuristics.

Original authors: Lecheng Yan, Ruizhe Li, Xicheng Han, Wenxi Li, Binwu Wang, Longyue Wang, Chenyang Lyu, Guanhua Chen

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lecheng Yan, Ruizhe Li, Xicheng Han, Wenxi Li, Binwu Wang, Longyue Wang, Chenyang Lyu, Guanhua Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Wolf in Sheep's Clothing" Tool

Imagine you hire a personal assistant (the AI Agent) to do your shopping. You tell them, "Go to the store and buy milk."

Usually, we assume that once the assistant picks a store (a Tool), the information they get back from that store is honest. If the store says, "We have milk," we trust it.

This paper argues that this assumption is dangerous.

The authors describe a new type of trick called "Cognitive Poisoning." It's like a wolf that dresses up as a sheep.

The Trick: A malicious tool (the wolf) acts perfectly normal at first. It answers your assistant's questions politely and correctly for several rounds of conversation. It builds up trust.
The Trap: The tool only reveals its true, harmful nature at the very last second. It waits until the assistant is about to make a final, big decision (like "Buy the milk and transfer $1,000 to this stranger"). Only when specific hidden conditions align does the tool flip a switch and say, "Actually, let's do the dangerous thing."

The scary part? If you look at any single message the tool sent, it looks innocent. The danger isn't in one bad sentence; it's in the story the tool told over time, which tricked the assistant into lowering its guard.

The Problem: Old Defenses Don't Work

The researchers tested how well current safety systems handle this. They found that most existing defenses are like security guards who only check ID cards at the door.

They look at the tool's name or the first few words of a message.
If the tool looks friendly and the first few messages are safe, the guard lets it through.
Result: These guards fail completely against "Cognitive Poisoning" because the tool was friendly for most of the interaction. The guards missed the subtle shift in the story.

The Solution: TRUST-BENCH and VISTA-GUARD

To study this, the team built two things:

1. TRUST-BENCH (The Training Ground)

They created a giant test bank called TRUST-BENCH.

The Setup: They took thousands of normal tasks (like searching the web or editing files) and created "evil twins" for them.
The Match: For every safe task, they made a malicious version where the tool acts nice for three rounds, then tries to trick the AI into doing something harmful on the fourth round.
The Goal: To see if an AI can tell the difference between a tool that is actually safe and one that is pretending to be safe until the last moment.

2. VISTA-GUARD (The Smart Detective)

They built a new defense system called VISTA-GUARD. Instead of just checking the ID card, this system acts like a detective reviewing a case file.

How it works: It doesn't just look at the final request. It looks at the entire history of the conversation.
The Analogy: Imagine a detective asking: "The suspect said they were innocent for three days, but on day four, they suddenly asked for a key to the vault. Does the whole story make sense, or does the sudden change in behavior look suspicious?"
The Secret Sauce: VISTA-GUARD converts the conversation into a structured report (like a scorecard of "trust signals") and then asks the AI: "Given this whole story, is the final action safe?"

The Results: Who Won?

The researchers tested VISTA-GUARD against many other methods (including other AIs and simple rule-based systems).

The Losers:
- Simple Rules: Systems that just look for "bad words" failed.
- Zero-Shot Judges: Even very smart, pre-trained AI models (like GPT-5.4) failed when asked to judge this without specific training. They were too "paranoid" (rejecting everything) or too trusting.
- Scalar Classifiers: Systems that just turned the conversation into a single number (like a "risk score") failed because they lost the details of the story.
The Winner:
- VISTA-GUARD was the only one that consistently succeeded.
- It learned to spot the "wolf" by analyzing how trust was built over time.
- The Score: It achieved a high safety score (84.2) in its own training environment and still performed well (56.9) when tested on completely new, unseen tools.

The Key Takeaway

The paper concludes that trust is a process, not a snapshot.

When an AI uses tools, it shouldn't just ask, "Is this tool's name safe?" or "Is this message safe?" It needs to ask, "Does the entire history of our interaction make sense, and does the final action fit the story we've been told?"

If the tool acted like a friend for three days and then suddenly tried to rob the bank on day four, the AI needs to realize that the story is the danger, not just the final sentence.

Summary in One Sentence

This paper shows that malicious tools can trick AI agents by acting nice for a while before striking, and the only way to stop them is to use a defense system that analyzes the entire history of the interaction, not just the final request.

Technical Summary: Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback

1. Problem Formulation: Cognitive Poisoning

This paper identifies a distinct failure mode in tool-using Large Language Model (LLM) agents termed cognitive poisoning. Unlike traditional indirect prompt injection or single-response poisoning, where maliciousness is localized to a specific text passage or tool descriptor, cognitive poisoning operates through the exploratory trust-formation process.

In this threat model, a malicious tool behaves plausibly during initial exploration steps, accumulating trust through benign-looking feedback. The tool only becomes harmful when a hidden trigger condition is met by the final executable action and the latent state of the environment. The attack succeeds not because a single message is overtly malicious, but because the interaction trajectory shapes the agent's prior to treat a dangerous final action as ordinary. The decisive risk is a property of state-action composition: the history of how trust was formed combined with the specific parameters of the final tool call.

Existing benchmarks and defenses often implicitly assume that once a tool is selected, its feedback is trustworthy. This paper argues that in black-box tool ecosystems, agents cannot directly verify internal trustworthiness and must infer it through interaction, creating a unique attack surface where the "decisive defense target" is not local prompt text, but the trajectory-conditioned risk of the final action.

2. Methodology

2.1 TRUST-BENCH: A Task-Conditioned Benchmark

To study this regime, the authors constructed TRUST-BENCH (Trajectory Risk under Untrusted Stateful Tools).

Construction: Derived from five upstream suites (OpenAgentSafety, RedTeamCUA, MCP-SafetyBench, MCPMark, Toolathlon), resulting in 1,970 curated episodes.
Structure: Each episode consists of a matched pair: a benign control and a malicious variant. Both share the same outward tool interface and task intent.
Hidden Triggers: Malicious variants are defined by a structured trigger specification comprising:
- required_all_tokens: Fields that must jointly appear (e.g., op=export, cross_check=off).
- required_any_tokens: Contextual anchors where at least one must appear.
- forbidden_tokens: Safety-enabling fields that, if present, suppress the trigger.
Interaction Protocol: A "blue team" agent probes the tool over a standardized three-round budget (collect baseline, check consistency, deep probe) before proposing a final action. The trigger rules and labels are hidden from the defender during inference.

2.2 Evaluation Metric: GUARDEDJOINT

The paper introduces GUARDEDJOINT, an asymmetric safety–utility metric designed to reflect real deployment risks where both false negatives (accepting malicious actions) and false positives (rejecting benign actions) are costly.

Formula: $G_\rho(AMR, RNR) = \max(0, 100 - (\rho + 0.5)AMR - 1.5RNR)$ $G_{ρ} (A M R, R N R) = max (0, 100 - (ρ + 0.5) A M R - 1.5 R N R)$
- $AMR$: Accepted Malicious Rate.
- $RNR$: Rejected Normal Rate.
- $\rho$ : Relative cost of accepting a malicious call (set to 1.5 in main experiments).
Significance: This metric penalizes "reject-all" or "execute-all" strategies, forcing defenses to simultaneously minimize malicious acceptance and benign rejection.

2.3 Defense Framework: VISTA-GUARD

The authors propose VISTA-GUARD (Variable-state Inference for Safe Tool Actions), a backbone-agnostic framework for final-action risk scoring.

Core Idea: Abstract multi-step tool interactions into structured environment variables that encode trust-formation dynamics.
Input Representation: The model receives a serialized triplet:
1. Trajectory State ( $z_{traj}$ ): Summarizes exploration evidence (e.g., counts of triggered phases, probe detections, mismatch scores, ID drift, warnings) into an anomaly density and state-risk summary.
2. Final Action Parameters ( $z_{param}$ ): Encodes the proposed final call's executable semantics (tool name, payload keys, counts of high-impact/bypass/safety markers).
3. Target ( $y$ ): A binary label ( $high\_risk$ / $low\_risk$ ) used only for training.
Mechanism: A backbone LLM (e.g., Mistral, Qwen, Llama) is fine-tuned to score the risk of the final action conditioned on the trajectory evidence. It learns to reject actions where the final parameter bundle is incompatible with the inferred state of the tool's behavior.

3. Key Results

3.1 In-Domain Performance

Under the GUARDEDJOINT metric ( $\rho=1.5$ ), VISTA-GUARD significantly outperforms existing baselines:

VISTA-GUARD (Mistral-7B): Achieved 84.2 in-domain score.
Other Backbones: All six LLM backbones tested (Qwen3, Qwen2.5, OLMo-2, Llama-3.1, Meta CWM) achieved positive scores (ranging from 39.0 to 67.6), demonstrating robustness across architectures.
Baseline Collapse: Methods optimizing only one side of the trade-off or relying on prompt-centric heuristics collapsed to 0.0:
- Prompt-centric judges (GPT-5.4 zero-shot) and heuristic safeguards (ToolShield, GuardAgent) failed to distinguish subtle trust dynamics, often defaulting to accepting all malicious actions or rejecting all benign ones.
- Scalarized classifiers (Gradient Boosting, SVM) and non-LLM text models (BERT) failed to capture the temporal context, resulting in scores near zero.

3.2 Out-of-Distribution (OOD) Transfer

VISTA-GUARD maintained effectiveness on balanced OOD data (unseen tool families from ToolEmu and SafeToolBench), achieving 56.9 GUARDEDJOINT.
This indicates that the structured trajectory representation transfers better than prompt-centric or scalarized approaches, which dropped significantly or collapsed.

3.3 Ablation Studies

Representation vs. Backbone: An isolation study showed that the gain comes primarily from the structured trajectory representation rather than the backbone choice alone. Providing non-LLM models with the same structured input improved their performance but still left a gap compared to LLM backbones.
Parameter-Only Failure: Models trained only on final action parameters (without trajectory context) collapsed, confirming that the trajectory history is essential for distinguishing benign controls from malicious final actions.

4. Significance and Claims

The paper claims that the decisive contribution of this work is representational: the defense problem becomes tractable when tool interaction is converted into structured environment variables rather than treated as raw text or isolated parameters.

Shift in Focus: The authors argue that for state-conditioned cognitive poisoning, the relevant defense target is final-action risk scoring based on how trust is formed across the trajectory.
Limitations: The authors are modest about their claims, noting that TRUST-BENCH is a constructed benchmark with standardized three-step exploration and a binary execute/reject action space. They acknowledge that real-world deployments may involve longer interactions, adaptive attackers, and more complex latent states.
Conclusion: The findings motivate a broader evaluation question: when tool feedback itself may be untrusted, security defenses must reason about the trajectory-conditioned trust formation rather than relying solely on local prompt inspection or tool descriptors.

The paper concludes that while scalar summaries discard too much trajectory information and raw text models recover some signal, sequence backbones benefit most when the interaction history is organized as trajectory-state and final-action evidence, providing a principled framework for secure tool generalization.