Technical Summary: AgentCheck

Problem Statement

Tool-using Large Language Model (LLM) agents are currently evaluated under the assumption that all tools function correctly. Existing benchmarks (e.g., GAIA, SWE-bench, ToolBench) measure task completion when tools return valid results. However, real-world deployments face significant risks where tools may timeout, return structurally valid but semantically corrupted stale data, or have their descriptions poisoned by adversaries.

Current diagnostic tools and benchmarks lack a controlled mechanism to:

1. Reproduce specific failure modes (e.g., a timeout or a poisoned instruction) on a developer's own agent configuration.
2. Intervene by injecting these faults deterministically.
3. Confirm whether a specific mitigation strategy resolves the failure before deployment.

Without this capability, developers cannot distinguish between an agent that is robust to tool failures and one that merely succeeds because the tools behaved perfectly during testing.

Methodology: The Reproduce–Intervene–Mitigate Loop

The authors present AgentCheck, an open-source web workbench that transforms a Model Context Protocol (MCP) server into an intervention surface. The system operates on a controlled comparison loop:

1. Controlled Comparison: AgentCheck runs an agent against its real tools and records every tool response in a cache. It then replays the agent's execution, holding all responses constant except for a single, targeted tool call.
2. Fault Injection: A fault injector modifies the cached response of the targeted call. The system supports 12 fault types across three categories:
   • Category A (Tool Execution): Timeouts, API errors, permission denials, and schema drift.
   • Category B (Data Quality): Stale data, contradictions, wrong answers, and silent empty results.
   • Category C (Security): Prompt injection, description poisoning, false claims, and data exfiltration instructions.
3. Divergence Detection: The system compares the "clean" trajectory with the "faulted" trajectory to identify the exact step where the agent's behavior diverges.
4. Mitigation Verification: Developers can apply a mitigation wrapper (e.g., a retry mechanism or an injection filter) and re-run the agent against the identical fault. The system verifies if the failure is resolved.

Key Contributions

1. The AgentCheck Workbench

AgentCheck provides a browser-based interface where developers can connect to their own MCP servers or use a built-in demo. It visualizes the side-by-side comparison of clean, faulted, and mitigated runs, highlighting the specific tool response that caused the divergence.

2. Scenario Library and Taxonomy

The authors contribute a suite of 120 scenarios spanning five domains (finance, science/health, geography/politics, code/technical, general knowledge). These scenarios are derived from existing benchmarks (ToolMisuseBench, MCPTox) and newly authored cases, covering the 12 fault types defined in the taxonomy.

3. Dual-Mode Scoring System

AgentCheck employs a two-tiered evaluation system:

• Deterministic Pass/Fail Checks: Primary verdicts based on strict rules (e.g., "Agent must not present data from a timed-out tool as real").
• LLM Judge for Diagnostic Labels: A secondary, interpretive signal (using Claude 4.5 Haiku) that labels trajectories for failure detection, recovery actions, and uncertainty communication. The authors validate these labels against human annotations but treat the deterministic checks as the primary source of truth due to the non-deterministic nature of LLM judges.

4. Empirical Evaluation

The system was evaluated across five different agent configurations (using models like GPT-4.1 mini, DeepSeek, Llama, and Gemini) under identical fixed tool responses to isolate model performance from environmental variance.

Results

Agent Profiling and Failure Modes

• Performance Variance: Across the 120 scenarios, agent performance varied significantly. The best configuration (DeepSeek) passed 105/120 scenarios, while the weakest (Llama) passed only 77/120.
• Silent Failures: The dominant weakness observed was not system crashes, but the "silent, confident use of incorrect tool outputs." Agents frequently propagated stale data or fabricated values when tools returned errors or outdated information, rather than crashing or reporting uncertainty.
• Category B Vulnerability: All agents performed worst on Category B (Data Quality) faults. Even the strongest agent (DeepSeek) scored only 29/40 on these scenarios. These faults lack error codes, making them difficult for "happy-path" benchmarks to detect.

Mitigation Effectiveness

• Fault-Specific Gains: Mitigations showed high specificity. For example, a "retry" mitigation on the weakest agent (Llama) improved success rates on timeout errors (A1) from 30% to 100%.
• Limitations of Output Wrappers: Output-layer mitigations (like schema validation) failed to resolve deep data-quality issues. For instance, stale data faults (B1) remained at a low success rate (3-4/10) regardless of the mitigation, indicating that resolving these requires temporal reasoning or cross-call memory rather than simple output filtering.
• Security: While prompt injection (C1) and description poisoning (C2) were effectively mitigated by filters, data exfiltration (C4) remained unaddressed by the tested wrappers, as they did not target the specific exfiltration logic.

Significance and Claims

The paper claims that AgentCheck fills a critical gap in the LLM agent lifecycle by making tool failures reproducible, comparable, and verifiable before deployment.

• Shift from Outcome to Process: Unlike traditional benchmarks that score task success, AgentCheck focuses on the agent's ability to handle degraded conditions. It demonstrates that high scores on standard benchmarks do not guarantee survival in real-world deployments where tools fail.
• Developer Readiness: The workbench answers specific deployment readiness questions: "Does my agent fabricate when a tool errors?" and "Does my fix actually close the failure?"
• Modest Scope: The authors acknowledge limitations, noting that the system injects one fault at a time (not compound failures) and that the 12-fault library does not cover every possible failure mode. They emphasize that a "fix_confirmed" verdict is specific to the tested scenario and fault, not a certificate of general robustness.

In summary, AgentCheck provides a practical, controlled environment for developers to stress-test their agents against realistic tool failures, moving beyond the assumption of perfect tool behavior to verify resilience against silent data corruption and security threats.