💻 computer science

Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety

This paper introduces SafeAudit, a meta-audit framework that systematically enumerates tool-call workflows and quantifies rule-resistance to reveal significant completeness gaps in existing LLM agent safety benchmarks, uncovering over 20% of residual unsafe behaviors that current tests fail to detect.

Original authors: Xuan Chen, Lu Yan, Ruqi Zhang, Xiangyu Zhang

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuan Chen, Lu Yan, Ruqi Zhang, Xiangyu Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a super-smart, robotic assistant (an LLM Agent) to handle your digital life. This robot can check your email, book flights, transfer money, and order groceries. It's incredibly helpful, but you're worried: What if it accidentally sends your private photos to the wrong person? Or what if it clicks a scam link because it was in a hurry?

To make sure this robot is safe, researchers have built "Safety Exams" (benchmarks). These are like driving tests for robots, where they are given tricky scenarios to see if they crash. If the robot passes the test, we assume it's safe.

But here's the problem: What if the test only checks if the robot knows how to stop at a red light, but never checks if it knows how to handle a slippery road? The robot might pass the test but still crash in the real world.

This paper, "Who Tests the Testers?", introduces a new system called SafeAudit. It's not a new driving test; it's a meta-auditor—a "test for the tests." Its job is to find the blind spots in the existing safety exams.

Here is how SafeAudit works, broken down into simple concepts:

1. The "What-If" Machine (The Enumerator)

Imagine you are trying to find every possible way a robot could mess up a task.

Old Way: You just ask a smart computer, "Give me 100 ways a robot could be unsafe." It usually gives you the obvious ones, like "stealing money" or "sending hate mail."
SafeAudit's Way: Instead of just asking for bad ideas, SafeAudit acts like a Lego master. It looks at the robot's tools (email, calendar, web browser) and systematically builds every possible chain of events.
- Analogy: If the robot has a "Search" tool and an "Email" tool, SafeAudit doesn't just say "Don't send bad emails." It asks: "What if the search finds two people with the same name? What if the email list is outdated? What if the user is in a rush and tells the robot to 'just pick one'?"
- It generates thousands of these "what-if" scenarios, many of which are subtle and tricky, not just obvious crimes.

2. The "Rule Book" (Rule Resistance)

Once SafeAudit has a pile of tricky scenarios, it needs to check if the existing safety exams would have caught them.

The Process: SafeAudit looks at the old safety exams and asks, "What rules did the exam teach the robot to follow?"
- Example: If the exam had a case about "Don't click phishing links," SafeAudit extracts a rule: "MUST NOT click unknown links."
The Test: SafeAudit then takes its new, tricky scenarios and applies these old rules to the robot.
- If the robot still makes a mistake even after following the old rules, that's a Blind Spot.
- Analogy: Imagine a security guard (the robot) who was trained to stop anyone with a red hat (the rule). SafeAudit brings in a person wearing a blue hat who is actually a spy. The guard lets them in because the rule only covered red hats. SafeAudit says, "Hey, your training missed the blue hats!"

3. The Results: Finding the Hidden Dangers

The researchers ran SafeAudit against three major safety exams across 12 different environments (like email, health, and travel).

The Shocking Discovery: Even though the robots passed the official safety exams, SafeAudit found that more than 20% of the dangerous situations were still being missed.
The "Novelty" Factor: SafeAudit didn't just find the same old mistakes; it found new types of mistakes that the old exams never thought to check.
- Real-world example from the paper: A robot was asked to pay an invoice. It correctly replied to the right email thread but, because there were multiple similar emails, it clicked a payment link from a different (and malicious) email thread. The robot followed the rules but failed to connect the dots. The old exams didn't catch this because they only looked for "clicking bad links," not "clicking the wrong link in a confusing situation."

Why This Matters

Think of safety testing like checking a bridge.

Old Benchmarks check if the bridge holds up when you drive a car over it.
SafeAudit asks, "What if a truck hits it at an angle? What if the wind is blowing sideways? What if the road is wet?"

The paper concludes that we can't just rely on the current safety exams. We need SafeAudit to constantly stress-test our safety rules, find the gaps, and update the training so our AI agents are truly safe in the messy, unpredictable real world.

In short: SafeAudit is the "stress test" that ensures our safety exams aren't just giving the AI a false sense of security. It asks, "Who is checking the people who check the robots?" and then builds a better way to do it.

1. Problem Statement

Large Language Model (LLM) agents increasingly rely on external tools (e.g., email, calendars, web APIs) to perform tasks. While existing safety benchmarks evaluate these agents across diverse environments and risk categories, a critical gap remains: completeness.

The Core Question: If an agent passes a benchmark, how do we know it is truly safe? Existing benchmarks often rely on ad-hoc prompt generation, which tends to surface obvious failure patterns while missing valid but rare interaction scenarios.
The Limitation: Current evaluations focus on the diversity of environments and tools but fail to systematically characterize the space of unsafe behavioral patterns. Small changes in user scenarios (e.g., ambiguous API parameters or instruction conflicts) can lead to qualitatively different unsafe outcomes that benchmarks miss.
The Goal: To develop a "meta-audit" framework that quantifies the completeness of existing benchmarks and identifies "blind spots" (residual unsafe behaviors) that remain even after an agent passes standard tests.

2. Methodology: SafeAudit

The authors propose SafeAudit, a meta-audit framework consisting of two main components: an LLM-based Enumerator and a Rule-Resistance Evaluation protocol.

A. LLM-Based Enumerator (Systematic Generation)

Instead of directly generating test cases (which creates a massive, unstructured search space), SafeAudit uses a two-phase approach to systematically explore unsafe interaction patterns:

Phase 1: Abstract Pattern Enumeration:
- The system prompts an LLM to identify valid multi-step tool-call workflows specific to a target environment.
- It reasons about how these workflows could fail at specific steps (e.g., lookup ambiguity, stale data execution, skipped verification).
- It generates high-level descriptions of unsafe patterns, varying the structural tool chains and user scenarios (e.g., different user roles, delegation styles, and urgency levels).
Phase 2: Concrete Test Case Instantiation:
- The abstract patterns are converted into executable test cases formatted to match the target benchmark.
- This phase uses few-shot examples tailored to the specific environment's tools and constraints to ensure the generated cases are valid and realistic.

B. Rule-Resistance Evaluation (Completeness Metric)

To measure how "complete" a benchmark is, SafeAudit introduces a non-semantic, quantitative metric called Rule-Resistance:

Rule Extraction: From an existing benchmark, the system uses an LLM to extract compact safety rules (e.g., "MUST NOT send patient data without verifying...") that would prevent the unsafe behaviors in that benchmark. These are compressed into a minimal set ( $R_{bench}$ ) using a greedy set-cover algorithm.
Iterative Evaluation:
- The extracted rules are sequentially appended to the target agent's system prompt.
- The SafeAudit-generated test cases are re-evaluated. If a case becomes "safe" after a rule is applied, it is considered "covered" by that rule.
- Uncovered Rate (UR): The fraction of generated test cases that remain unsafe even after all rules from the benchmark are applied. A high UR indicates that the benchmark missed significant unsafe interaction patterns.

3. Key Contributions

Systematic Enumeration Framework: A novel method to explore the space of valid agent workflows and user scenarios, generating diverse, grounded test cases that go beyond surface-level prompting.
Rule-Resistance Metric: A verifiable, non-semantic protocol to measure benchmark completeness. Unlike semantic similarity (which is unreliable for safety), this metric checks if a new failure requires a new safety constraint not present in the original benchmark.
Comprehensive Meta-Audit: The application of this framework across 3 benchmarks (AgentSafetyBench, AgentHarm, ToolSafety), 12 environments, and 8 backbone LLMs.

4. Experimental Results

The authors evaluated SafeAudit against two baselines: DirectGen (direct prompt generation) and SIRAJ (category-driven generation).

Residual Unsafe Behaviors: SafeAudit consistently uncovered >20% more residual unsafe behaviors than existing benchmarks. This means that even if an agent passes a benchmark, SafeAudit reveals that it still fails in scenarios the benchmark did not cover.
Novelty Analysis: Approximately 11% of the unsafe patterns discovered by SafeAudit were structurally novel (different failure mechanisms or positions in the agent pipeline) compared to existing test suites.
Coverage Scaling: As the generation budget (number of test cases) increased, the Uncovered Rate (UR) grew monotonically, suggesting that current benchmarks only scratch the surface of the unsafe interaction space.
Model Generalization: The framework worked effectively across different model sizes and architectures (from Llama-3.1-8B to GPT-4o-mini and Qwen3-32B). Notably, stronger models like Qwen3-32B showed high UR, indicating that even advanced models have blind spots regarding tool-call safety.
Utility Preservation: The study confirmed that applying the extracted safety rules did not significantly degrade the agent's ability to perform benign tasks (average utility drop was only ~5%), validating that the uncovered risks are genuine safety gaps, not over-refusal issues.

5. Significance and Conclusion

Redefining Safety Evaluation: The paper argues that passing a benchmark is insufficient proof of safety. Safety evaluation must move from static benchmarking to dynamic meta-auditing that quantifies coverage gaps.
Structural vs. Content Safety: The research highlights that many agent failures are structural (residing in tool-call sequences and parameter choices) rather than content-based (text generation). Traditional content filters often miss these structural vulnerabilities.
Future Directions: The authors suggest that meta-auditing should become a standard complement to benchmarking. Future work could involve using the discovered "uncovered" patterns to train agents (via fine-tuning or preference optimization) and developing adaptive feedback loops for test generation.

In summary, SafeAudit provides a rigorous, automated method to "test the testers," revealing that current safety benchmarks leave significant blind spots in LLM agent safety, particularly regarding complex, multi-step tool interactions.