PolicyFaultBench: Mutation-Based Assurance of Policy Mediation and Proposal-Interface Conformance for Tool- Using AI Agents
PolicyFaultBench is a mutation-based benchmark that validates tool-using AI agents by rigorously testing runtime policy mediation and proposal-interface conformance, revealing that while agents can achieve high execution success, they may still fail strict acceptance criteria due to subtle interface deviations that require targeted probes to detect.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where your favorite AI assistant is like a super-smart, eager intern. It can write code, send emails, and even transfer money if you ask it to. But here's the catch: this intern is incredibly creative and sometimes a bit too chatty. If you tell it, "Send an email to my boss," it might add a funny joke, a weird emoji, or a long story about its weekend before actually sending the email. In the high-stakes world of computer security, that extra chatter isn't just annoying; it's dangerous. If the system checking the intern's work doesn't look at the exact message, it might miss a hidden trap or accidentally approve a dangerous action. This is the problem of "policy mediation": making sure that when an AI suggests an action, a strict security guard checks it exactly as written before letting it happen. The big question isn't just "Is the AI smart?" but "Is the security guard doing its job perfectly, even when the AI gets a little weird?"
This paper, titled PolicyFaultBench, is like a rigorous inspection of that security guard. The researchers, Hasan Fadhil Qasim and Sarah Abdulzahra Kadim, didn't just ask the AI to do random things; they set up a controlled game where the AI had to act as a "transcriber." Instead of planning a complex mission, the AI was given a specific, frozen task (like "transfer $10 to Bob") and asked to write it down in a very specific format. The researchers then tested the security guard (the "mediator") to see if it would catch mistakes, weird formatting, or hidden tricks. They used a clever trick called "mutation testing," which is like intentionally breaking the security guard's rulebook in small ways to see if the guard notices. For example, they might pretend the guard forgot to check if the money transfer was approved, or they might pretend the guard allowed a transfer to a different person.
The study found that while the security guards (from companies like OpenAI and Anthropic) are generally good at stopping bad actions, they aren't perfect. In a test of 400 trials with one provider, the guard caught every single bad attempt. However, when they tested another provider, 9 out of 400 times, the AI added a tiny, extra piece of text to its message (like adding a "scope" field that wasn't asked for). The security guard, following a strict rule that says "the message must match exactly," rejected these 9 messages. This is a "fail-closed" design, meaning it's better to reject a safe message because of a tiny typo than to accidentally accept a dangerous one. The researchers showed that while the AI's intent was safe in those 9 cases, the strict check was necessary to ensure the system didn't get confused by extra text.
The paper also discovered that the security guard's rulebook had some hidden holes. When they intentionally broke the rules (the "mutation" part), the guard missed 5 out of 12 types of tricks in the first round. It only caught the rest after the researchers added special "probe" tests designed specifically to find those missing holes. This suggests that just because a system works today doesn't mean it's safe against every possible trick tomorrow. The researchers also tried to see if these results would hold up with a different set of tasks (derived from a benchmark called AgentDojo) and a different security system. The results were mixed: the AI passed 391 out of 400 times with the new tasks, failing only because of that same "extra text" issue.
In short, the paper suggests that we cannot rely on AI models to be perfect on their own. We need a separate, strict security layer that checks the AI's proposals with extreme precision. If the AI adds even one extra word that wasn't requested, the system should stop and ask for clarification, rather than guessing. The study proves that while current systems are strong, they need constant, rigorous testing to ensure they don't miss subtle errors. The researchers conclude that safety isn't a single "pass" or "fail" grade; it's a collection of different checks—checking the message format, checking the rules, and checking the final result—all working together to keep the digital world safe.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.