🤖 AI

FraudBench: Stress-Testing Policy-Grounded Banking Agents Against Adaptive Fraud

This paper introduces FraudBench, a novel executable benchmark designed to stress-test policy-grounded banking agents against adaptive, history-dependent fraud scenarios by simulating dynamic interactions between agents and malicious callers over mutable account states and internal policy documents.

Original authors: Dheeraj Mohandas Pai, Lu Xian

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dheeraj Mohandas Pai, Lu Xian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern financial world, a bank's customer service representative is no longer just a person answering a phone. Increasingly, that representative is a computer program, an artificial intelligence agent, designed to hold a conversation, understand complex requests, and take action. These agents are given access to powerful tools: they can look up a customer's account balance, verify a lost password, freeze a stolen credit card, or even move money from one account to another. They also have access to the bank's internal rulebooks, thousands of pages of documents that dictate exactly what the agent is allowed to do and what it must refuse. The promise of this technology is convenience; a caller can resolve a crisis in minutes without waiting on hold. The risk, however, is that the same powerful tools that help a legitimate customer can be turned against the bank if a caller knows how to manipulate the conversation. If a person can trick the agent into believing they are someone else, or into ignoring a safety rule because of a cleverly worded story, the agent might perform an action that causes real financial loss. The central question for researchers is not whether these agents can understand language, but whether they can understand danger when a human is actively trying to deceive them.

A team of researchers has built a new testing ground to answer this question, a system they call FraudBench. Instead of simply asking an artificial intelligence to read a list of suspicious transactions or to identify a scam email, they created a live simulation where a computer program plays the role of a bank agent, and another computer program plays the role of a fraudster. The two programs engage in a long, multi-turn conversation, much like a real phone call. The fraudster's goal is to manipulate the agent into performing an unsafe action, such as transferring money to a criminal account or revealing private information. The agent's job is to listen, check the bank's internal rules, verify the caller's identity, and decide whether to help or to stop the request. The researchers found that while these agents are good at following instructions, they often fail when the instructions are part of a longer, deceptive story.

The simulation is built on a realistic foundation. The agent operates within a digital bank environment containing 125 fictional customers, their accounts, and their transaction histories. It also has access to a massive library of 698 internal policy documents, which it must search through to find the correct rule for any given situation. The agent has 17 different tools it can use, ranging from simple lookups to high-stakes actions like resetting a personal identification number or unfreezing a card. The fraudster, controlled by a separate program, is not just asking for money; it is trying to exploit the agent's trust. The fraudster might start by asking a harmless question to build rapport, then slowly introduce a false sense of urgency, or admit to a small lie to test the agent's reaction. The most dangerous attacks are those that are adaptive, meaning the fraudster changes their strategy based on how the agent responds. If the agent refuses a request, the fraudster might try a different angle, or if the agent makes a small mistake earlier in the conversation, the fraudster might use that error to justify a larger, unsafe request later on.

To test the agents, the researchers created 150 specific scenarios, each designed to target a different type of fraud. These include attempts to steal an account, tricks to bypass identity checks, and schemes to move money through a series of fake accounts. They also included a set of "chain" attacks, where a series of small, seemingly innocent steps leads to a major security breach. The researchers then ran four different artificial intelligence models through these 150 scenarios to see how well they could protect the bank. The results were mixed. The best-performing model was able to stop the fraudsters in about 65 percent of the cases, while the weakest model succeeded in only about half. This means that in a significant number of simulations, the agent was tricked into performing an action it should have refused. The researchers found that the agents struggled most with two specific types of deception: "money mule" schemes, where the fraudster tries to use the victim's account to move stolen funds, and "chain" attacks, where the danger is hidden in the history of the conversation rather than in the current request.

The study highlights a critical flaw in how these agents are currently evaluated. Many existing tests look only at the final message in a conversation to decide if the agent did the right thing. However, FraudBench showed that safety depends on the entire history of the interaction. An agent might correctly refuse a request for a large transfer, but if it accidentally revealed a secret code earlier in the chat, the conversation is already compromised. The researchers found that agents often failed to remember these earlier mistakes or to connect them to the current request. They also discovered that the agents struggled to find the right rule in the 698-document library when they had to search for it themselves, rather than being given the document directly. When the researchers forced the agents to search for their own rules, the success rate of the best model dropped by 13 percentage points, showing that the ability to find information is just as important as the ability to reason about it.

Perhaps the most important finding is that safety and helpfulness are in tension. An agent that is too eager to help a customer might be tricked into doing something harmful, while an agent that is too cautious might refuse to help a legitimate customer who is just having a bad day. The researchers did not find a single model that was perfect at both stopping fraud and helping real people. Instead, they observed a trade-off: the models that were better at blocking attacks were sometimes more likely to refuse valid requests. This suggests that building a truly safe banking agent is not just a matter of making the software smarter, but of carefully balancing its willingness to act with its ability to detect deception. The researchers emphasize that their work is a simulation using fictional data, so the results do not predict exactly how real banks will perform, but they do provide a clear map of where the current technology is vulnerable. By exposing these weaknesses in a controlled environment, the study offers a path forward for developers to build agents that can navigate the complex, dangerous landscape of modern banking fraud without losing their ability to serve the public.

Technical Summary: FraudBench

Problem Statement

Conversational agents in banking are increasingly granted privileged access to customer databases, internal policy documents, and execution tools (e.g., transferring funds, resetting PINs, changing contact details). While existing benchmarks evaluate agents on static transaction classification, prompt injection, or generic safety, they fail to address a critical gap: whether a policy-grounded banking agent can safely act when a caller manipulates identity, authorization, and trust over a multi-turn conversation.

Current financial fraud datasets (e.g., PaySim, Elliptic) focus on static record classification rather than interactive decision-making. Conversely, agent safety benchmarks often lack the specific workflow controls, large internal policy corpora, and mutable account states required to simulate real-world banking fraud. There is no existing evaluation framework that tests if an agent can prevent unsafe outcomes while maintaining service for legitimate users, specifically when the adversary adapts their strategy based on the agent's previous responses.

Methodology

The FraudBench Framework

FraudBench is an executable benchmark built upon the $\tau^2$ -bench dual-control framework and the $\tau$ -Knowledge banking environment. It simulates a high-stakes interaction between a defender agent and a simulated adaptive caller.

Environment: The system utilizes a mutable banking database containing 125 fictional users, 126 accounts, and transaction histories. It is backed by a corpus of 698 internal policy documents.
Toolset: Both the agent and the caller interact via a shared set of 17 privileged tools. These include read operations (e.g., checking balances) and consequential write operations (e.g., fund transfers, card freezing, dispute filing). The agent controls which tools are discoverable by the caller.
Adversarial Setup: The caller follows a specific scenario policy with a defined goal (e.g., money laundering, account takeover) and pressure tactics. The caller has no direct database access; they can only learn protected information if the agent reveals it.
History-Dependent Grading: Safety is not determined by a single turn but by the entire conversation trajectory ( $\tau$ ). A request that appears valid in isolation may be unsafe if it follows a prior admission, a failed verification attempt, or a "trust contamination" event.

Scenario Construction

The benchmark consists of 150 hand-authored adversarial scenarios:

Frozen Public Set (107 tasks): Used for all reported runs.
- 90 Core Tasks: Distributed across 10 fraud mechanisms (9 tasks each), including Money Mule, Account Takeover, Synthetic Identity, and First-Party Fraud.
- 17 Chain Tasks: Multi-step adaptive attacks where a later, locally valid request becomes unsafe due to earlier probes or admissions.
Held-Out Set (43 tasks): Additional chain tasks reserved for future evaluation.

Each task is annotated with observable evidence, prohibited actions, safe dispositions, and intervention points. The grading process involves a two-stage pipeline:

Deterministic Grading: Checks for specific prohibited tool calls, unsafe state changes, and structured data leaks.
LLM-as-Judge: Evaluates natural-language assertions (e.g., "Was the scam recognized?", "Was PII leaked?") based on the full transcript.

Evaluation Protocol

The study evaluates four agents (Gemini 3.6 Flash, Nemotron-3 Ultra, Gemini 3.1 Flash-Lite, and gpt-oss-120b) under two retrieval conditions:

Oracle Retrieval: The agent is given the exact policy documents required for the task (isolating policy reasoning).
All-Tools Retrieval: The agent must search the 698-document corpus to find governing rules (simulating real-world deployment).

The primary metric is Attack-Security ( $S$ ), defined as the fraction of tasks where the agent prevents unsafe outcomes and meets the safe disposition. A secondary metric, Utility ( $U$ ), measures the ability to complete legitimate requests without over-refusal.

Key Contributions

FraudBench Benchmark: The first executable benchmark for policy-grounded banking agents under adversarial conversation. It integrates privileged tools, a mutable customer database, and a large internal policy corpus, requiring safety decisions to be history-dependent.
Adversarial Dataset: A collection of 150 hand-authored tasks covering ten distinct fraud mechanisms and adaptive chain attacks. The dataset includes detailed annotations for evidence, prohibited actions, and intervention points.
Adversarial Extension of Tool-Agent-User: A novel setting where the caller's private goal conflicts with policy, and the correct outcome may be refusal, escalation, or status quo preservation rather than task completion.
Multi-Model Evaluation: A preliminary evaluation demonstrating that the suite discriminates between models, with security scores ranging from 49% to 65%, and identifies specific weaknesses in handling money mule and adaptive chain attacks.

Results

Preliminary single-trial evaluations on the 107-task set yielded the following Attack-Security ( $S$ ) scores:

Gemini 3.6 Flash: 64.5% (Strongest)
Nemotron-3 Ultra 550B: 57.9%
Gemini 3.1 Flash-Lite: 53.3%
gpt-oss-120b: 49.5% (Weakest)

Key Observations:

Mechanism Difficulty: Money Mule fraud was the most difficult mechanism for all models. Chain (adaptive) attacks also showed a sharp performance gap, with stronger agents defending 8–9 of 17 tasks compared to 4 of 17 for weaker models.
Retrieval Impact: When moving from Oracle retrieval to All-Tools retrieval, Gemini 3.6 Flash's security dropped from 64.5% to 51.0%. This indicates that imperfect self-directed retrieval is a distinct and significant failure source, separate from policy reasoning.
Discriminative Power: The 15-point spread between the best and worst models suggests the benchmark is neither saturated nor trivially failed.

Significance and Claims

The paper positions FraudBench as a necessary evolution in agent safety evaluation. It claims that ordinary customer-service skills are inseparable from authorization, fraud detection, and policy compliance in banking. The benchmark highlights three critical distinctions that aggregate task success often obscures:

Authentication $\neq$ Authorization: A verified customer (or one with stolen static PII) can still make unsafe requests.
Local Validity $\neq$ Global Safety: A request meeting immediate preconditions may be prohibited by earlier conversation history (e.g., a prior lock or admission).
Safety $\neq$ Refusal: A usable agent must still complete legitimate requests; blanket refusal is not a viable defense strategy.

The authors conclude that while stronger general models may improve boundary reasoning, they may also become more compliant with persuasive attackers. Therefore, a security–utility frontier is more informative than a single security score. The paper emphasizes that FraudBench is currently an attack-only release; the full evaluation of over-refusal and utility requires matched legitimate controls, which are planned for future iterations. The release is modest, noting that the current results show discriminative power rather than final rankings, and that the synthetic nature of the bank limits claims about real-world fraud prevalence or regulatory compliance.