Technical Summary: Toward Auditable Fraud Detection

Problem Statement

Financial institutions face a dual challenge in fraud detection: the need to scale with rising transaction volumes while maintaining sufficient transparency, documentation, and human oversight. While agentic AI systems are increasingly proposed as tools to automate evidence gathering and alert review, it remains unclear whether such agents improve decision quality beyond the underlying classifiers they rely on. Furthermore, there is limited evidence on how graph-derived structural features and anomaly signals behave specifically on "difficult" cases (those with intermediate classifier scores) and how disagreement between an agent and a classifier can be leveraged to determine when human review is necessary. This work addresses these gaps by evaluating a layered fraud detection pipeline that integrates gradient-boosted classification, graph features, anomaly signals, and a bounded Large Language Model (LLM) investigation agent.

Methodology

Data and Preprocessing

The study utilizes the PaySim dataset, a synthetic mobile money transaction dataset containing over 6.3 million transactions. A critical preprocessing step involved identifying and removing a simulator-specific balance shortcut. Analysis revealed that the difference between the recorded origin account balance change and the transaction amount was nearly deterministic for fraudulent transactions due to the simulation process. To prevent inflated baseline performance, the origin balance fields were excluded from the feature set, while destination balance fields were retained as they did not exhibit the same artifact.

System Architecture

The proposed pipeline consists of four main stages:

1. Feature Engineering:
   • Tabular Features: Corrected transaction attributes (e.g., amount, type, timing).
   • Graph Features: Derived from a transaction graph where accounts are nodes and transfers are edges. Features include engineered structural statistics (e.g., node degree, distinct counterparties) and node2vec embeddings learned from the subgraph of recurring accounts.
   • Anomaly Signal: A one-class autoencoder trained solely on legitimate transactions. The reconstruction error is used as a single feature to capture deviations from normal behavior.
2. Baseline Detection: An XGBoost gradient-boosted decision tree ensemble serves as the primary classifier. It is trained on the corrected tabular features and is compatible with TreeSHAP for local feature attribution.
3. Agentic Investigation Layer: A bounded LLM agent (Llama 3.3 70B) is activated only for cases where the classifier produces an intermediate score (0.2 to 0.8). The agent accesses four tools:
   • Transaction details.
   • Classifier score and top 5 SHAP attributions.
   • Graph context for origin/destination accounts.
   • Feature-space distances to the nearest labeled fraud and legitimate reference cases.
     The agent must inspect at least two evidence sources and output a structured verdict (Fraud, Not Fraud, or Escalate) with a written rationale.
4. Escalation Policy: An exploratory rule flags cases for human review if the agent disagrees with the classifier and the classifier score is extreme (≥0.65 or ≤0.35).

Experimental Design

• Model Comparison: Four variants were evaluated on a stratified 80/20 train/test split: Hard Baseline (tabular only), Graph-Augmented, Autoencoder-Augmented, and Combined.
• Controlled Ring Experiment: A synthetic experiment introduced 40 multi-account fraud rings (5 accounts, 6 transfers each) to test the ability of graph features to detect coordinated patterns that tabular models might miss.
• Agent Evaluation: The agent was tested on a balanced case-control sample of 60 cases (30 fraud, 30 legitimate) selected from the intermediate-score subset. Performance was compared against a simple thresholding of the classifier score (threshold = 0.5).

Key Results

Detection Performance

• Full Test Set: After correcting for the balance shortcut, the hard tabular baseline achieved the highest Average Precision (0.7677). Neither the graph features nor the autoencoder signal improved performance on the full test set; in fact, the graph-augmented variant showed a slight decrease (0.7611).
• Difficult Subset: Within the subset of cases receiving intermediate scores (0.2–0.8), the autoencoder-augmented model achieved the highest Average Precision (0.0138), outperforming the baseline (0.0062) and the graph-augmented model (0.0086).
• Controlled Ring Experiment: The graph-augmented model successfully detected all 39 injected test transactions from the synthetic fraud rings, whereas the hard baseline detected only 30 (76.9% recall). This indicates that engineered structural features are effective at identifying recurring multi-account patterns, even though they did not improve aggregate performance on the native PaySim fraud cases.

Agentic Investigation Performance

• Accuracy: On the 60-case sample, the agent achieved 65.0% accuracy (39/60 correct), while directly thresholding the classifier score achieved 71.7% accuracy (43/60 correct).
• Disagreement Analysis: The agent disagreed with the classifier in 8 cases. In 6 of these 8 cases, the agent replaced a correct classifier decision with an incorrect verdict. In the remaining 2 cases, it corrected a classifier error.
• Rationale Quality: Despite making incorrect decisions, the agent produced coherent, plausible written rationales in every case, often incorrectly prioritizing the distance to reference cases over the classifier score.
• Escalation Rule: The exploratory disagreement-based rule flagged 2 of the agent's errors for human review without flagging any correct agent decisions in this sample, suggesting disagreement can serve as a signal for escalation, though this finding is limited by the small sample size.

Significance and Claims

The paper concludes that the components of a layered fraud system contribute value only under specific conditions, and that a plausible rationale from an investigation agent is not evidence of a better decision.

1. Conditional Value of Signals: Graph features and anomaly signals do not universally improve detection. Their utility is conditional: autoencoder signals may help rank fraud within uncertain cases, while graph features are critical for detecting coordinated multi-account patterns (fraud rings) that individual transaction attributes miss.
2. Limits of Agentic Investigation: Access to additional context (SHAP values, graph data, reference cases) and the ability to generate structured rationales did not enable the LLM agent to outperform the underlying classifier. In fact, the agent frequently introduced errors while maintaining a high degree of confidence and coherence. This suggests that agents acting as "re-organizers" of existing model outputs may not inherently improve decision quality.
3. Audibility vs. Correctness: The study distinguishes between auditability (the ability to reconstruct the decision process) and correctness. An auditable error remains an error. The value of the agent in this architecture is framed as a bounded investigation and documentation layer that prepares evidence for human review, rather than an autonomous decision-maker.
4. Escalation as a Control Mechanism: Disagreement between the agent and the classifier, combined with strong classifier scores, can serve as a transparent signal to escalate cases to human review, though this requires validation on independent data.

The authors emphasize that these findings are based on synthetic data and a specific experimental setup. They call for future validation on real transaction data with temporal splits and user studies with fraud analysts to determine if such systems improve operational efficiency or review quality in production environments.