📊 statistics

PolicyShift-AML: Auditing Selective-Label Policy Dependence in Anti-Money-Laundering Transaction Monitoring

The paper introduces PolicyShift-AML, a controlled auditing protocol that reveals how Anti-Money-Laundering models' offline performance often reflects institutional scrutiny policies rather than true event status, while proposing a non-negative positive-unlabeled estimator that significantly improves latent recall under policy-dependent label selection compared to standard baselines.

Original authors: Muhammetalp Erdem

Published 2026-09-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammetalp Erdem

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes world of banking, a silent war plays out every second against money laundering. Banks use automated systems to scan millions of transactions, looking for patterns that suggest illegal activity. However, these systems face a fundamental blind spot: they rarely see the full picture. They only see the transactions that an alert system flags for human review. The vast majority of transactions, even those that might be suspicious, pass through unseen. This creates a distorted view of reality, where a computer model learns to recognize the bank's own rules for investigation rather than the actual behavior of criminals. If the bank changes its rules for what gets flagged, the model's performance can crash, not because the criminals changed, but because the model was trained on a specific, narrow slice of data. This problem, known as selective labeling, means that a system appearing perfect in a test might fail completely in the real world because it has memorized the investigator's habits instead of the crime itself.

A researcher at Trabzon University has developed a new way to test these systems, stripping away the confusion to see how much they truly rely on the specific rules used to train them. The study, titled PolicyShift-AML, does not look at real bank data, which is often messy and legally restricted. Instead, it uses a massive, carefully constructed simulation of 3 million transactions. In this digital laboratory, the researchers created a fixed stream of "latent" events, representing the true state of whether a transaction was actually suspicious or not. They then ran five different investigation policies over this same stream. Each policy acted like a different bank with different rules for deciding which transactions to investigate, yet all five policies were designed to review the same total number of transactions. By keeping the underlying truth constant while changing only the rules for what gets labeled as suspicious, the team could isolate exactly how much a model's success depended on the specific policy it was trained on, rather than on the actual ability to detect crime.

The results revealed a significant vulnerability in how these models are currently evaluated. When the researchers trained models using data from one specific investigation policy and then tested them on the fixed stream of true events, the performance varied wildly depending on which policy was used for training. For one common type of model, the ability to find the top 500 suspicious transactions ranged from a low of about 26% to a high of nearly 38% simply by changing the training policy. This 26% to 29% spread in performance demonstrates that the model was not learning a universal truth about money laundering; it was learning the specific quirks of the investigation method it was fed. Even more telling was the discovery that standard evaluation methods are dangerously optimistic. When a model is tested using only the labels it was trained on, it appears to perform much better than it actually does against the true ground truth. In one scenario, a standard model showed an apparent success rate that was more than 10 percentage points higher than its actual ability to find real threats, creating a false sense of security for the institutions relying on it.

To address this, the study introduced a more robust method for training models that can handle these missing pieces of information. This approach, known as positive-unlabeled learning, treats the uninvestigated transactions not as harmless, but as unknowns that need to be weighed carefully against the known suspicious ones. The researchers built a new estimator that accounts for the likelihood of a transaction being selected for review, correcting for the bias introduced by the investigation policy. In simulations with a realistic thirty-day delay between a transaction and the confirmation of whether it was truly suspicious, this new method achieved a success rate of about 40% in finding the top 500 threats. This was a substantial improvement over the standard models, which hovered around 23% under the same conditions. The new method proved so accurate in the simulation that it nearly matched the performance of an "oracle" system—a theoretical perfect model that knows the true probabilities in advance—suggesting that with the right mathematical corrections, models can learn to see past the limitations of their training data.

The study also tested whether adding a history of past interactions between accounts, represented as a graph of connections, would make the models more stable. While these graph-based features were essential for the new method to work well, they did not make the models immune to the training policy. The models still showed sensitivity to how they were trained, debunking the idea that complex network data alone could solve the problem of policy dependence. Furthermore, the researchers tested a scenario where a specific type of criminal behavior was completely hidden from the training data. Even with advanced techniques, the models could not reliably identify these hidden patterns without any examples to learn from, highlighting that no amount of statistical correction can identify a threat that has zero presence in the training history. The findings suggest that while better statistical tools can significantly improve detection, they cannot replace the need for diverse and representative data.

Ultimately, this research serves as a critical stress test for the financial industry's defenses. It shows that a model's performance is not a fixed number but a variable that shifts with the rules used to generate its training data. The study argues that banks and regulators must move beyond simple accuracy scores and instead audit how their models behave under different investigation policies. By understanding that a system might be excellent at finding the specific type of fraud its current rules catch, but poor at finding others, institutions can avoid deploying tools that are fragile and misleading. The work does not claim to have solved money laundering, but it provides a clear, reproducible framework for exposing the hidden assumptions in current systems, ensuring that when a model is deployed, it is ready for the complexity of the real world, not just the narrow view of its training.

Technical Summary: PolicyShift-AML

Problem Statement

Anti-money laundering (AML) transaction monitoring systems operate under a selective-label problem. Models are trained on data generated by specific investigation policies (rules, thresholds, or incumbent models) rather than a random sample of all illicit activity. Consequently, offline performance metrics often reflect an institution's historical scrutiny policy rather than the latent status of transactions. Treating unobserved transactions as benign can cause models to learn the audit mechanism itself rather than the underlying money laundering patterns. Furthermore, conventional temporal drift evaluations often conflate changes in criminal behavior with changes in alert policies, making it difficult to isolate the impact of label-generation mechanisms on model performance.

Methodology

The paper introduces PolicyShift-AML, a controlled stress-test protocol designed to audit the dependence of model performance on the policy that reveals positive labels.

Experimental Protocol

Fixed Latent Stream: The protocol utilizes a synthetic transaction stream (TransXion and IBM AMLWorld) with fixed latent ground-truth labels ( $Y$ ).
Variable Audit Policies: Five distinct audit policies are simulated (Uniform, Amount-focused, Rule-based, Model-feedback, and Typology-blind). Each is calibrated to an identical expected audit capacity (15% of the stream) to ensure fair comparison.
Delayed Confirmation: The protocol simulates a confirmation lag ( $\tau$ ), where the positive proxy $S_t = A_t Y_t$ (where $A_t$ is the audit decision) only becomes available after a delay (e.g., 30 days).
Holdout Strategy: A fixed latent holdout set is used to evaluate all models, ensuring that performance differences arise solely from the training policy, not deployment population shifts.

Estimators and Models

The study evaluates several learning approaches:

Visible-Label Baselines: Standard LightGBM and Multi-Layer Perceptron (MLP) models trained on observed positives ( $S=1$ ) and treating unobserved transactions ( $S=0$ ) as negatives.
Positive-Unlabeled (PU) Learning: A deployable non-negative PU (nnPU) estimator is implemented.
- Propensity Estimation: A LightGBM classifier fits the audit propensity $e(x) = P(A=1|X)$ using the full historical audit log.
- Class Prior: A Horvitz–Thompson estimator is used to estimate the class prior ( $\pi$ ) on a "matured cohort" (transactions where the confirmation delay has passed).
- Risk Decomposition: The nnPU risk function uses self-normalized inverse-propensity weights for observed positives and a clipped negative risk term derived from an independent uniform sample of the matured cohort.
Feature Engineering: The study compares transaction-only features against graph-history features (aggregates of sender/receiver prior counts, totals, and activity spans computed strictly from past events to prevent leakage).

Key Metrics

Latent Recall@K: Performance measured against the true latent labels ( $Y$ ) on the fixed holdout.
Training-Policy Sensitivity Index (TPSI): Measures the relative spread in performance between different training policies.
Visible-Label Optimism: The difference between performance on visible proxies ( $S$ ) and latent truth ( $Y$ ).

Key Results

1. Selective Labels Distort Model Comparison

On the TransXion dataset (3,029,170 transactions), the choice of training policy significantly alters model rankings and apparent performance, even when the latent holdout is fixed.

Performance Spread: For LightGBM, Latent Recall@500 ranged from 0.2655 (Uniform policy) to 0.3757 (Feedback policy), a 29.3% spread. For BCE/MLP, the spread was 26.1% (0.2660 to 0.3600).
Optimism Bias: Models trained on "Model-feedback" policies showed significant optimism. The apparent Recall@500 exceeded the latent Recall by 0.1085 for the BCE baseline, indicating that visible metrics can be highly misleading.
Correction of Previous Claims: The study corrects an earlier finding of a "39.7% deployment collapse." The revised analysis shows that while training policy sensitivity is high, cross-policy deployment collapse is not universally established; in some cases, feedback labels actually improved latent ranking.

2. Effectiveness of Estimated nnPU

The deployable nnPU estimator demonstrated robustness under benchmark assumptions:

Superiority over Baselines: Under Rule-based selection with a 30-day delay, the estimated nnPU achieved a Latent Recall@500 of 0.4021, significantly outperforming LightGBM (0.2352) and BCE (0.1787).
Oracle Proximity: The estimated nnPU results were nearly identical to an "oracle" nnPU (which uses true propensities), with a mean difference of at most 0.0005.
Policy Dependence: The nnPU advantage held under Rule and Amount policies but was not statistically significant under Model-feedback policies (where the difference from LightGBM was 0.0168, $p=0.25$ ).

3. Graph-History and Stress Tests

Graph-History Necessity: Graph-history features were found to be necessary for useful nnPU ranking in this configuration. However, the study refutes the claim that graph features guarantee policy invariance; the TPSI for graph-history models was not significantly lower than for transaction-only models in all cases.
Blind Typology Stress Test: In the IBM AMLWorld stress test, where a specific typology had zero audit support (exploration probability = 0), the nnPU model achieved a non-zero Recall (0.1660). The author clarifies this is inductive transfer from features learned on other typologies, not evidence of successful identification of the blind stratum. As exploration increased, recall did not improve monotonically, suggesting that nominal exploration does not automatically improve ranking within the tested range.

Significance and Claims

The paper positions itself as a re-audit and correction of previous work, emphasizing rigorous experimental design over claims of real-world causal recovery.

Policy-Aware Evaluation: The primary contribution is a reproducible protocol that isolates the impact of label-generation policies. The results support the conclusion that evaluation design is as consequential as model choice. Unequal audit capacity and mislabeled estimands can produce strong but false performance headlines.
Feasibility of PU Learning: The study demonstrates that a carefully implemented nnPU estimator, using audit propensities fitted on complete historical logs and a Horvitz–Thompson class prior, can effectively recover latent performance in synthetic, delayed-feedback environments.
Modesty on Real-World Application: The author explicitly states that these results do not establish efficacy on real institutional investigations. The benchmarks are synthetic, and assumptions such as perfect specificity ( $S=AY$ ) and no unmeasured confounding may not hold in practice. The work serves as a "stress-test" to expose assumptions and failures before deployment claims are made, rather than a prescription for immediate operational deployment.
Operational Recommendations: The paper advises validation teams to retain audit-decision logs, distinguish between decision time and confirmation time, and report stress tests under plausible label policies. It cautions that random exploration may help establish support but does not prescribe a universal percentage, noting that real auditing is constrained by legal, privacy, and fairness requirements.