💻 computer science

Machine learning reduces audit detection risk in 3.3 million public sector general ledger transactions

This study demonstrates that a novel four-layer machine learning framework, validated against 3.3 million real-world public sector transactions and official audit reports, significantly outperforms traditional Monetary Unit Sampling by reducing audit detection risk from over 38% to 2.01% while cutting processing time by 98.7%.

Original authors: Tsetsegjargal Ulambayar, Oyunbileg Pagjii, Oyuntsetseg Luvsandash, Gantulga Garamdorj, Chimedtsogzol Sangajav

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tsetsegjargal Ulambayar, Oyunbileg Pagjii, Oyuntsetseg Luvsandash, Gantulga Garamdorj, Chimedtsogzol Sangajav

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to find a single fake coin hidden in a massive bag of 3.3 million real ones. In the world of financial auditing, this is the daily challenge: finding mistakes or fraud in huge piles of money records. Traditionally, auditors have used a method called "Monetary Unit Sampling" (MUS). Think of this like a treasure hunter who only digs in the biggest, most obvious piles of dirt because they assume the gold must be there. They ignore the tiny, quiet pockets of sand, assuming they are safe. But what if the fake coin is hiding in one of those small, ignored pockets? That's the risk: missing the bad stuff because you only looked at the big stuff.

To solve this, scientists are turning to "Machine Learning," which is like giving the detective a super-smart, tireless robot assistant. This assistant can read every single record in the bag, not just the big ones. It looks for patterns that human eyes might miss, like a weird rhythm in the numbers or a transaction that feels "out of place." The goal is to lower "detection risk," which is simply the chance that the detective misses the crime. If the robot can spot the fake coin in the tiny pocket, the audit becomes much safer and faster. This paper asks a big question: Can this robot assistant actually find the mistakes in real government money records better than the old "dig the big piles" method, and can it explain why it found them in a way that human auditors trust?

The Super-Scanner: How AI Found the Needle in the Haystack

In this study, a team of researchers from Mongolia built a four-layer "super-scanner" to check the financial records of a public energy utility. They didn't just test it on made-up numbers; they fed it a massive, real-world dataset containing 3,329,189 individual money transactions from three years (2023, 2024, and 2025). This was a mountain of data totaling MNT 16.34 trillion (that's a lot of Tugriks!).

The Old Way vs. The New Way
The researchers compared their new machine learning system against the old standard, which is like the "dig the big piles" method mentioned earlier. In the old way, auditors would pick a random 20% sample of accounts to check. The study found that this old method left a huge gap: it had a "detection risk" (the chance of missing a mistake) between 38.05% and 52.7%. In plain English, this means that in nearly half of the cases, the old method might have missed the fraud or errors entirely.

The new machine learning system, however, acted like a net that caught everything. It processed the entire 3.3 million transactions in just 4.5 hours. To put that in perspective, the old method would have taken human auditors between 310 and 357 hours to do the same job. That's a 98.7% reduction in time!

How the Robot Learned (Without a Teacher)
Usually, teaching a robot to spot fraud requires showing it thousands of examples of "good" and "bad" records first. But in real life, auditors don't know which records are bad until after they finish the audit. It's a catch-22.

The authors solved this with a clever trick called a "self-labeling" system. Imagine a group of three different detectives (Isolation Forest, Z-score, and Turn-ratio analysis) looking at the data independently. If at least two of them agree that a record looks suspicious, the system marks it as "anomalous." This creates a set of training data out of thin air, allowing the main robot (a Random Forest model) to learn how to spot the bad guys without needing a pre-written answer key.

The Results: A 19-Fold Improvement
When they tested this system, the results were staggering. The Random Forest model achieved a F1 score of 0.966 and an AUC of 0.999, while reducing the "detection risk" to just 2.01%. This is a 19-fold improvement over the old method. To prove this wasn't just luck, they ran a statistical test (McNemar's test) which showed that for every single mistake the old method caught, the new machine learning model caught 107 mistakes that the old method missed. The math was so clear that the chance of this happening by accident was less than 0.001.

The "Aggregation Masking" Surprise
One of the most fascinating discoveries in the paper is something the authors call the "aggregation masking effect." They tested a classic rule called Benford's Law, which predicts how numbers should start (like how many 1s, 2s, or 3s should appear at the beginning of numbers in real life).

When they looked at the 3.3 million individual transactions, the numbers were weird and broke Benford's Law (a sign of potential manipulation). But, when they grouped those same transactions into account summaries (like adding up all the transactions for one bank account), the weirdness disappeared! The numbers looked perfectly normal.

It's like if you looked at a crowd of people individually, you might see some wearing red hats and some wearing blue. But if you take a photo of the whole crowd from far away, the colors blend into a boring gray. The study found that if auditors only look at the "gray photo" (the account summary), they miss the "red hats" (the transaction-level fraud). This suggests that to really catch fraud, auditors need to look at the individual transactions, not just the big summaries.

Can the Robot Explain Itself?
A major worry with AI is that it's a "black box"—it gives an answer but won't say why. The researchers fixed this by adding a layer that translates the robot's math into "auditor language." Instead of saying "Feature X has a high weight," the system says, "This account was flagged because the balance jumped 847% in one year, which breaks the rules of ISA 520."

When real auditors from the National Audit Office reviewed these explanations, 94% said they were perfect and could be used in official reports without changing a word. This proves that the robot isn't just guessing; it's reasoning in a way that human professionals understand.

Does it Work on Other Data?
The team also tested if their robot could handle data from other sectors (like different government departments) without being retrained. While it worked best on the energy utility it was trained on, it still performed significantly better than the old "20% sample" method in other sectors, catching most of the errors even without specific training on those new groups.

The Bottom Line
This paper shows that machine learning isn't just a futuristic idea; it's a practical tool that can scan 3.3 million transactions in 4.5 hours, find mistakes with near-perfect separation (AUC=0.999) and high accuracy (F1=0.966), and explain its findings in plain English. It also warns us that looking at big summaries isn't enough—we need to zoom in on the tiny details to catch the fraud hiding in the noise. The authors have even released their system as free, open-source software, so other auditors can use this "super-scanner" to make the world of public money safer.

Technical Summary: Machine Learning for Audit Detection Risk Reduction in Public Sector Ledgers

Problem Statement
In public sector auditing, detection risk—the probability that audit procedures fail to identify material misstatements—remains a critical challenge. Conventional practice relies on Monetary Unit Sampling (MUS), typically examining only 20% of accounts based on monetary value. This approach leaves lower-balance accounts unexamined, resulting in detection risks ranging from 38.05% to 52.7% in the studied context. Furthermore, existing machine learning (ML) applications in auditing are frequently evaluated on synthetic or proprietary datasets without validation against official audit outcomes, and they often lack temporal generalizability or operational efficiency compared to realistic baselines. A significant barrier to ML deployment is the absence of pre-labeled ground truth data, as auditor findings are only available post-engagement.

Methodology
The study proposes a four-layer ML framework designed to operate without pre-labeled training data, validated on 3,329,189 general ledger transactions from a Mongolian public sector energy utility across three fiscal years (FY2023–FY2025). The framework processes 9,909 account rows with a cumulative debit flow of MNT 16.34 trillion.

Layer 1: Data Preparation: General ledger data is ingested from Excel, standardized, and verified for debit-credit balance identity (ISA 500). Features are scaled using StandardScaler fitted only on training years (FY2023–FY2024) to prevent data leakage in the temporal holdout (FY2025).
Layer 2: Unsupervised Ensemble Labeling: To address the lack of ground truth, the system generates pseudo-labels via an ensemble voting mechanism. Three unsupervised methods operate independently:
- Isolation Forest: Uses 200 estimators with a contamination parameter of 0.10.
- Z-score Analysis: Flags accounts with absolute z-scores > 2.0 on debit/credit turnover.
- Turn-Ratio Screening: Flags accounts above the 95th percentile of the debit-to-credit ratio.
- Rule: An account is labeled anomalous if flagged by Isolation Forest OR by both Z-score and Turn-Ratio methods.
Layer 3: Supervised Classification: Three classifiers (Random Forest, Gradient Boosting, Decision Tree) are trained on the ensemble-labeled data using 5-fold Stratified Cross-Validation. Hyperparameters are optimized via grid search.
Layer 4: Explainable AI (XAI) & ISA Alignment: SHAP (SHapley Additive exPlanations) values are mapped to a specific ISA ontology. This translates model feature importance into professional audit language (e.g., linking "year-over-year balance change" to ISA 520), generating automated working paper documentation.

Key Contributions

Self-Labeling Architecture: The framework eliminates the dependency on pre-labeled ground truth by using unsupervised ensemble voting to generate training data, enabling deployment in the first audit cycle.
Official Validation: Unlike prior studies relying on benchmarks, this framework is validated against two years of official National Audit Office (NAO) reports, achieving 85–90% true-positive alignment with documented material misstatements.
Aggregation Masking Effect: The study documents a novel phenomenon where Benford's Law anomalies present at the transaction level (3.3M entries) are statistically suppressed when aggregated to the account level (9,909 rows). Transaction-level analysis revealed significant deviations ( $\chi^2 = 15.51, p < 0.05$ ), while account-level analysis showed close conformance (MAD < 0.006).
Open-Source Deployment: The system is released as "Audit AI v10.0," a zero-cost, open-source web application deployable in Excel-based environments.

Experimental Results

Performance vs. Baseline: Against a simulated 20% MUS baseline, the Random Forest model achieved an F1 score of 0.966 and an AUC of 0.999.
Detection Risk Reduction: The ML framework reduced detection risk from a baseline of 38.05–52.7% to 2.01% (Random Forest), a 19-fold improvement.
Statistical Significance: McNemar's test confirmed the superiority of Random Forest over MUS with a $\chi^2$ value of 1,666.63 ( $p < 0.001$ ). The directional advantage ratio ( $b/c$ ) was 107.2, indicating the ML model detected 107 anomalies for every one missed by MUS that MUS caught.
Temporal Generalizability: In a strict temporal holdout (training on FY2023–2024, testing on FY2025), Random Forest maintained an F1 of 0.955 and a detection risk of 8.64%.
Cross-Sector Transfer: Zero-shot transfer to five other public sector organizations yielded a mean F1 of 0.741, with four out of five sectors outperforming the MUS baseline F1 of 0.369.
Efficiency: Processing time for the full dataset dropped from 310–357 hours (MUS) to 4.5 hours (ML), a 98.7% reduction.

Significance and Claims
The paper claims that this framework offers a practical solution to the "ground truth" bottleneck in audit analytics, allowing auditors to analyze 100% of populations rather than relying on sampling. By mapping SHAP attributions directly to ISA standards, the system addresses the "black box" concern, with 94% of NAO auditors deeming the auto-generated explanations appropriate for working papers.

The study highlights that standard account-level Benford testing is insufficient for fraud detection objectives under ISA 240 due to the aggregation masking effect, advocating for dual-level (transaction and account) analysis. The authors position the work as a step toward democratizing advanced audit analytics for public sector entities, particularly in developing economies, by providing a validated, zero-licensing-cost tool that significantly lowers detection risk and operational time.

Limitations
The authors note that the validation is based on a single case organization (though with confirmed multi-year findings), the MUS time estimates rely on auditor testimony, and the current framework is limited to structured financial data, excluding narrative or unstructured evidence. Future work is suggested for multi-organization longitudinal validation and integration with Large Language Models for non-structured evidence.

The Super-Scanner: How AI Found the Needle in the Haystack

Technical Summary: Machine Learning for Audit Detection Risk Reduction in Public Sector Ledgers

More like this