Audit Detection Risk Reduction Using Machine Learning: Evidence from 3.3 million Transactions

Technical Summary: Audit Detection Risk Reduction Using Machine Learning

Problem Statement
The study addresses the critical limitation of detection risk in public sector auditing, specifically within the context of the International Standards on Auditing (ISA). Under ISA 200, detection risk is the probability that audit procedures fail to identify material misstatements. Traditional public sector audits rely heavily on Monetary Unit Sampling (MUS), where accounts are selected proportionally to their monetary value. In entities with thousands of accounts, a standard 20% sample leaves a significant portion of lower-balance accounts unexamined. The authors note that in their case study, this approach results in a detection risk ranging from 38.05% to 52.7%, implying that roughly half of anomalous accounts remain unreviewed. Furthermore, existing machine learning (ML) research in auditing often relies on proprietary or synthetic datasets lacking validation against official audit outcomes, and frequently fails to address the "ground truth" problem where labeled fraud data is unavailable prior to the audit conclusion.

Methodology
The authors propose and evaluate a four-layer ML framework designed to reduce detection risk without requiring pre-labeled training data. The study utilizes a dataset of 3,329,189 general ledger transactions from a Mongolian public sector energy utility across three fiscal years (FY2023–FY2025), comprising 9,909 unique account rows and a cumulative debit flow of MNT 16.34 trillion.

The framework architecture operates as follows:

1. Data Preparation (Layer 1): Excel-based general ledgers are ingested, and debit-credit balance identities are verified (ISA 500). Features are scaled using StandardScaler fitted only on training years (FY2023–FY2024) to prevent data leakage into the holdout year (FY2025).
2. Unsupervised Ensemble Labeling (Layer 2): To overcome the lack of labeled ground truth, the system generates "pseudo-labels" using an ensemble of three unsupervised methods:
   • Isolation Forest: Uses 200 estimators with a contamination parameter of 0.10.
   • Z-score Analysis: Flags accounts with absolute z-scores > 2.0 on debit/credit turnover.
   • Debit-Credit Ratio Screening: Flags accounts above the 95th percentile of the debit-to-credit ratio.
   • Voting Rule: An account is labeled anomalous if the Isolation Forest flags it, OR if both Z-score and Turn-ratio methods independently flag it.
3. Supervised Classification (Layer 3): The pseudo-labeled data trains three supervised classifiers: Random Forest (RF), Gradient Boosting (GB), and Decision Tree (DT). Models are evaluated using 5-fold stratified cross-validation and a strict temporal holdout (FY2025 data withheld from all training).
4. Explainable AI (XAI) & ISA Alignment (Layer 4): SHAP (SHapley Additive exPlanations) values are mapped to specific ISA requirements (e.g., mapping log_abs_change to ISA 520 analytical procedures). This generates audit-ready documentation that translates ML feature importance into professional audit language.

Key Results

• Performance vs. Baseline: The Random Forest model achieved an F1 score of 0.966 and an AUC of 0.999. Crucially, it reduced Detection Risk to 2.01%, a 19-fold improvement over the MUS 20% baseline (38.05%–52.7%).
• Statistical Significance: McNemar's test confirmed the superiority of the ML model over MUS with a chi-square value of 1,666.63 (p < 0.001). The directional advantage (b/c ratio) was 107.2, meaning for every account MUS detected that RF missed, RF detected 107 accounts that MUS missed.
• Validation: Against official National Audit Office (NAO) reports for FY2023 and FY2024, the framework achieved 85% to 90% true-positive alignment with documented material misstatements.
• Temporal Generalizability: On the unseen FY2025 data (temporal holdout), the Random Forest model maintained an F1 of 0.955 with perfect precision (1.000).
• Efficiency: Processing time for the full population dropped from 310–357 hours (MUS) to 4.5 hours (ML), a 98.7% reduction.
• Cross-Sector Transfer: When applied to five other sectors without retraining, the model achieved a mean F1 of 0.741, outperforming the MUS baseline in four of five sectors.

Novel Findings: The Aggregation Masking Effect
A significant contribution of the paper is the documentation of the "aggregation masking effect" regarding Benford's Law.

• Transaction Level: Analysis of 3.3 million individual transactions revealed statistically significant deviations from Benford's Law (Chi-squared = 15.51, p < 0.05).
• Account Level: When the same data was aggregated to the account level (9,909 rows), the Mean Absolute Deviation (MAD) dropped to 0.0052, falling below the threshold for "close conformance" (0.006).
• Implication: The authors argue that standard account-level Benford testing suppresses transaction-level anomalies, potentially leading to false negatives in fraud detection. They conclude that dual-level analysis is necessary to meet the fraud risk detection objectives of ISA 240.

Significance and Claims
The paper claims four primary contributions:

1. Self-Labeling Architecture: It demonstrates a deployable framework that generates training data via unsupervised ensemble voting, solving the "ground truth" barrier that prevents ML deployment in the first audit cycle of an engagement.
2. Official Validation: Unlike prior studies using synthetic data, this framework is validated against official government audit reports (NAO), achieving high alignment with confirmed material misstatements.
3. Benford's Law Insight: It provides empirical evidence that aggregation masks digit-distribution anomalies, challenging the standard practice of applying Benford's Law solely at the account level.
4. Operational Accessibility: The system is released as "Audit AI v10.0," an open-source web application compatible with Excel-based environments, requiring no licensing fees. This aims to reduce the analytics capability gap between large and small audit offices.

The authors maintain a modest tone regarding generalizability, acknowledging that the validation is based on a single case organization and that the framework is currently limited to structured financial data, excluding narrative or PDF-based evidence. They suggest future research should focus on multi-organization longitudinal validation and the integration of active learning components.