AuditFraudBench: Benchmarking Audit Judgment in Detecting Fraudulent Misstatements

Technical Summary of AuditFraudBench: Benchmarking Audit Judgment in Detecting Fraudulent Misstatements

Problem Statement
While Large Language Models (LLMs) have demonstrated proficiency in financial analysis, forecasting, and detecting surface-level factual or numerical errors, their capacity to identify fraudulent financial misstatements remains underexplored. Unlike simple errors, fraudulent misstatements often involve plausible, seemingly compliant disclosures that obscure distorted performance drivers, opportunistic accounting choices, or omitted risk contexts. These disclosures may be internally consistent in their figures but strategically framed to mislead investors regarding the true sources of performance (e.g., attributing growth to sustainable operations when it is actually driven by revenue timing or expense deferral). Existing benchmarks primarily focus on fact verification, numerical consistency, or rule compliance, failing to evaluate the more complex task of detecting strategically misleading narratives and disguised accounting manipulation grounded in actual enforcement cases.

Methodology
To address this gap, the authors introduce AuditFraudBench, an enforcement-grounded benchmark constructed from authentic company filings and regulatory materials. The dataset is derived from U.S. Securities and Exchange Commission (SEC) Accounting and Auditing Enforcement Releases (AAERs), original and restated 10-K and 10-Q filings, structured financial statements, and Management's Discussion and Analysis (MD&A) disclosures.

The benchmark construction pipeline involves:

1. Data Collection: Crawling SEC AAER index pages to extract metadata and downloading enforcement documents.
2. Processing & Cleaning: Text extraction, segmentation, and quality control to remove corrupted or non-parsable files.
3. Case Identification: Using a combination of rule-based features and semantic understanding (via DeepSeek) to identify substantive accounting manipulation cases.
4. Structured Enhancement & Alignment: Extracting specific fields (e.g., manipulation mechanisms, fiscal periods) and aligning AAERs with corresponding original and restated filings from the SEC EDGAR system to create "original–restated" pairs.
5. Sample Construction: Integrating structured financial fact signals (XBRL data) and narrative evidence signals (MD&A passages) to form supervised samples.

The benchmark evaluates LLMs through three complementary tasks:

• Task 1: Profit Source Attribution: Determining whether a management's explanation for a performance change is correct or misleading, and identifying the true underlying driver (e.g., premature revenue recognition, reserve release) based on original vs. restated figures.
• Task 2: Misleading Narrative Detection: Assessing whether a disclosure passage creates a misleading impression (e.g., through selective emphasis or omitted context) even if the text is not explicitly false in isolation.
• Task 3: Fraud Pattern Classification: Mapping the specific misconduct mechanism described in enforcement actions to predefined categories: Revenue Timing, Accounting Estimate, Earnings Smoothing, Expense Deferral, or Narrative Distortion.

Key Contributions

1. AuditFraudBench Dataset: The creation of the first enforcement-grounded benchmark specifically designed to evaluate LLMs on detecting fraudulent financial misinformation in audited corporate reporting, focusing on misleading narratives and disguised accounting manipulation.
2. Task Design: The formulation of three complementary tasks that require models to jointly reason over financial figures, disclosure framing, restatement evidence, and enforcement-grounded fraud mechanisms, moving beyond simple fact verification.
3. Evidence-Grounded Evaluation: The construction of the benchmark from authentic regulatory sources (AAERs) and filings, enabling an evaluation that better reflects audit judgment and investor-relevant disclosure analysis.

Evaluation Results
The authors evaluated a range of proprietary (GPT series) and open-source (DeepSeek, Qwen series) LLMs, including both reasoning and non-reasoning variants.

• Task 1 (Profit Source Attribution): Most models achieved near-perfect classification accuracy (often 1.000), correctly identifying whether an attribution was misleading. However, explanation metrics (ROUGE) remained low (e.g., GPT-5.5 achieved only 0.249 ROUGE-1), indicating that models often fail to precisely articulate the true accounting driver despite correct classification.
• Task 2 (Misleading Narrative Detection): This proved to be the most difficult task. All models performed poorly, with the best model (Qwen3-32b-R) achieving only 0.245 accuracy and 0.393 F1. This suggests significant challenges in detecting misleading impressions that are not explicitly false.
• Task 3 (Fraud Pattern Classification): Models showed moderate but limited ability. GPT-5.5 achieved the highest classification accuracy (0.522), while Qwen3-32b-R achieved the best explanation scores.
• General Observations:
  • Reasoning vs. Non-Reasoning: Reasoning-tuned models did not consistently outperform non-reasoning counterparts; in some cases, they performed worse.
  • Model Scale: Increasing model size (e.g., from Qwen3-8b to Qwen3-32b) did not guarantee improvement, particularly for explanation quality.
  • Family Performance: Strong proprietary models (GPT, DeepSeek) did not reliably dominate smaller open-source models (Qwen) across all metrics.

Significance
The paper posits that AuditFraudBench provides a challenging testbed for audit-relevant, evidence-grounded evaluation of LLMs. The results indicate that current models, regardless of size or architecture, struggle to jointly reason over financial evidence, narrative framing, and enforcement-grounded fraud mechanisms. The authors conclude that detecting such high-risk, intentionally disguised misinformation requires specialized financial and accounting reasoning capabilities that extend beyond general model scale or generic reasoning tuning. The benchmark thus serves as a necessary tool for systematically evaluating whether LLMs can support financial analysis and audit assistance in recognizing strategic misinformation.