🤖 AI

AUDITFLOW: Executable Symbolic Environments for Structured Financial Reporting Verification

AuditFlow is a graph-grounded multi-agent framework that integrates a symbolic environment with adaptive search and deterministic verification to significantly improve structured financial audit accuracy by linking reported facts to taxonomy concepts and recomputing values, achieving 82.09% accuracy on a benchmark dataset.

Original authors: Yan Wang, Xuguang Ai, Jaisal Patel, Xueqing Peng, Fengran Mo, Yupeng Cao, Haohang Li, Mingyu Cao, Lingfei Qian, Víctor Gutiérrez-Basulto

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yan Wang, Xuguang Ai, Jaisal Patel, Xueqing Peng, Fengran Mo, Yupeng Cao, Haohang Li, Mingyu Cao, Lingfei Qian, Víctor Gutiérrez-Basulto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to verify a complex math problem written on a whiteboard, but the numbers are scattered across different rooms, and the rules for solving it are written in a giant, dusty encyclopedia.

This is exactly the challenge AUDITFLOW solves, but instead of a math problem, it's checking financial reports (like the ones public companies file with the government).

Here is how the paper explains it, broken down into simple concepts:

The Problem: Why AI Struggles with Money

Current AI models (like the ones that write essays or chat with you) are great at reading text. But financial audits aren't just about reading; they are about math and rules.

The Issue: To check if a number is correct, an AI can't just "guess" based on context. It has to find the specific number in a filing, look up the rule in a massive accounting encyclopedia (called a taxonomy), do the math itself, and then compare the result.
The Failure: If you ask a standard AI to do this, it often hallucinates (makes up numbers) or misses the rule because it tries to "reason" through text instead of actually doing the calculation. The paper notes that without help, the best AI gets this right only about 14% of the time.

The Solution: AUDITFLOW (The "Three-Person Audit Team")

The authors built a system called AUDITFLOW. Instead of asking one AI to do everything, they created a symbolic environment (a digital workspace) and a team of three specialized agents.

Think of it like a construction site:

The Environment (The Blueprint & The Tools):
The system builds a digital map. One part is the static map (the accounting rules/encyclopedia), and the other is the dynamic map (the actual company's financial report).
Crucially, the AI doesn't "guess" the math. It uses typed tools (like a calculator or a ruler) that are hard-coded to work perfectly. The AI decides which tool to use, but the tool does the actual math.
The Team (The Agents):
- Junior Auditor A (The Rule Checker): This agent looks at the rules first. It asks, "What does the accounting encyclopedia say this number should be?" It uses the static map.
- Junior Auditor B (The Evidence Hunter): This agent looks at the facts first. It digs through the company's actual filing to find the numbers, check if they match up with last year, and see if the units make sense. It uses the dynamic map.
- The Senior Auditor (The Judge): This agent listens to both juniors. If they agree, the job is done. If they disagree (e.g., one says "It's a violation" and the other says "It's fine"), the Senior Auditor asks them to look deeper into the specific area of conflict.

How They Reach a Verdict

The system doesn't just ask the AI, "What do you think?"
Instead, it uses a process called Evidential Aggregation.

Imagine the two Junior Auditors are holding evidence cards.
If both cards say "Violation," the system is 100% sure.
If one says "Violation" and the other is unsure, the system calculates a "trust score" based on how much evidence they actually found.
The final answer is a mix of the verdict (Pass/Fail), the expected value (what the number should be), and a trust score.

The Results: Why It Matters

The paper tested this on a dataset of real financial filings.

The Winner: AUDITFLOW got the answer right 82% of the time.
The Losers: The next best method (a single AI agent trying to do it all) only got 67% right.
The "Magic" Ingredient: The most important finding was that if you remove the deterministic tools (the hard-coded math checkers) and let the AI guess the math, the accuracy crashes to 17%.

The Bottom Line

The paper claims that for high-stakes tasks like financial auditing, you cannot rely on an AI's "intuition" or text-based reasoning alone. You need to separate the search (finding the info) from the computation (doing the math).

AUDITFLOW works because it treats the AI as a smart manager who knows how to use a calculator and a rulebook, rather than trying to be the calculator itself. It forces the AI to use a "symbolic environment" where the math is done by code, not by the AI's brain.

Technical Summary: AUDITFLOW

Problem Statement

Structured financial audit verification presents a significant challenge for Large Language Model (LLM) agents. Unlike standard text-based tasks, correctness in financial auditing depends on structured evidence rather than natural language alone. An agent must link reported facts to specific taxonomy concepts, traverse complex calculation or dimensional relations, recompute expected values, and apply audit rules.

Current approaches often fail because they rely on the LLM to perform the final interpretation, arithmetic, or rule application. Benchmarks like FinAuditing show that even the strongest evaluated models achieve only 13.86% accuracy on numerical-consistency tasks. The core difficulty lies in the fact that correctness is not stored in a single sentence but emerges from the interaction between reported facts and the structural constraints that govern them. Existing tool-augmented or multi-agent frameworks often leave the burden of verification within the language model, which struggles with deterministic arithmetic and strict rule adherence.

Methodology: AUDITFLOW

The authors propose AUDITFLOW, a graph-grounded multi-agent framework designed to separate adaptive search (handled by LLM agents) from deterministic verification (handled by a symbolic environment).

1. Dual-Graph Symbolic Environment

AUDITFLOW constructs a symbolic environment ( $E$ ) composed of two interconnected graphs:

Static Regulatory Graph ( $G^y_T$ ): Encodes the US-GAAP taxonomy. Nodes represent concepts with metadata (labels, definitions, data types), and edges encode presentation, calculation, and dimensional relationships.
Dynamic Filing Graph ( $G_F$ ): Encodes the specific XBRL filing. Nodes represent reported facts, contexts, periods, and units.
Bridge Edges: Link reported facts in the filing graph to their governing taxonomy concepts in the static graph, allowing the system to traverse from a specific number to its regulatory constraints and back.

2. Typed Action-Observation Interface

Agents interact with this environment through typed, deterministic tools rather than free-form graph operations. The tools are categorized into:

Retrieval: Fetching concept metadata, facts, and contexts.
Traversal: Navigating calculation trees and dimensional structures.
Forensic: Comparing facts across periods, dimensions, and magnitudes.
Deterministic Checkers: Executing terminal rule-specific checks (e.g., sign consistency, calculation-tree consistency, dimensional aggregation). These checkers return structured observations (verdicts, expected values, supporting facts) that the LLM cannot override.

3. Three-Agent Audit Protocol

The framework employs a specialized multi-agent workflow:

Compliance Auditor ( $A_1$ ): Adopts a "specification-first" approach. It starts from taxonomy metadata and rules, then checks reported facts against constraints. It uses static taxonomy tools.
Forensic Auditor ( $A_2$ ): Adopts an "evidence-first" approach. It starts from reported facts, performs period comparisons and dimensional breakdowns, and then confirms results using the same deterministic checks. It uses both static and dynamic tools.
Senior Auditor ( $A_s$ ): Reviews the reports from $A_1$ and $A_2$ . If they agree, the system proceeds. If they disagree, the Senior Auditor identifies the source of conflict (e.g., missing facts, wrong period) and provides targeted feedback, sending the juniors back for further investigation. The Senior Auditor manages the interaction but does not replace the deterministic tools.

4. Evidential Aggregation

The final verdict is not a simple vote but a result of evidential reasoning (Dempster-Shafer theory). Each junior auditor's report is mapped to a mass function over the hypotheses {Violation, Consistent, Ignorance}. These masses are fused to produce a final verdict, an expected value, an evidence trail, and a trustworthiness score ( $\tau$ ), which measures the amount of unresolved evidence.

Key Contributions

Definition of Graph-Grounded Verification: The paper defines a new task formulation where the answer depends on filing facts, taxonomy constraints, numerical checks, and inspectable evidence, rather than text generation alone.
AUDITFLOW Framework: A dual-graph, multi-agent system that connects US-GAAP taxonomy knowledge with filing-specific evidence, exposing both through typed deterministic tools.
Search-Computation Separation: The authors demonstrate that separating the LLM's role (guiding the search) from the environment's role (executing deterministic checks) is crucial for reliable audit verification.

Experimental Results

The framework was evaluated on a 67-instance subset of the FinAuditing FinMR task, covering three Data Quality Committee (DQC) rule families.

Performance: Under GPT-5.5, AUDITFLOW achieved 82.09% joint audit accuracy, outperforming the strongest baseline (Single Agent) by 14.93 percentage points. It also outperformed retrieval-based methods (GraphRAG, TreeRAG) and existing benchmarks (FinAuditing, Herculean) across all tested backbones (GPT-4o, Claude Sonnet 4.6, Qwen-397B).
Robustness: The system maintained high accuracy (80.60%) across different strong backbone models, indicating the framework's effectiveness is not tied to a single proprietary model.
Ablation Studies:
- Deterministic Checks: Removing the symbolic environment's deterministic checks caused accuracy to plummet from 82.09% to 17.91%, proving that the LLM cannot reliably replace the verification step.
- Required-Tool Gate: Removing the gate that forces agents to complete specific checks before finalizing increased invalid outputs from 1.49% to 17.91%.
- Evidential Aggregation: Removing the fusion step reduced accuracy to 77.61%.

Significance and Claims

The paper argues that reliable audit agents must be built around executable structured environments rather than text-only reasoning. The primary contribution is the demonstration that while LLMs are effective at navigating complex information spaces (search), they are unreliable at performing the final, load-bearing verification steps (computation).

The authors claim that AUDITFLOW successfully addresses the "verification gap" in financial AI by:

Offloading arithmetic and rule application to deterministic tools.
Using multi-agent protocols to explore evidence from complementary perspectives.
Providing inspectable evidence trails and trustworthiness scores, making the audit process transparent and auditable.

The paper concludes that this design pattern—separating search from computation in a symbolic environment—is essential for high-stakes domains where correctness depends on structured constraints rather than semantic plausibility.