💬 NLP

FinBalance: A Multi-Document Accounting Reconciliation Benchmark

The paper introduces FinBalance, a multi-document accounting reconciliation benchmark built from source documents across diverse industries and difficulty levels, which reveals that current large language models struggle to accurately reconcile source materials into consistent balance sheets, often exhibiting significant gaps between their reported financial statements and those derived from their own generated entries.

Original authors: Sasank Tumpati, Devansh Agarwal, Ayush Kedia, Arjun Neekhra, Murari Mandal, Krishna Garg, Yash Sinha, Suman Gupta, Dhruv Kumar

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sasank Tumpati, Devansh Agarwal, Ayush Kedia, Arjun Neekhra, Murari Mandal, Krishna Garg, Yash Sinha, Suman Gupta, Dhruv Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a financial mystery. You have a messy pile of clues: invoices, bank statements, contracts, and receipts. Your job isn't just to read them; it's to figure out exactly what happened, write down a formal report (a "journal entry") for every single transaction, and then add them all up to create a final "Balance Sheet" (a snapshot of the company's financial health).

If you find a clue that contradicts another (like an invoice saying you paid \ $100 but a bank statement saying you paid \$ 120), you must flag that error immediately. You can't just ignore it or guess the answer.

This is exactly what the paper FinBalance is about. It introduces a new "exam" for Artificial Intelligence (AI) to see if it can do this real-world accounting detective work.

The Problem: AI is Good at Reading, Bad at Reconciling

The authors point out that most current AI tests are like giving a student a finished textbook chapter and asking, "What is the main idea?" The answer is already written there; the AI just has to find it.

But real accounting is harder. It's like giving the student a shoebox full of crumpled receipts, a bank statement, and a contract, and saying, "Figure out the math, write the official report, and tell me if anything doesn't add up."

The Solution: The FinBalance Benchmark

The researchers built a massive, synthetic "training ground" called FinBalance.

The Setup: They created a computer program that acts like a strict accountant. It generates thousands of fake but realistic business scenarios across 8 different industries (like retail, healthcare, and tech).
The Documents: For each scenario, it creates a bundle of documents with text inside them (simulating scanned PDFs). Some are real evidence, some are "distractors" (fake papers designed to trick you), and some contain deliberate contradictions.
The Ground Truth: Because a computer generated the scenario, the AI knows the exact correct answer. It knows the right journal entries, the right final balance sheet, and exactly which documents support which numbers.

The Test: How Did the AI Do?

The researchers tested six of the smartest AI models available today on 710 of these accounting puzzles. The results were surprising and humbling:

The "Math vs. Report" Gap:
Imagine a student who can do the math perfectly in their head but writes the final answer on the test paper incorrectly.
- The AI models were surprisingly good at figuring out the individual transactions (the "journal entries").
- However, when they tried to add those up and write the final "Balance Sheet," they often failed.
- The Stat: In four out of six models, there was a huge gap. They could reconstruct the correct financial picture from their own notes, but the report they actually submitted was wrong. It's like a chef cooking a perfect meal but serving it on the wrong plate with the wrong label.
The "Lost Receipt" Problem:
The biggest failure wasn't the math; it was linking the answer to the proof.
- When the AI got a number right, it often couldn't point to the specific document that proved it.
- Even when the researchers told the AI, "You must cite your sources," it barely improved. The AI was struggling to find the needle in the haystack, not because it didn't want to, but because it couldn't distinguish the real evidence from the distractors.
The "Second Opinion" Trick:
The researchers tried a clever fix. After the AI submitted its answer, they ran the AI's own numbers through a strict calculator (a "ledger") and showed the AI the difference between what it said and what its own numbers implied.
- Result: This "feedback" helped the AI fix its final report significantly. It's like a teacher saying, "You said the total is \ $100, but if you add your own list, it's \$ 120. Check your math."
- The Catch: This fix had a downside. When the AI was trying to spot contradictions (errors in the documents), this feedback sometimes made it worse at spotting the errors, because it got too focused on fixing the math instead of realizing the documents were broken.

The Verdict

The paper concludes that while AI is getting better at understanding financial text, it is not yet ready to be a reliable accountant. It struggles to:

Connect the dots between different documents.
Aggregate (add up) its own findings into a consistent final report.
Distinguish between real evidence and fake distractions.

The authors emphasize that this benchmark is a "stress test." If an AI can't solve these clean, computer-generated puzzles perfectly, it certainly can't be trusted with messy, real-world accounting where receipts are blurry, pages are missing, and the rules are even more complicated.

In short: AI is currently a great "reader" of financial documents but a poor "reconciler" of them. It can find the numbers, but it often loses the trail of evidence and fails to write the final report correctly.

Technical Summary: FinBalance: A Multi-Document Accounting Reconciliation Benchmark

Problem Definition

Current financial Natural Language Processing (NLP) benchmarks predominantly evaluate models on pre-processed artifacts, such as extracted tables from filings or pre-computed financial statements. Models are typically tasked with answering questions or performing analysis over these already-structured outputs. However, real-world accounting workflows begin earlier: they require the reconciliation of raw source documents (invoices, bank statements, contracts, payment notices) into journal entries, the aggregation of those entries into a balance sheet, and the detection of contradictions within the document bundle.

Existing benchmarks fail to test this "source-to-statement" pipeline. They do not require models to:

Extract balanced journal entries citing specific supporting documents.
Aggregate entries into a final balance sheet that respects the accounting equation.
Identify and classify inconsistencies among source documents rather than silently resolving them.

Methodology: The FinBalance Benchmark

The authors introduce FinBalance, a synthetic but human-authored benchmark designed to simulate the full accounting reconciliation pipeline.

Benchmark Construction

Generator Architecture: A deterministic generator composes records based on human-authored specifications. These specifications define business scenarios, document schemas, accounting policies, tax/foreign exchange (FX) treatments, and inconsistency templates.
Data Composition: The generator samples from these specifications to create document bundles containing:
- Posting Documents: Invoices, bills, statements, vouchers.
- Support Documents: Contracts, rate sheets, certificates.
- Adjustment Documents: Period-end accruals, rollforwards.
- Distractors: Visually plausible but neutral documents (duplicates, void notices).
- Opening Trial Balance: The starting state for the ledger.
Ground Truth: Because the generator uses a fixed double-entry ledger, the ground truth (journal entries, final balance sheet, and inconsistency codes) is computable, auditable, and reproducible. No manual labeling occurs after generation.
Scope: The benchmark covers eight industries (e.g., retail, healthcare, SaaS), three period types (month, quarter, year), and five difficulty levels. It includes concept flags for complex accounting areas like ASC 606 revenue recognition, leases, deferred tax, and multi-jurisdictional tax.
Inconsistency Handling: A specific subset of records (32% of the main split) contains "forced inconsistencies" (e.g., invoice-total mismatches, invalid exemption certificates). For these, the correct model output is an inconsistency flag and code, not a reconciled balance sheet.

Evaluation Metrics

The paper introduces two complementary metrics to distinguish between entry generation and self-consistency:

$BS_{exact}$ : Measures whether the model's self-reported final balance sheet matches the ground truth exactly.
$BS_{recon}$ : Parses the model's generated journal entries, replays them through the deterministic ledger against the opening balance, and checks if the implied balance sheet matches the ground truth.
- The difference ( $BS_{recon} - BS_{exact}$ ) is defined as the aggregation gap, indicating whether a model can generate correct entries but fails to aggregate them consistently.
Entry Matching: Strict matching requires correct debit/credit accounts, amounts, and sorted document references ( $doc\_refs$ ).
Inconsistency Code Match: For forced-inconsistency records, the rate of correctly identifying the specific contradiction code.

Experimental Setup

Models: Six contemporary Large Language Models (LLMs) were evaluated: Gemini 3 Flash, GPT-5, Claude Haiku 4.5, Grok-4.3, DeepSeek Chat, and Qwen 3 235B.
Ablation Studies: The authors tested various interventions, including:
- Citation Pressure: Explicit prompts to cite documents.
- Visibility: Removing account lists or support documents.
- Context Stress: Adding distractors or delaying relevant evidence.
- Ledger Feedback: Providing the model with the delta between its reported balance sheet and the one implied by its own entries, allowing one revision turn.
- Best-of-3: Sampling multiple candidates and selecting the best.
Validation: Expert finance professionals from major investment banks and a certified accountant reviewed a subset of records to validate the benchmark design, difficulty calibration, and label correctness.

Key Results

1. Low Overall Performance

On the 710-record evaluation split, the highest-performing model (GPT-5) achieved only 46% exact final-balance-sheet accuracy ( $BS_{exact}$ ). Strict document-grounded end-to-end accuracy remained significantly lower.

2. The Aggregation Gap

A critical finding is the divergence between entry quality and final statement consistency:

Four of the six models exhibited a 26–41 percentage point gap between $BS_{recon}$ (entries replayed) and $BS_{exact}$ (self-reported).
For example, DeepSeek Chat achieved a $BS_{recon}$ of 47.3% but only 6.7% $BS_{exact}$ .
This indicates that models often generate numerically plausible journal entries but fail to bind them to evidence or aggregate them into a self-consistent statement. GPT-5 was an outlier, showing high self-consistency (46.5% $BS_{exact}$ vs. 46.3% $BS_{recon}$ ).

3. Document Linking as the Primary Failure Mode

Retrieval Disambiguation: On entries with correct accounts and amounts, 52% cited the wrong supporting documents.
Prompt Ineffectiveness: Explicit "citation-pressure" prompting reduced this error rate by only 1.4 percentage points.
Evidence Dependency: When models were given an "evidence-only" oracle (hiding distractors), the mismatch rate dropped by 26.4 percentage points, suggesting the failure is in selecting the right evidence, not in following citation instructions.

4. Impact of Ledger Feedback

Repairing Balance Sheets: Providing models with the ledger feedback (deltas between reported and implied balance sheets) improved $BS_{exact}$ by 30–33 percentage points for models with an aggregation gap.
Trade-off: This improvement came at a cost. Ledger feedback degraded inconsistency-code detection by 13–52 percentage points on smaller models. The feedback mechanism biases the model toward reconciling contradictory documents rather than flagging them as errors.

5. Difficulty and Context

Difficulty Scaling: Accuracy dropped sharply from 88% at Difficulty Level 1 to 0% at Level 5.
Context vs. Logic: While adding distractors (context stress) reduced accuracy, the primary driver of failure was the complexity of accounting dependencies and document choice, not merely the length of the context.

Significance and Claims

The authors position FinBalance not as a solved problem but as a diagnostic testbed and a controlled prerequisite for real-world deployment.

Benchmark Necessity: The paper argues that if a model cannot reconcile clean, fully specified document bundles with exact ground truth, it is unlikely to be reliable in real-world accounting environments where layouts, OCR errors, and missing evidence are messier.
Synthetic but Human-Authored: While the data is synthetic, the accounting logic, policies, and scenarios are human-authored, ensuring the benchmark tests genuine domain reasoning rather than model-invented patterns.
Diagnostic Value: The benchmark reveals specific failure modes (aggregation gaps, retrieval disambiguation) that are invisible in standard QA benchmarks.
Future Utility: By releasing the generator (rather than just a static dataset), the authors enable the creation of fresh, stratified samples to prevent data contamination and allow for the expansion of the scenario space (new industries, policies, contradictions) without manual re-labeling.

The paper concludes that current LLMs struggle with the full pipeline of document-grounded accounting. While they can often identify plausible entries, they fail to consistently link these to evidence and aggregate them into a correct final statement, highlighting a significant gap between current capabilities and the requirements of professional accounting workflows.