FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets

FinSheet-Bench introduces a synthetic benchmark modeled on real private equity fund structures to evaluate LLMs on financial spreadsheet tasks, revealing that even the best-performing models currently lack the accuracy required for unsupervised professional use, particularly on complex, large-scale documents, and suggesting that reliable extraction will require separating document understanding from deterministic computation.

The Gist

Imagine you are a detective trying to solve a mystery, but instead of reading a clear novel, you are handed a massive, messy, handwritten ledger filled with numbers, crossed-out lines, and sticky notes. Now, imagine you hire a super-smart AI assistant to read this ledger, find specific clues, and do the math for you.

This paper, FinSheet-Bench, is essentially a report card on how well these AI detectives are doing at that specific job.

Here is the breakdown in simple terms:

1. The Problem: The "Messy Ledger"

In the world of finance (like Private Equity), people don't use neat, clean databases. They use Excel spreadsheets. These are often chaotic:

• They have merged cells (where one big box covers four smaller ones).
• They have headers that span three rows.
• They use bold text or colors to mean "this is a total" or "this is a warning."
• They have multiple tabs (sheets) that reference each other.

For a human, this is manageable. For an AI, it's a nightmare. The AI is like a robot that can only read a long string of text. When you feed it an Excel file, the computer has to turn that 2D grid into a 1D line of text. In doing so, the AI loses the "visual map" (like knowing that a bold number is a total because it's at the bottom of a column).

2. The Test: "FinSheet-Bench"

Since real financial data is top-secret (like a bank vault), the researchers couldn't use real documents. So, they built a giant, fake financial universe.

• They took the structure of real, messy financial spreadsheets.
• They filled them with fake company names and fake numbers.
• They created 24 different versions of these spreadsheets, ranging from "easy" (a small list) to "nightmare" (a massive document with 152 companies and 8 different funds).

Then, they asked 10 different top-tier AI models (from companies like OpenAI, Google, and Anthropic) to answer questions like:

• "How many funds are there?" (Easy)
• "List all companies in the newest fund." (Medium)
• "Calculate the average debt-to-EBITDA ratio for all funds." (Hard)

3. The Results: The "Smart but Clumsy" AI

The results were a mix of "Wow" and "Yikes."

• The Good News: The AIs are getting much smarter. Two years ago, they were barely passing. Now, the best AI (Gemini 3.1 Pro) gets about 82% accuracy. That sounds good, right?
• The Bad News: In the world of finance, 82% is a disaster.
  • If you are writing a poem, 82% is great.
  • If you are managing billions of dollars, getting 1 out of every 6 answers wrong is unacceptable. It's like a pilot landing a plane with a 1-in-6 chance of crashing.
  • The paper notes that for a professional to trust an AI, they need about 97% accuracy. We aren't there yet.

The "Complexity Cliff":
The AIs did great on simple questions (like "How many funds?"). But as soon as the task required math or sorting (like "Find the company with the highest debt and calculate the average"), their scores plummeted.

• On simple lookups: ~90% accuracy.
• On complex math: ~20–30% accuracy.

It's as if the AI is a brilliant librarian who can find a book on a shelf instantly, but if you ask it to add up the prices of 50 books on that shelf, it starts counting on its fingers and gets the total wrong.

4. Why Do They Fail?

The paper suggests two main reasons:

1. The Translation Problem: Converting a visual spreadsheet into text is like translating a map into a poem. You lose the spatial clues (like "this number is at the bottom, so it's a sum").
2. The Math Problem: AIs are great at predicting the next word in a sentence. They are not great at being calculators. They try to "guess" the math based on patterns rather than actually doing the calculation.

5. The Solution: Don't Ask the AI to Do Everything

The paper proposes a clever fix. Instead of asking the AI to "Read this whole spreadsheet and do the math," we should split the job:

1. The AI as the Scanner: Ask the AI to just find the numbers (e.g., "What is the debt for Company X?"). The AI is actually very good at this simple lookup.
2. The Computer as the Calculator: Once the AI finds the numbers, pass them to a standard computer program (like a calculator or Python script) to do the actual math, sorting, and averaging.

The Analogy:
Think of the AI as a super-fast intern and the computer as a calculator.

• Current Approach: You ask the intern to read the ledger, find the numbers, and do the long division in their head. They get tired and make mistakes.
• Proposed Approach: You ask the intern to just find the numbers and write them down. Then, you hand the list to the calculator to do the math. The intern never has to do the math, so they don't make mistakes.

The Bottom Line

AI is getting incredibly good at reading financial documents, but it is not ready to work alone in a bank or investment firm yet. It's too prone to calculation errors.

To make it useful, we need to stop treating the AI as a "brain" that does everything, and start treating it as a "scanner" that finds data, which we then feed into a reliable calculator. Until we build that bridge, humans will still need to double-check every single number.

Technical Summary

1. Problem Statement

The paper addresses a critical gap in the application of Large Language Models (LLMs) to the financial services industry. While LLMs have proven effective for text-heavy tasks (e.g., summarizing legal agreements or extracting clauses), they struggle with structured tabular data extraction and reasoning from complex financial spreadsheets.

Key challenges identified include:

• Complexity of Real-World Data: Financial spreadsheets (e.g., private equity portfolio monitors) are not clean CSVs. They feature multi-sheet structures, merged cells, hierarchical headers, embedded formulas, and visual formatting (bolding, colors) that convey semantic meaning.
• Low Error Tolerance: In professional finance, even minor numerical errors can lead to significant misvaluation. Industry standards often require ~97% accuracy for unsupervised automation, a threshold current models have not met.
• Lack of Benchmarks: Existing benchmarks (e.g., WikiTableQuestions, FinQA) use simplified, public data that fails to capture the irregular, confidential, and messy nature of private market fund data.

2. Methodology

A. Dataset Construction: FinSheet-Bench

To overcome the lack of public data, the authors created FinSheet-Bench, a synthetic benchmark modeled on real private equity fund structures.

• Source: Derived from 8 real portfolio monitoring spreadsheets sent by General Partners (GPs) to family offices.
• Synthesis Process:
  • Structural Fidelity: The layout, sheet organization, column naming, and fund dividers were preserved from the original files to maintain realistic complexity.
  • Data Anonymization: All cell values were regenerated. Company names, dates, and numerical values were replaced with fictitious data using scaling factors and perturbation to prevent reconstruction of original values.
  • Variants: The process generated 24 evaluation files (8 "Version A" originals, plus 8 "Version B" and 8 "Version C" variants with structural modifications like removed columns, split headers, and altered row counts).
• Ground Truth: Answers were computed via deterministic Python functions on standardized tables to ensure 100% verifiable correctness.

B. Experimental Setup

• Models Evaluated: 10 configurations from OpenAI, Google, and Anthropic, including frontier models (GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6) and their "reasoning" or "thinking" variants.
• Task Design: 16 question templates categorized into 7 types (e.g., Simple Lookup, Counting, Aggregation, Sorting, Complex Aggregation) across 4 complexity levels (Low to Very High).
• Input Modality: A critical methodological choice was text serialization. Excel files were converted to comma-separated text (CSV-like) with sheet delimiters. This reflects current industry RAG pipelines but strips visual cues (colors, borders) and spatial grid structure.
• Evaluation Protocol: A three-tier cascading verification system was used to grade model answers:
  1. Tier 1 (Exact): Strict regex and set matching (25% of answers).
  2. Tier 2 (Fuzzy): Tolerant matching with 5% numerical tolerance (25% of answers).
  3. Tier 3 (LLM Adjudication): A secondary LLM resolves ambiguous cases (48% of answers), with manual review for discrepancies.

3. Key Results

A. Overall Performance

• No Unsupervised Viability: No standalone model achieved the ~97% accuracy required for unsupervised professional use.
• Best Performer: Gemini 3.1 Pro achieved the highest accuracy at 82.4% (approx. 1 error per 6 questions).
• Other Top Models: GPT-5.2 (Reasoning) at 80.4%, Claude Opus 4.6 (Thinking) at 80.2%, and Gemini 3 Pro at 80.2%.
• Context Limits: Older/smaller models like GPT-3.5-Turbo failed to process 25% of files due to token limits, resulting in a low effective accuracy of 24.8%.

B. The Complexity Gap

Performance degrades significantly as task complexity increases, a trend consistent across all models:

• Simple Lookups: High accuracy (~89–94% for top models).
• Counting & Filtering: Moderate accuracy (~66–84%).
• Aggregation & Sorting: Sharp decline (~53–79%).
• Complex Aggregation (Very High): Critical failure point. The best models dropped to 33.3% accuracy on tasks requiring median calculations or multi-step statistical operations.

C. Impact of Reasoning

Enabling "reasoning" or "thinking" modes yielded substantial gains:

• GPT-5.2: Improved from 57.7% to 80.4% (+22.8 percentage points).
• Claude Opus 4.6: Improved from 66.7% to 80.2% (+13.6 percentage points).
• Cost: Reasoning modes significantly increased output token usage (5x–11x) and response time, though accuracy improved.

D. File Complexity

Accuracy is inversely correlated with file size and structural complexity.

• The simplest file yielded 86.2% accuracy.
• The most complex file (152 companies, 8 funds) yielded only 48.6% accuracy across all models.
• This suggests that current LLM architectures struggle to maintain context and structural integrity over long, complex documents.

4. Key Contributions

1. FinSheet-Bench: The first benchmark specifically designed for private equity financial spreadsheet extraction, addressing the "data gap" caused by confidentiality.
2. Empirical Evidence of Limitations: Demonstrates that even the most advanced LLMs (as of early 2026) cannot reliably perform unsupervised financial extraction, particularly for complex reasoning tasks.
3. Failure Mode Analysis: Distinguishes between parsing failures (due to text serialization losing visual cues) and reasoning failures (inability to perform multi-step arithmetic or sorting). The paper argues that reasoning failures are the dominant bottleneck.
4. Architectural Insight: Proposes that the solution lies not in larger models, but in decomposing the workflow: separating document understanding (schema discovery) from deterministic computation (calculations).

5. Significance and Future Directions

• Industry Impact: The findings suggest that financial institutions cannot yet rely on "end-to-end" LLMs for due diligence automation. Human review remains essential for high-stakes decisions.
• Proposed Solution (Staged Pipeline): The authors suggest a "brute-force" conceptual approach: use LLMs only for schema discovery and single-value lookups (where they excel at ~94% accuracy), then perform all aggregations, sorting, and calculations via deterministic code. This hybrid approach could bypass the reasoning limitations of LLMs.
• Future Work: The paper calls for research into:
  • Multimodal Inputs: Testing vision-capable models on rendered spreadsheet images to recover lost visual cues.
  • Alternative Formats: Evaluating structured JSON or HTML inputs vs. text serialization.
  • Agentic Workflows: Developing multi-stage pipelines that offload calculation to code interpreters.

In conclusion, while LLMs have made rapid progress in financial text analysis, they hit a "reasoning wall" when applied to complex, structured financial spreadsheets. The path forward requires architectural shifts that leverage LLMs for retrieval while offloading computation to deterministic systems.