Empirical Evaluation of PDF Parsing and Chunking for Financial Question Answering with RAG

This paper presents an empirical study evaluating various PDF parsers and chunking strategies within Retrieval-Augmented Generation (RAG) systems for financial question answering, introducing a new benchmark called TableQuest to provide practical guidelines for building robust document understanding pipelines.

The Gist

Imagine you are a financial analyst trying to find a specific number in a massive, 200-page PDF report about a company's earnings. The problem? That PDF is designed for humans to read, not for computers to understand. It's like a beautifully decorated book where the text, tables, and charts are all mixed together without clear labels. If you ask a computer to "find the profit margin," it might get confused, looking at the wrong page or mixing up numbers from different tables.

This paper is essentially a scientific taste test to figure out the best way to teach computers how to read these messy financial PDFs and answer questions accurately.

Here is the breakdown using simple analogies:

1. The Problem: The "Messy Library"

Think of a financial PDF as a library where all the books have been shredded and thrown into a giant pile.

• The Goal: You want to find a specific sentence (e.g., "What was the revenue in 2023?").
• The Challenge: Before a computer can find that sentence, it has to:
  1. Reassemble the pages (Parsing): Figure out which text belongs to a table and which belongs to a paragraph.
  2. Cut the pages into bite-sized pieces (Chunking): Computers can't read a whole 200-page book at once; they have a "short attention span" (limited memory). So, you have to cut the text into small chunks.
  3. Find the right piece (Retrieval): When you ask a question, the system needs to grab the right chunk immediately.

2. The Experiment: The "Cooking Competition"

The researchers set up a massive experiment to see which "chefs" (tools) and "cutting styles" (strategies) work best. They tested this on two types of "recipes":

• FinanceBench: Questions based on text (like reading a story).
• TableQuest (New!): Questions based on tables (like reading a spreadsheet). This is a big deal because most previous tests ignored tables, which is where the real money data usually hides.

They tested:

• 6 Different "Scissors" (PDF Parsers): Some are basic scissors (just cut text), while others are high-tech laser cutters that can see tables and images.
• 6 Different "Cutting Styles" (Chunking): Some cut every 50 words, some cut at every sentence, and some use AI to cut only where the meaning changes.
• Different "Search Engines" (Retrievers): How the computer looks for the right piece.
• Different "Brains" (LLMs): The AI that actually reads the piece and answers the question.

3. The Key Findings (The "Secret Sauce")

🏆 The Best Scissors (Parsing)

• For Text: A tool called pdfminer was the champion. It's like a careful librarian who reads the raw text stream very accurately.
• For Tables: A tool called pdfplumber won. It's like a specialist who knows exactly how to separate a spreadsheet from the surrounding text.
• The "All-in-One" Trap: The fancy tool that uses heavy AI (Unstructured) was very accurate but extremely slow. It's like using a supercomputer to chop a single onion—it works, but it takes forever. For business, speed matters.

✂️ The Best Cutting Style (Chunking)

• Don't cut too finely: If you cut the text into tiny, random pieces, you lose the context (like cutting a sentence in half).
• The "Overlap" Trick: The researchers found that when cutting the text, you should let the pieces overlap slightly (like overlapping roof tiles). If you cut a page into chunks, make sure the last 25% of one chunk is repeated at the start of the next. This ensures that if a number or a sentence is split by the cut, the computer still sees the full picture.
• The Winner: A "Neural" cutter (AI that understands meaning) worked best, but a simple "Sentence" cutter was almost as good and much cheaper/faster.

🧠 The Best Brain (The AI Model)

• Bigger is better (but with limits): Bigger AI models (like GPT-5 or large open-source models) gave much better answers than small ones.
• The Sweet Spot: You don't need the biggest, most expensive model to get 90% of the way there. A "medium" sized model often gives the best balance of cost and accuracy.

4. The "TableQuest" Discovery

The researchers realized that previous tests were like testing a car only on smooth highways (text). They built TableQuest to test the car on off-road terrain (tables).

• Result: Many systems that were great at reading text failed miserably at reading tables. They couldn't figure out which row and column a number belonged to.
• Lesson: If you want to build a financial AI, you must test it on tables, or it will fail in the real world.

5. The Final Takeaway for Business

If you are a bank or a company trying to build an AI to read financial reports:

1. Don't overcomplicate it: You don't need the most expensive, slowest tools. A combination of pdfplumber (for tables) and a 25% overlap when cutting text works wonders.
2. Overlap is key: Always let your text chunks overlap slightly so you don't lose context.
3. Test on tables: Don't just test if your AI can read paragraphs; test if it can read spreadsheets.

In a nutshell: This paper is a guidebook that says, "Here is the exact recipe for building a financial AI that doesn't hallucinate, doesn't get confused by tables, and does it fast enough to be useful in a real bank."

Technical Summary

1. Problem Statement

PDF files are the standard format for financial disclosures (e.g., annual reports, earnings statements), yet they are designed for human consumption, not automated processing. They contain heterogeneous content (text, complex tables, images, forms) that poses significant challenges for Retrieval-Augmented Generation (RAG) systems.

• The Bottleneck: RAG performance relies heavily on the pre-processing stage. If PDF parsing fails to extract structure correctly (e.g., misaligned tables) or if chunking strategies disrupt logical units (e.g., splitting a table across chunks), the retrieval component fails to find the correct context, leading to hallucinations or incorrect answers.
• The Gap: While individual studies exist on PDF parsers or chunking strategies, there is no comprehensive, systematic evaluation of how these components interact within a unified RAG pipeline specifically for financial domain tasks, which require high precision in both narrative text and tabular data.

2. Methodology

The authors conducted a rigorous empirical study using a modular, end-to-end RAG pipeline to evaluate the impact of different components on retrieval accuracy and answer correctness.

Datasets

• FinanceBench: A public benchmark containing 150 text-based QA pairs derived from 84 financial PDFs (10-K, 10-Q, 8-K, earnings reports).
• TableQuest (New Contribution): A novel, publicly available benchmark created by the authors to address the lack of table-focused evaluation.
  • Construction: Built from the same 84 FinanceBench PDFs. It involves automated table detection, page filtering, and stratified sampling to create 116 QA pairs.
  • Difficulty Levels:
    • Easy: Single-table extractive (lookup values).
    • Medium: Single-table numerical (sums, ratios).
    • Hard: Multi-table analytical (synthesizing data across tables).
  • Validation: QA pairs were generated via GPT-4o with Chain-of-Thought prompting and verified by a majority vote of three proprietary LLMs (GPT-5, o3, GPT-4.1).

Experimental Setup

The study varied three critical components in a full-factorial design:

1. PDF Parsers (6 tools): Ranging from lightweight rule-based libraries (PyPDF2, PyMuPDF, pdfminer.six, pypdfium2, pdfplumber) to model-based/OCR systems (Unstructured).
2. Chunking Strategies (6 types): Token, Sentence, Recursive, Semantic, Semantic Double-Pass Merging (SDPM), and Neural chunking.
3. Retrievers (4 families): Keyword (BM25), Sparse-Hybrid (SPLADE), Dense (E5), and Late-Interaction (ColBERT).
4. LLMs: Tested various open-source models (1.5B to 27B parameters) and proprietary models (GPT-4.1-mini, GPT-5) as the generator.

Metrics:

• Retrieval: Precision@1 (P@1), Recall@3 (R@3), and Mean Reciprocal Rank (MRR) at the page level.
• Generation: Answer correctness evaluated via an LLM-as-a-judge (GPT-4o-mini).

3. Key Contributions

1. TableQuest Benchmark: A new, automatically verified dataset specifically designed to test RAG systems on complex tabular reasoning in financial reports, filling a gap left by text-only benchmarks.
2. Systematic Interaction Analysis: The first study to jointly evaluate the synergy between PDF parsers and chunking strategies, moving beyond isolated component testing.
3. Comprehensive RAG Benchmarks: A full evaluation of end-to-end pipelines across financial text and table domains, providing actionable guidelines for practitioners.
4. Open Source Release: All code, prompts, datasets, and configurations are publicly available to ensure reproducibility.

4. Key Results & Findings

RQ1: Retriever Selection

• Narrative Text (FinanceBench): Dense bi-encoders (E5) performed best, effectively modeling global semantics.
• Tabular Data (TableQuest): Late-interaction models (ColBERT) outperformed others significantly, as token-level matching is crucial for aligning queries with specific table cells.
• Hybrid Approach: SPLADE (sparse-hybrid) offered the best overall balance, performing robustly across both text and table queries.

RQ2: PDF Parsers

• Narrative Text: pdfminer yielded the highest retrieval performance (MRR 0.646), outperforming OCR-based tools.
• Tabular Data: pdfplumber was the clear leader (MRR 0.807) due to its superior table detection and cell grouping capabilities.
• Stability: pdfminer and Unstructured showed the least variance across different chunking strategies, making them robust choices.
• Runtime: Unstructured (OCR-based) was ~100x slower than stream parsers (e.g., PyMuPDF, pypdfium2), highlighting a trade-off between layout accuracy and latency.

RQ3: Chunking Strategies

• Best Strategy: Neural chunking (model-driven) achieved the highest MRR on both datasets, followed closely by Sentence chunking.
• Overlap: A 25% overlap (128 tokens) provided the optimal balance, significantly improving retrieval over 0% overlap without the storage costs of 50% overlap.
• Granularity: Excessive granularity (e.g., Semantic chunking creating many small chunks) increased indexing costs without consistent accuracy gains.

RQ4: Parser-Chunker Synergy

• Synergy Matters: Simple splits (Sentence/Token) paired with pdfplumber yielded top results for tables.
• Complexity vs. Gain: Advanced, structure-aware chunkers (Semantic, SDPM) only approached the performance of simple splits when paired with layout-preserving parsers (like pdfplumber), and often lagged behind.

RQ5: LLM Impact

• Scale Matters: Larger models significantly improved answer correctness. Medium models (12B–27B) outperformed small models (1.5B–4B) by 8–15 percentage points.
• Proprietary Edge: Proprietary models (GPT-5) maintained a clear lead, especially on complex table reasoning (TableQuest), though top open-source models are narrowing the gap.

5. Significance and Practical Guidelines

This paper provides critical, data-driven guidelines for building RAG systems in high-stakes financial domains:

1. Retriever Choice: Use ColBERT or SPLADE for financial documents containing tables; use E5 for pure text.
2. Parser Selection: Use pdfplumber if tables are critical; use pdfminer for text-heavy reports. Avoid heavy OCR (Unstructured) unless necessary due to latency costs.
3. Chunking: Use Neural or Sentence chunking with a 25% overlap. Avoid over-segmentation.
4. Model Scaling: Investing in medium-to-large LLMs yields substantial accuracy gains, but the choice of retrieval and parsing components has an equally profound impact on the system's ability to find the right information in the first place.

The study underscores that in RAG pipelines, the "garbage in, garbage out" principle applies strictly to the parsing and chunking stages; optimizing these pre-processing steps is as vital as selecting the generation model.