💬 NLP

Decomposing Retrieval Failures in RAG for Long-Document Financial Question Answering

This paper identifies and addresses the critical failure mode in financial RAG systems where the correct document is retrieved but the specific page containing the answer is missed, proposing a domain fine-tuned page scorer that significantly improves page and chunk-level retrieval performance on long regulatory filings.

Original authors: Amine Kobeissi, Philippe Langlais

Published 2026-02-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amine Kobeissi, Philippe Langlais

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a financial detective trying to solve a mystery. You have a massive library of thousands of books (these are the SEC filings, which are long, boring, and full of numbers). Your job is to answer a specific question, like "How much money did Company X make last year?"

To help you, you have a super-smart robot assistant (the AI) who can read and write answers. But the robot is blind; it can't see the whole library at once. So, you need a Librarian (the Retriever) to find the right pages and hand them to the robot.

The Problem: Finding the Right Book, but the Wrong Page

In the past, researchers thought the main problem was finding the right book. If the Librarian handed the robot the correct annual report, everyone assumed the robot would do a great job.

But this paper discovered a sneaky failure mode: The Librarian finds the right book, but hands over the wrong pages.

Imagine you ask for the recipe for "Chocolate Cake." The Librarian grabs the correct cookbook (the Document) but hands you the pages about "How to wash dishes" or "The history of flour" instead of the cake recipe. The robot, seeing the right book but the wrong pages, tries to guess the recipe. It might sound plausible, but it's wrong.

In the financial world, this is dangerous. If the robot guesses the wrong profit number, investors could lose money.

The Solution: A "Page Scout"

The authors realized that simply handing the robot chunks of text wasn't enough. They needed a smarter way to find the specific pages inside the book before even looking at the tiny text chunks.

They built a new tool called a "Page Scout" (a domain-fine-tuned page scorer).

Here is how their new system works, using an analogy:

The Old Way (Direct Chunk Search): You ask the Librarian, "Find me the text about cake." The Librarian scans the whole library, pulls out random 10-page snippets from everywhere, and hopes one of them has the recipe. It's like searching for a needle in a haystack by grabbing random handfuls of hay.
The New Way (Page-then-Chunk):
- Step 1 (The Scout): First, the "Page Scout" looks at the Table of Contents or the spine of the book. It asks, "Which specific pages in this book are likely to have the cake recipe?" It ignores the other 90% of the book that is just about washing dishes.
- Step 2 (The Search): Once the Scout says, "It's definitely on pages 40 to 45," the Librarian only looks at those specific pages to find the exact paragraph.

Why This Matters

The authors tested this on a dataset called FinanceBench (150 tricky financial questions).

The "Oracle" Test: They imagined a perfect scenario where a human knew exactly which book and which pages were right. Even then, the robot struggled if it didn't get the exact text. This proved that finding the right pages is the hardest part.
The Results: Their new "Page Scout" system was much better than the old methods.
- It found the correct pages 55% of the time, compared to the old methods which only found them 34-46% of the time.
- Because the robot got the right pages, it gave the right answers much more often.

The Catch (Limitations)

The paper is honest about its limits:

The Training: The "Page Scout" was trained mostly on 10-K reports (the big annual reports). It works great there, like a specialist who knows annual reports inside out. But when they tried it on Earnings Calls (which are more like casual conversations/transcripts), the Scout got confused because the "book" looked different.
The Data: They only tested 150 questions. To be truly sure it works for everyone, they'd need to test it on thousands more.

The Big Takeaway

In the world of AI and finance, finding the right document isn't enough. You have to find the right page inside that document.

Think of it like this: If you are looking for a specific quote in a 500-page novel, finding the book is only 50% of the battle. The other 50% is knowing exactly which page to open. This paper teaches us how to build a better "Page Scout" so our AI doesn't just guess the answer, but actually finds the proof.

1. Problem Statement

Retrieval-Augmented Generation (RAG) is increasingly used for financial question answering (QA) over long regulatory filings (e.g., SEC 10-K, 10-Q reports). However, current systems suffer from a specific failure mode: the correct document is retrieved, but the specific page or chunk containing the answer is missed.

The Gap: Standard RAG pipelines often retrieve the correct filing (document-level success) but fail to locate the precise context (page/chunk-level failure) within that filing. This leads to generators extrapolating from incomplete context, producing plausible but incorrect answers.
Limitations of Existing Metrics: Standard metrics like answer accuracy or ROUGE-L conflate retrieval failures with generation errors. Document-level recall metrics obscure the fact that the answer-bearing context was missed.
Domain Challenges: Financial filings are long, semi-structured, contain dense tables, and use formal accounting terminology that often mismatches natural language queries.

2. Methodology

The authors propose a framework to decompose retrieval performance into three granularities: Document, Page, and Chunk.

A. Oracle-Based Analysis

To establish empirical upper bounds and isolate failure points, the authors define three retrieval settings:

Standard Retrieval: Retrieval over the entire corpus.
Oracle Document Retrieval: The search space is restricted only to the gold (correct) filing. This measures the system's ability to find the correct page/chunk given the right document.
Oracle Page Retrieval: The search space is restricted to the gold filing and the gold pages. This measures the system's ability to find the correct chunk given the right pages.

These "Oracle" settings quantify the "headroom" available if retrieval were perfect at specific levels.

B. Proposed Solution: Domain Fine-Tuned Page Scorer

The core contribution is a novel Page-then-Chunk retrieval strategy. Instead of retrieving chunks directly from the whole corpus, the system operates in two stages:

Stage 1 (Page Ranking): A bi-encoder, fine-tuned specifically for financial filings, ranks pages within the corpus.
- Training: The model is trained using contrastive learning (Multiple Negatives Ranking loss) on pairs of (Question, Gold Page). It treats other pages in the batch as negatives.
- Architecture: Initialized from BAAI/BGE-M3, it maps queries and normalized page text (truncated to 2000 chars) into a shared vector space.
Stage 2 (Chunk Retrieval): A standard chunk retriever searches only within the top- $P$ pages identified in Stage 1.

This approach exploits the semantic coherence of pages (a page often contains a single coherent topic) to reduce noise from irrelevant sections of a long document.

C. Experimental Setup

Dataset: A 150-question subset of FinanceBench, covering 10-K, 10-Q, Earnings Call, and 8-K filings.
Baselines: Evaluated diverse strategies including Dense (BGE-M3), Sparse (BM25, SPLADE), Hybrid (RRF), Query Reformulation (HyDE, Multi-HyDE), Hierarchical (Parent-Child), and Reranking (Cross-Encoder).
Generator: Qwen-2.5-7B-Instruct (fixed across all experiments).
Metrics: Document Recall, Page Recall, Max BLEU/ROUGE-L between retrieved chunks and gold evidence, and Numeric Match accuracy.

3. Key Contributions

Oracle-Based Decomposition: Introduced a framework to separate "Document Discovery" failures from "Within-Document Context" failures, providing clear upper bounds for retrieval performance.
Comprehensive Evaluation: Systematically compared dense, sparse, hybrid, and hierarchical retrieval strategies on a shared multi-document index for financial QA.
Novel Page Scorer: Developed and fine-tuned a bi-encoder specifically for page-level relevance in financial filings, treating pages as an intermediate retrieval unit.
Granular Analysis: Demonstrated that retrieval difficulty varies significantly by question type (Metrics vs. Narrative) and document type (10-K vs. Earnings Calls).

4. Key Results

Retrieval Performance

The Retrieval Gap: Standard baselines (e.g., BGE-M3 + Multi-HyDE + Reranker) achieved high Document Recall (0.93) but significantly lower Page Recall (0.46). The Oracle Document setting showed a potential Page Recall of 0.60, indicating a 0.14 gap in finding the right context even when the document is known.
Page Scorer Superiority: The proposed Learned Page Scorer achieved 0.55 Page Recall, significantly outperforming all baselines and narrowing the gap to the Oracle Document bound to just 0.05.
Chunk Quality: The Page Scorer also achieved the highest chunk-level similarity scores (Max BLEU: 0.33, Max ROUGE-L: 0.46), surpassing even the Oracle Document baseline. This confirms that filtering by relevant pages improves the quality of retrieved chunks.

Generative Performance

Numeric Accuracy: The Page Scorer improved Numeric Match accuracy to 0.30, outperforming the best baseline (0.12) and the Oracle Document setting (0.26).
Reasoning: The method showed particular strength on "Metrics Generated" questions (requiring table extraction), where it exceeded the Oracle Document performance, likely due to the structured nature of financial tables in 10-K filings.

Breakdown by Document Type

10-K Reports: The method excelled on 10-K filings (Page Recall 0.62), likely due to their consistent structure and representation in the training data.
Earnings Calls: Performance was weak (Page Recall 0.10) on Earnings Call transcripts, highlighting a limitation in handling less standardized, conversational structures compared to formal reports.

5. Significance and Conclusion

This paper highlights that document-level retrieval is insufficient for long-document financial QA. The primary bottleneck lies in locating the specific page or section within a retrieved document.

Practical Impact: By introducing a domain-specific page scorer, the authors demonstrate that explicitly modeling page-level relevance can close the retrieval gap more effectively than simply improving chunk matching or query reformulation.
Limitations: The study is limited to 150 questions (mostly 10-Ks), and the page scorer requires gold annotations for training. Future work could explore hard negative mining, multi-document aggregation, and applying this to other document types.
Takeaway: For high-stakes financial QA, granular retrieval (Page $\to$ Chunk) combined with domain-adapted modeling is essential to ensure the generator has access to the precise evidence required for accurate, verifiable answers.