💻 computer science

From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents

This paper benchmarks ten retrieval strategies on a large-scale financial text-and-table QA dataset, revealing that a hybrid retrieval pipeline with neural reranking outperforms single-stage methods and that BM25 often surpasses dense retrieval for financial documents, while offering actionable insights on the limited utility of query expansion for precise numerical queries.

Original authors: Meftun Akarsu, Recep Kaan Karaman, Christopher Mierbach

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Meftun Akarsu, Recep Kaan Karaman, Christopher Mierbach

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a complex financial mystery. You have a massive library of documents (annual reports, earnings statements) that are a mix of stories (text) and spreadsheets (tables). Your goal is to find the exact page and the exact number that answers a specific question, like "How much profit did Company X make in 2023?"

This paper is essentially a grand experiment to figure out the best way for a computer (an AI) to search through this library to find that needle in the haystack. The researchers tested 10 different "search strategies" to see which one works best for these tricky documents.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Text vs. Table" Trap

Most AI search tools are great at finding stories. If you ask, "Tell me about the company's history," a standard AI search works like a librarian who knows exactly which book to pull off the shelf based on keywords.

But financial documents are tricky. They contain tables full of numbers.

The Issue: If you ask, "What was the revenue growth?", a standard AI might get confused. It might look for the word "revenue" in the text but miss the number hidden in a spreadsheet cell. Or, it might understand the concept of growth but miss the specific year because the wording was slightly different.
The Analogy: It's like trying to find a specific ingredient in a recipe. A "keyword search" finds the word "sugar," but a "semantic search" (understanding meaning) might find "sweetener." But in finance, you need the exact gram amount. If you get the wrong number, the whole recipe fails.

2. The Contenders: Who Did They Test?

The researchers pitted different search strategies against each other:

The Old School (BM25): This is like a very strict librarian who only finds books if you use the exact words written on the spine. It doesn't understand synonyms, but it's incredibly good at finding specific terms like "Company Name" or "2023."
The Modern AI (Dense Retrieval): This is like a smart assistant who understands the vibe and meaning of your question. It can find documents even if you use different words, but it sometimes gets fuzzy on exact numbers.
The Hybrid (RRF): This is hiring both the strict librarian and the smart assistant, then letting them vote on which book is best.
The "Guess the Answer" (HyDE): This strategy asks the AI to pretend it already knows the answer, write a fake document, and then search for that. (Think of it like a detective drawing a sketch of a suspect before finding them).
The "Second Opinion" (Reranking): This is a senior editor who looks at the top 50 books the search found and re-sorts them to make sure the absolute best one is at the top.

3. The Big Surprises (The Results)

🏆 The Winner: The "Two-Stage Pipeline"
The absolute best strategy wasn't just one tool; it was a teamwork approach.

Step 1: Use the Hybrid method (Librarian + Smart Assistant) to cast a wide net and grab the top 50 most likely documents.
Step 2: Pass those 50 documents to a Super-Editor (a "Cross-Encoder Reranker") who reads the question and the document together to pick the single best match.

Why it won: It combined the speed of casting a wide net with the precision of a deep dive. It found the right document 81.6% of the time, beating everyone else by a huge margin.

🚫 The Loser: "Guess the Answer" (HyDE)
The researchers tried the strategy where the AI guesses the answer first.

The Result: It actually made things worse.
The Analogy: Imagine a detective drawing a sketch of a suspect based on a vague description. In finance, if the AI "hallucinates" (guesses) a number like "$50 million" when the real number is "$48 million," the search gets confused and looks in the wrong place. For precise numbers, guessing is dangerous.

🤔 The Surprise: The Old School Librarian Won (Sometimes)
Everyone thought the "Smart Assistant" (Dense AI) would beat the "Strict Librarian" (BM25) because AI is supposed to be smarter.

The Result: On financial documents, the Strict Librarian (BM25) actually beat the Smart Assistant.
Why? Financial reports are full of specific jargon (ticker symbols, exact metric names). The AI sometimes gets too "chatty" and misses the exact match, while the librarian just matches the words perfectly.

4. What Should You Do? (The Takeaway)

If you are building an AI system to read financial reports or documents with tables, here is the recipe the authors suggest:

Don't rely on just one search method. Combine the "keyword" search with the "meaning" search.
Add a "Second Opinion" step. Always have a powerful AI re-check the top results to ensure the best one is on top. This is the single biggest upgrade you can make.
Add "Context" before you search. Before indexing the documents, have an AI write a short summary of what the document is about (e.g., "This is Apple's 2023 report") and attach it to the file. This helps the search engine understand the document better.
Stop guessing. Do not use strategies that try to "hallucinate" an answer to help the search. In finance, precision is king.
Test on your own data. Don't just trust general AI benchmarks. Financial documents are unique; what works for news articles might fail for spreadsheets.

Summary

The paper proves that for complex documents mixing text and numbers, brute force isn't enough, and pure AI intuition isn't enough either. The secret sauce is hybrid teamwork: cast a wide net, then have a super-smart editor pick the winner. And remember, when dealing with money, exact words matter more than vague meanings.

1. Problem Statement

Retrieval-Augmented Generation (RAG) systems rely heavily on the quality of the retrieval component. However, there is a lack of systematic comparison among modern retrieval methods for heterogeneous documents containing both text and tabular data (e.g., financial filings, earnings reports).

The Challenge: Financial questions often require locating precise numerical values within tables while understanding textual context.
- Semantic (Dense) Retrieval: Often fails to capture exact numerical targets or specific entity names.
- Lexical (Sparse) Retrieval: Misses paraphrased context or semantic relationships.
The Gap: Existing benchmarks (like BEIR) focus on general text, and previous financial QA benchmarks (like FinQA) often used "oracle-context" settings where the correct document was provided, failing to test actual retrieval capabilities.

2. Methodology

A. Dataset: T2-RAGBench

The authors utilize T2-RAGBench, a financial QA benchmark comprising:

Scale: 23,088 queries over 7,318 unique financial documents.
Sources: Aggregated from FinQA, ConvFinQA, and TAT-DQA.
Content: Mixed text and Markdown-formatted tables extracted from SEC filings and annual reports (avg. 920 tokens/doc).
Reformulation: Unlike original datasets, questions were reformulated using Llama 3.3-70B to include identifying information (company name, sector, year), ensuring a single correct answer regardless of context. This makes the dataset suitable for retrieval evaluation.
Ground Truth: All answers are numerical.

B. Retrieval Strategies Evaluated

The study benchmarks 10 distinct strategies across four categories:

Sparse Retrieval: BM25 (Okapi, $k_1=1.2, b=0.75$ ).
Dense Retrieval: OpenAI text-embedding-3-large (3,072 dims) with FAISS exact search.
Hybrid Fusion: Reciprocal Rank Fusion (RRF) combining BM25 and Dense.
Two-Stage Pipelines: Hybrid RRF followed by a Cross-Encoder reranker (Cohere Rerank v4.0 Pro).
Query Expansion:
- HyDE: Generating hypothetical answer documents to query the index.
- Multi-Query: Generating 3 semantic variants per query and merging results via RRF.
Index Augmentation:
- Contextual Retrieval: Prepending LLM-generated summaries (entities, metrics, periods) to documents before indexing.
Adaptive Retrieval: CRAG, which classifies retrieved documents and triggers query rewriting if confidence is low.

C. Evaluation Metrics

Retrieval Quality: Recall@k ( $k \in \{1, 3, 5, 10, 20\}$ ), Mean Reciprocal Rank (MRR@3), nDCG@10, MAP.
Generation Quality: Number Match (NM) (primary metric, checking if the generated number matches the ground truth within 1% tolerance), plus Token-level F1, ROUGE-L, and BERTScore.
Statistical Significance: Paired bootstrap tests ( $B=10,000$ ) with Bonferroni correction ( $p < 0.05$ ).

3. Key Contributions

Comprehensive Benchmark: The first systematic evaluation of 10 retrieval methods on a large-scale, mixed text-and-table financial corpus.
Multi-Dimensional Analysis: Combines retrieval metrics with end-to-end generation metrics and statistical significance testing.
Ablation Studies: Isolates the impact of fusion strategies (RRF vs. Convex Combination) and reranker candidate depth.
Actionable Guidelines: Provides cost-accuracy recommendations for practitioners building RAG systems for heterogeneous documents.

4. Key Results

A. Performance Rankings

Best Overall: The Two-Stage Pipeline (Hybrid RRF + Neural Reranker) dominates all other methods.
- Recall@5: 0.816 (vs. 0.695 for Hybrid RRF alone).
- MRR@3: 0.605 (vs. 0.433 for Hybrid RRF alone).
- This represents a massive improvement over single-stage methods.
BM25 vs. Dense: Surprisingly, BM25 outperformed the state-of-the-art dense retriever (text-embedding-3-large) on almost all metrics (except Recall@20).
- Reasoning: Financial documents rely on precise terminology (ticker symbols, metric labels) where lexical matching is superior to semantic embeddings.
Query Expansion (HyDE/Multi-Query): Provided limited or negative benefits.
- HyDE underperformed vanilla dense retrieval (Recall@5: 0.544 vs. 0.587) because LLMs hallucinate plausible but incorrect financial figures, corrupting the embedding space.
- Multi-query showed diminishing returns as financial queries are already specific.
Contextual Retrieval: Yielded consistent gains (+2.2pp to +2.8pp Recall@5) by enriching documents with metadata summaries at indexing time.
CRAG: Provided moderate gains over BM25 but failed to match the performance of simple Hybrid Fusion.

B. End-to-End Generation

There is a strong positive correlation ( $r > 0.98$ ) between retrieval quality (Recall@5) and generation quality (Number Match).
Better retrieval directly translates to better answers, confirming that retrieval is the bottleneck in RAG pipelines for this domain.

C. Error Analysis

Dominant Failure Mode: Table Structure Mismatch (73%). Standard embeddings struggle to map queries like "Net Income 2019" to tabular rows where these terms are in separate cells.
Numerical Reasoning (20%): Failures occur when the answer requires computation (e.g., YoY growth) rather than direct lookup.
Subset Difficulty: TAT-DQA (heavy on table operations) was the hardest subset, while ConvFinQA was the easiest.

5. Significance and Recommendations

Significance

Challenges Assumptions: The finding that BM25 outperforms dense retrieval in financial domains challenges the prevailing assumption that semantic search universally dominates sparse methods.
Domain Specificity: Highlights that general-purpose benchmarks (MTEB/BEIR) do not predict performance on structured, numerical, domain-specific data.
Architecture Validation: Validates the "Hybrid Retrieval + Cross-Encoder Reranking" pipeline as the gold standard for heterogeneous document retrieval.

Practical Recommendations for Practitioners

Baseline: Start with Hybrid Retrieval (BM25 + Dense via RRF).
Maximize Quality: Add a Cross-Encoder Reranker (e.g., Cohere Rerank) to the top 50 candidates. This is the single most impactful component.
Indexing: Use Contextual Retrieval (prepending summaries) for consistent, one-time cost gains.
Avoid: Do not use HyDE for domains requiring precise numerical facts, as hallucination degrades performance.
Evaluation: Always evaluate on domain-specific data; general benchmarks are insufficient.

Conclusion

The paper establishes that for text-and-table financial documents, a two-stage pipeline combining hybrid retrieval (BM25 + Dense) with neural reranking is the optimal strategy. It demonstrates that lexical precision remains critical for financial data and that query expansion techniques like HyDE can be counterproductive in this specific domain. The authors release their code and benchmark to facilitate future research in heterogeneous RAG systems.