Enhancing Financial Report Question-Answering: A Retrieval-Augmented Generation System with Reranking Analysis

1. Problem Statement

Financial analysts face significant challenges in extracting precise information from lengthy 10-K reports (annual filings), which often exceed 100–300 pages and contain dense prose, complex tables, and regulatory jargon. While Retrieval-Augmented Generation (RAG) systems offer a solution by combining information retrieval with Large Language Models (LLMs), achieving high accuracy in the financial domain remains difficult.

The core issue is that standard retrieval methods (keyword-based or semantic similarity) often return irrelevant or marginally relevant passages. These "noisy" contexts mislead the generation model, leading to hallucinations or incorrect answers. The paper specifically investigates the underexplored impact of neural reranking on RAG performance within the financial domain, asking: How much does neural reranking contribute to the accuracy of financial question-answering?

2. Methodology

The authors developed an end-to-end RAG pipeline designed to process 10-K reports from all 500 S&P 500 companies. The system architecture consists of two main phases:

A. Document Processing (Offline)

• Data Ingestion: 10-K reports (HTML) are converted to PDFs using Playwright to preserve formatting, then parsed via PyMuPDF.
• Chunking: Text is split into fixed-size chunks (2,500 characters) with a 50% overlap (1,250 characters) to ensure context continuity across boundaries.
• Indexing:
  • Keyword Search: Chunks are stored in a SQLite database with FTS5 extension for full-text search.
  • Semantic Search: Chunks are embedded using OpenAI's text-embedding-3-small (1,024 dimensions) and indexed in FAISS for approximate nearest neighbor search.

B. Query Processing (Online)

1. Query Rewriting: An LLM (GPT-4.1) clarifies ambiguous user queries, corrects typos, and extracts keywords for dual retrieval paths.
2. Hybrid Retrieval:
   • Full-Text Search (FTS): Retrieves top-20 chunks via keyword matching.
   • Semantic Search: Retrieves top-30 chunks via vector similarity.
   • Fusion: Results are combined using Reciprocal Rank Fusion (RRF) to leverage the strengths of both methods.
3. Neural Reranking (The Core Innovation):
   • The fused candidate list (up to 30 chunks) is re-scored using Jina Reranker v2 (a 278M-parameter cross-encoder).
   • Adaptive Cutoff: The system employs two strategies to filter noise:
     • Cumulative Probability Threshold: Stops when remaining probability < 45%.
     • Score Cliff Detection: Truncates if the score drops >0.15 from the top.
   • This stage typically selects 5–10 high-quality chunks.
4. Answer Generation: The final context is fed to GPT-4.1 with a system prompt enforcing strict adherence to the provided text.

C. Experimental Setup

• Dataset: The FinDER benchmark, specifically designed for financial QA, containing 5,703 query-evidence-answer triplets.
• Sampling: 1,500 queries were sampled into five independent groups (300 each) to ensure statistical robustness.
• Ablation Study: The system was evaluated in two configurations:
  1. With Reranking: Full pipeline including the neural reranker.
  2. Without Reranking: RRF fusion followed by a simple Top-10 selection (baseline).
• Evaluation: An LLM-as-a-judge (GPT-4.1) scored answers on a 1–10 scale (1 = completely incorrect, 10 = fully correct).

3. Key Contributions

1. Complete RAG Pipeline: Implementation of a production-ready system processing 500 S&P 500 companies' 10-K reports using hybrid retrieval.
2. Systematic Benchmarking: Rigorous evaluation using 1,500 queries across five independent groups to minimize sampling bias.
3. Empirical Evidence on Reranking: A controlled ablation study proving that neural reranking is a critical component for financial RAG, distinct from improvements due to better LLMs or retrieval alone.
4. Performance Baselines: Established new performance benchmarks (49% correctness) for financial QA, significantly outperforming prior work (33%).

4. Key Results

The study demonstrates that neural reranking provides substantial, statistically significant improvements:

Metric	Without Reranking	With Reranking	Improvement
Average Score (1-10)	4.95	6.02	+1.07 (+21.6%)
Correctness (Score ≥ 8)	33.5%	49.0%	+15.5 pp
Critical Errors (Score = 1)	35.3%	22.5%	-12.8 pp
Perfect Answers (Score = 10)	10.2%	13.8%	+3.6 pp

• Consistency: The improvements were consistent across all five experimental groups (average score gains ranged from 1.00 to 1.21 points).
• Error Reduction: The rate of completely incorrect answers dropped by 36.3% relative to the baseline.
• Attribution: The system without reranking achieved ~33.5% correctness, matching prior baselines. This confirms that the jump to 49% is primarily driven by the neural reranking stage, rather than just the use of modern LLMs or hybrid retrieval alone.

5. Significance and Conclusion

This paper establishes that neural reranking is not merely an optional optimization but a critical necessity for high-accuracy financial RAG systems.

• Why it works: Cross-encoder models (rerankers) capture fine-grained semantic interactions between queries and documents that bi-encoders (standard semantic search) miss. They effectively filter out "superficial matches" that pass keyword or vector similarity checks but lack substantive relevance.
• Impact: By reducing noise in the context window, reranking significantly lowers hallucination rates and improves factual accuracy, which is paramount in financial decision-making.
• Future Directions: The authors suggest exploring graph-based retrieval for multi-hop reasoning, integrating multi-modal data (tables/charts), and optimizing reranking costs for production deployment.

In summary, the paper provides a robust technical blueprint for building reliable financial QA systems, demonstrating that combining hybrid retrieval with neural reranking and modern LLMs can nearly double the rate of "basically correct" answers compared to baseline methods.