💻 computer science

Hierarchical Reranking for Scalable Financial RAG System

This paper introduces Hierarchical Reranker, a scalable RAG framework that integrates pre-retrieval optimization, a two-stage ranking mechanism, and long-context management to significantly enhance retrieval precision and factual consistency for complex financial document analysis, as evidenced by its top performance in the ACM-ICAIF '24 FinanceRAG Challenge.

Original authors: Joohyun Lee, Sungwoo Hong

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Joohyun Lee, Sungwoo Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a single crime scene, you are handed a library containing millions of books, each filled with thousands of pages of tiny, confusing text and complex charts. This is the daily reality for financial analysts who need to read documents like 10-K filings (huge reports companies file with the government) to find specific numbers or facts. In the world of Artificial Intelligence, there's a tool called RAG (Retrieval-Augmented Generation) that acts like a super-smart librarian. It doesn't just read the books; it searches them to find the exact pages you need and then writes a summary based only on what it found. However, standard librarians often get overwhelmed by the sheer size of these financial libraries, get confused by the mix of words and numbers, or make up facts when the documents are too long. This paper tackles the problem of how to build a librarian that is fast, accurate, and doesn't get lost in the details when the job gets huge.

The authors of this paper, Joohyun Lee and Sungwoo Hong, propose a new system called the Hierarchical Reranker. Think of this system as a two-step hiring process for a team of detectives. Instead of hiring one giant, expensive, and slow detective to read every single page of a 100,000-page book, this system uses a "small-and-large" team strategy. First, a fast, lightweight detective (a small AI model) quickly scans the entire library and throws out the pages that are obviously irrelevant, narrowing the pile down to the top 100 most promising pages. Then, a highly skilled, expert detective (a large, powerful AI model) takes a deep, careful look at just those 100 pages to pick the final top 20 that truly answer the question. This approach saves time and money because the expensive expert only does the hard work on the most likely candidates, rather than wasting energy on the whole library.

But finding the right pages is only half the battle. The paper also introduces a clever way to handle the documents before the detectives even start looking. Financial documents are messy; they use abbreviations like "YoY" (Year-over-Year) or "EPS" (Earnings per Share), and they mix paragraphs of text with giant tables of numbers. The system's first step is to "clean" the query and the documents. It turns those confusing abbreviations into full words, standardizes units (so "1M" and "1,000,000" look the same), and, crucially, converts those messy tables into a structured format called JSON. This is like taking a handwritten spreadsheet and turning it into a digital database so the AI doesn't get confused about which number belongs to which category. By doing this "pre-cleaning," the system ensures the AI understands the question and the evidence perfectly before it even starts searching.

The third major trick the authors use is a strategy for handling documents that are simply too long to fit in the AI's memory at once. Even though modern AI models claim they can read huge amounts of text, the paper found that when the text gets past a certain point (specifically 64,000 tokens, which is a measure of text length), the AI starts to make mistakes, especially with numbers. To fix this, the system acts like a chef preparing a massive banquet. Instead of trying to eat the whole meal at once, it chops the long document into two smaller, manageable chunks. It asks the AI to analyze the first chunk and write a draft answer, then analyzes the second chunk and writes another draft. Finally, a "fusion" step combines these two drafts into one final answer, checking to make sure they agree and resolving any conflicts. This ensures that even if the document is a massive 100,000-page report, the AI doesn't lose track of the facts.

When the researchers tested their system on several famous financial benchmarks (like FinQA and FinanceBench), the results were impressive. The system achieved a score of 0.7918 on a metric called NDCG@20, which measures how well the system ranks the correct information at the top of its list. This score was significantly better than systems that didn't use their special cleaning and two-step ranking methods. In fact, this approach was so effective that it helped the team secure second place in the ACM-ICAIF '24 FinanceRAG Challenge, a competition designed to test how well AI can handle financial data. The paper suggests that by combining these three innovations—cleaning the data, using a two-step detective team, and smartly chopping up long documents—financial institutions can now build AI systems that are not only smart but also reliable enough to be used for real-world tasks like auditing and investment analysis, without the AI making up facts or getting lost in the noise.

Technical Summary: Hierarchical Reranking for Scalable Financial RAG System

Problem Statement

The financial industry faces significant challenges in automating the analysis of complex documents such as 10-K filings, tabular disclosures, and macroeconomic reports. These tasks require expert reasoning and exhaustive cross-referencing across long, heterogeneous documents. While Retrieval-Augmented Generation (RAG) offers a pathway to ground Large Language Models (LLMs) in underlying evidence, off-the-shelf RAG pipelines fail in financial domains due to three specific modes of failure:

Embedding Drift: Domain-specific jargon, units, and abbreviations cause semantic mismatches between queries and corpus embeddings, degrading recall.
Hybrid Structure Misalignment: Retriever models trained on prose struggle to align hybrid text–table evidence, leading to numerically incorrect generations.
Long-Context Degradation: Reasoning quality collapses on inputs exceeding standard context windows (e.g., 64k tokens), even in models nominally supporting 100k+ tokens, particularly when dense numerical references are involved.

Methodology

The authors propose the Hierarchical Reranker, a domain-optimized RAG framework designed to address these failures through a three-stage pipeline: Pre-Retrieval Optimization, Hierarchical Reranking, and Long-Context Management.

1. Pre-Retrieval Optimization

This phase focuses on normalizing queries and the corpus to enhance semantic coherence before embedding.

Normalization: Queries and documents undergo lowercase conversion, typo correction, and the expansion of financial abbreviations (e.g., "EPS" to "Earnings per share"). Units like "K" or "M" are standardized to consistent numeric expressions.
Keyword Extraction: Domain-specific keywords are appended to queries to increase terminology density, improving alignment with the corpus.
Paraphrasing: Queries are semantically expanded into multiple linguistically diverse variants to broaden the search space.
Hypothetical Document Generation (HyDE): Synthetic pseudo-documents are generated to act as semantic anchors, bridging the gap between abstract queries and specific domain contexts.
Table-to-JSON Conversion: Crucially, Markdown tables are deterministically converted to JSON structures using rule-based scripts (not LLMs) to preserve the strict relationship between numerical values and their headers, preventing hallucinations during retrieval.
Summarization: For extremely large corpora (>10k tokens), documents are summarized to retain only essential financial indicators and results, removing unnecessary narrative.

2. Hierarchical Reranker Architecture

To balance precision with inference cost, the system employs a two-stage cascade rather than a single monolithic retriever.

Stage 1 (Lightweight Filtering): A fast, low-complexity model (jina-reranker-v3) processes the full candidate pool to prune low-relevance candidates, narrowing the pool to the top 100 passages. This stage leverages extended context support (up to 131k tokens) for lexical and shallow semantic similarity.
Stage 2 (Fine-Grained Reranking): A high-capacity model (Qwen3-Reranker-8B) performs deep semantic adjudication on the top 100 survivors to extract the final top 20 passages. This stage focuses on cross-sentence financial dependencies and numerical consistency between tables and text.

3. Long-Context Management

Recognizing that performance degrades beyond 64k tokens, the system implements an adaptive partitioning and fusion strategy for inputs exceeding this threshold.

Partitioning: The top 20 retrieved corpora are split into two semantically coherent subsets.
Fusion: Intermediate answers ( $R_1, R_2$ $R_{1}, R_{2}$ ) are generated for each subset. A conditional fusion mechanism merges these outputs:
- If only one subset yields a definitive answer, it is adopted.
- If both yield answers, the one with higher confidence is selected.
- If neither is clear, an explicit "unknown" response is generated.
  This approach prevents hallucinated synthesis across partitions while maintaining reasoning consistency.

Key Contributions

The paper outlines three primary contributions:

Pre-Retrieval Optimization: A deterministic pipeline that normalizes units, expands keywords, and converts tables to JSON, yielding measurable retrieval gains without introducing LLM-induced hallucinations during preprocessing.
Hierarchical Reranker: A two-stage architecture that pairs a fast filter with a high-capacity adjudicator, capturing complex financial dependencies while bounding inference costs.
Long-Context Management: An evidence-aware fusion mechanism that handles ambiguous or contradictory partial answers in inputs exceeding 64k tokens.

Results

The system was evaluated on multiple financial benchmarks, including FinQABench, FinanceBench, FinQA, ConvFinQA, and TATQA.

Retrieval Performance: The full framework achieved an NDCG@20 score of 0.7918.
- Pre-retrieval optimization alone improved the baseline by +5.9%.
- The hierarchical reranking architecture improved relevance by +6.5% compared to a single-reranker setup.
Generation Performance: The system demonstrated superior factual consistency. In the ACM-ICAIF '24 FinanceRAG Challenge, the framework secured 2nd place.
Ablation Studies: Experiments confirmed that the combination of Query Normalization, Keyword Extraction, and Table-to-JSON conversion provided the highest retrieval gains. The split-and-fusion mechanism for long contexts provided marginal but consistent accuracy improvements (0.08% to 0.49%) across various LLMs.

Significance and Claims

The authors position this work as a bridge between research-grade RAG and institutional-grade deployment. They claim that the framework is deployable, scalable, and domain-optimized, addressing the specific bottlenecks of financial reasoning:

Robustness: The system maintains accuracy in environments with dense numerical data and hybrid text-table structures.
Efficiency: By decomposing the problem into cooperating components (small-and-large model collaboration), the system keeps end-to-end latency and token costs bounded, a property deemed essential for institutional scale.
Practicality: The framework has been validated in real-world auditing and investment workflows. The authors argue that careful engineering—specifically deterministic preprocessing and adaptive context management—is as critical as model scale for bringing RAG into production.

The paper concludes that this pipeline enhances both the accuracy and scalability of financial reasoning, paving the way for automated audit reporting and quantitative investment analysis. The source code is intended for public release upon acceptance.