💬 NLP

CompactRAG: Reducing LLM Calls and Token Overhead in Multi-Hop Question Answering

CompactRAG is a cost-efficient framework for multi-hop question answering that minimizes LLM calls and token overhead by decoupling offline corpus restructuring into an atomic QA knowledge base from an online reasoning stage that relies on dense retrieval and answer extraction, invoking the LLM only twice regardless of reasoning complexity.

Original authors: Hao Yang, Zhiyu Yang, Xupeng Zhang, Wei Wei, Yunjie Zhang, Lin Yang

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Yang, Zhiyu Yang, Xupeng Zhang, Wei Wei, Yunjie Zhang, Lin Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex mystery, like figuring out who the director of a specific movie is, but the answer isn't in one place. You have to read a book about the movie, find the director's name, then look up that director's biography to find where they were born, and finally check a map to see the city.

This is what Multi-Hop Question Answering is: solving a puzzle that requires jumping between different pieces of information (hops) to get the final answer.

The Problem: The "Over-Worked Librarian"

Current systems (called RAG or Retrieval-Augmented Generation) try to solve this by asking a super-smart AI (a Large Language Model or LLM) to do the work. However, the way they do it is inefficient.

Think of the current method like hiring a very expensive, high-paid librarian to solve your mystery.

You ask the librarian a question.
The librarian runs to the shelves, grabs a book, reads it, and writes a note.
The librarian runs back to you, reads your note, and says, "Okay, now I need to know where the director was born."
You ask again. The librarian runs to the shelves again, grabs a different book, reads it, and writes another note.
They repeat this process for every single step of the mystery.

The result? The librarian gets tired, the process takes forever, and you have to pay a huge bill (in "tokens" or computing power) because the librarian is making so many trips back and forth. Also, sometimes the librarian gets confused about who "he" refers to in the second step, leading to wrong answers.

The Solution: CompactRAG (The "Pre-Packaged Knowledge Box")

The authors of this paper, CompactRAG, propose a smarter way. Instead of making the expensive librarian run around the library every time you ask a question, they reorganize the library before you ever show up.

Step 1: The Offline Prep (The "One-Time Setup")

Before any user asks a question, the system uses an AI to read the entire library of documents once.

It breaks every document down into tiny, perfect, self-contained "fact cards."
Instead of a whole paragraph saying, "The movie was made in 1953 by Arthur Crabtree," it creates a specific card that says: "Question: Who directed 'The Wedding of Lilli Marlene'? Answer: Arthur Crabtree."
It does this for every fact in the library. This creates a Compact Knowledge Base.

Analogy: Imagine instead of having a messy library, you have a giant, perfectly organized box of index cards. Every card has a specific question on the front and the exact answer on the back. No fluff, no extra words.

Step 2: The Online Reasoning (The "Two-Trip Rule")

Now, when a user asks a complex question, the system works like this:

The Breakdown (Trip 1): The expensive librarian (LLM) is called once just to break the big mystery into small, simple steps.
- User: "Where was the director of the movie born?"
- LLM: "Okay, Step 1: Who directed the movie? Step 2: Where was that person born?"
The Search (No Librarian Needed): The system doesn't call the expensive librarian again. Instead, it uses a cheap, fast robot to look up the answers in the pre-made "fact card" box.
- The robot finds the card for "Who directed..." and gets "Arthur Crabtree."
- The robot then rewrites the next question to be clear: "Where was Arthur Crabtree born?" (This prevents the "he" confusion).
- The robot finds the card for "Where was Arthur Crabtree born?" and gets "London."
The Final Answer (Trip 2): Once the robot has all the little answers, the expensive librarian is called one last time to put the pieces together and give you the final answer.

The Magic: The expensive librarian is only called twice per question, no matter how many steps (hops) the mystery has. Whether the puzzle has 2 steps or 10 steps, the cost stays the same.

Why This Matters

Saves Money: You stop paying the expensive librarian to run back and forth. You only pay them twice.
Saves Time: The process is much faster because the "fact cards" are easy to find and read.
Fewer Mistakes: By rewriting the questions to include specific names (like "Arthur Crabtree" instead of "he"), the system doesn't get confused about who it's talking about.

The Results

The paper tested this on three difficult puzzle datasets (HotpotQA, 2WikiMultiHopQA, and MuSiQue).

Accuracy: CompactRAG was just as good at solving the puzzles as the old, expensive methods.
Efficiency: It used significantly fewer "tokens" (the currency of AI computing). In some cases, it used less than half the resources of the other methods.

Summary

CompactRAG is like turning a chaotic, expensive, back-and-forth investigation into a streamlined, pre-packaged operation. It does the heavy lifting of organizing the knowledge once upfront, so that solving any new mystery becomes a quick, cheap, and two-step process.

Technical Summary: CompactRAG

Problem Statement

Retrieval-Augmented Generation (RAG) has become a standard paradigm for knowledge-intensive question answering. However, existing multi-hop RAG systems face significant inefficiencies when handling complex queries that require integrating evidence from multiple documents. Current iterative approaches (e.g., Self-Ask, IRCoT, Iter-RetGen) alternate between retrieval and LLM reasoning at every step. This design leads to three primary issues:

Scalability and Cost: Token consumption and latency grow linearly with the number of reasoning hops due to repeated LLM invocations.
Redundancy: Retrieved contexts often contain redundant information, inflating token usage.
Entity Drift: Automatic decomposition of complex questions into sub-questions can lose explicit entity mentions (e.g., replacing "Where was he born?" after identifying "he" as a scientist). This ambiguity causes retrieval errors and inconsistent results across hops.

Methodology: CompactRAG

The authors propose CompactRAG, a framework that decouples offline corpus restructuring from online reasoning to achieve efficient multi-hop inference. The system operates in two distinct stages:

1. Offline Stage: Atomic QA Knowledge Base Construction

Instead of retrieving raw corpus passages during inference, CompactRAG processes the corpus once offline.

Process: An LLM reads the raw corpus and reformulates document content into a set of atomic QA pairs.
Granularity: Each pair represents a minimal, fine-grained factual unit (e.g., "Who directed X?" $\to$ "Y").
Entity Anchoring: Entities within the corpus are automatically annotated (using SpaCy) and explicitly enforced during generation to ensure semantic completeness and prevent referential loss.
Indexing: Each atomic QA pair is embedded into a shared semantic space using dense retrieval (Contriever). The question and answer are concatenated ( $[q; a]$ ) to preserve full factual scope, enabling the retriever to match both intent and content.

2. Online Stage: Two-Call Inference

During inference, the system resolves complex queries using a fixed, lightweight pipeline that invokes the main LLM only twice, regardless of the reasoning depth.

Step 1: Decomposition (LLM Call #1): The complex query is decomposed into a dependency-ordered graph of sub-questions.
Step 2: Iterative Resolution (Lightweight Modules):
- Retrieval: Sub-questions are used to retrieve top- $k$ atomic QA pairs from the pre-built knowledge base.
- Answer Extraction: A lightweight RoBERTa-based Answer Extractor identifies the correct answer span from retrieved pairs, avoiding LLM calls for extraction.
- Sub-Question Rewriting: A Flan-T5-based Sub-Question Rewriter takes the extracted answer entity and rewrites the next sub-question to explicitly ground ambiguous references (e.g., changing "Where was he born?" to "Where was [Entity] born?"). This prevents entity drift.
Step 3: Synthesis (LLM Call #2): Once all sub-questions are resolved and evidence aggregated, the main LLM performs a single synthesis step to generate the final answer.

Key Contributions

Scalability Analysis: The paper analyzes how token consumption and LLM calls in iterative RAG pipelines scale linearly with reasoning depth, identifying this as a bottleneck for large-scale deployment.
CompactRAG Framework: The introduction of a two-call RAG architecture that utilizes an offline atomic QA knowledge base and lightweight online modules (extractor and rewriter) to enable efficient multi-hop reasoning without repeated LLM invocations.
Empirical Validation: Comprehensive evaluation on HotpotQA, 2WikiMultiHopQA, and MuSiQue demonstrating that CompactRAG achieves competitive accuracy while drastically reducing token consumption compared to strong iterative baselines.

Experimental Results

Evaluated on three multi-hop benchmarks using LLaMA-3.1-8B as the backbone LLM:

Accuracy: CompactRAG achieves competitive performance. On HotpotQA, it reached an Exact Match (EM) of 45.20% and an LLM-based Accuracy of 70.40%, comparable to or exceeding iterative baselines like IRCoT (42.80% EM) and Iter-RetGen (46.80% EM).
Efficiency: The system significantly reduces token overhead.
- CompactRAG: ~1.9K tokens per sample.
- Iter-RetGen: ~4.7K tokens.
- IRCoT: ~10.2K tokens.
- Self-Ask: ~6.9K tokens.
Ablation Studies: Removing the Sub-Question Rewriter or Answer Extractor modules resulted in consistent accuracy drops (e.g., removing the rewriter on HotpotQA dropped accuracy from 70.4% to 63.2%), confirming their necessity for maintaining entity grounding and retrieval precision.
Upper Bound: Using GPT-4 for the offline QA construction further improved accuracy (e.g., 49.60% EM on HotpotQA), suggesting that the quality of the offline knowledge base directly influences inference performance.

Significance and Claims

The paper posits that CompactRAG offers a practical, cost-efficient alternative to iterative RAG systems. Its primary significance lies in:

Fixed-Cost Inference: By limiting LLM calls to two per query (decomposition and synthesis), the system decouples inference cost from reasoning depth, making it highly scalable for deep multi-hop tasks.
Decoupling Retrieval and Reasoning: The framework demonstrates that high-quality multi-hop reasoning does not strictly require repeated LLM interactions. Instead, structuring external knowledge into semantically aligned, atomic units and leveraging lightweight local modules can yield comparable accuracy with significantly lower computational overhead.
Deployability: Unlike methods relying on internal model signals (e.g., attention entropy) which are unavailable in closed-weight models, CompactRAG's architectural approach is compatible with both open and closed LLMs.

The authors conclude that this approach provides a blueprint for developing scalable, token-efficient RAG systems suitable for large-scale, knowledge-intensive applications.