💬 NLP

Dissecting Agentic RAG: A Component Ablation for Multi-Hop QA with a Local 7B Model

This paper presents a component ablation study of an Agentic RAG system using a local 7B model on HotpotQA, revealing that fixed hybrid retrieval and limited retrieval iterations outperform complex adaptive routing and deeper loops, demonstrating that simpler, non-adaptive designs are more effective for multi-hop QA under resource constraints.

Original authors: Sheroz Shaikh

Published 2026-06-23✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sheroz Shaikh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex mystery, like figuring out who stole the cookie jar and why. You have a very smart but small detective (a local 7B AI model) and a fixed, closed set of documents provided specifically for that case (a local corpus of supporting and distractor passages).

The paper asks a simple question: How do we get our small detective to solve these multi-step mysteries most effectively without wasting time or compute?

The researchers built a "super-detective" system called Agentic RAG. Instead of just asking the document pile for one page and guessing the answer, this system breaks the big mystery into smaller clues, hunts for evidence within the provided texts, checks if the clues make sense, and then puts the pieces together.

To find the best way to run this system, the researchers didn't just guess; they performed a "component ablation." Think of this as taking apart a high-end car engine bolt by bolt to see which parts actually make the car go faster, and which parts are just heavy, expensive decorations.

Here is what they discovered, using simple analogies:

1. The "Smart Router" vs. The "Fixed Hybrid"

The Setup: The system had a "traffic cop" (an adaptive router) designed to decide which search tool to use for each clue.

If the clue had a name or a date, the cop sent it to the Keyword Search (BM25), which looks for exact word matches.
If the clue was vague, it went to the Semantic Search (Dense), which understands the meaning of words.
Otherwise, it used a Mix of both.

The Surprise: The researchers thought the "Smart Traffic Cop" would be the best. But it turned out the cop was a bit clumsy. Because multi-step mysteries almost always involve names and dates, the cop sent 79% of all clues to the Keyword Search. It ignored the Semantic Search too often.

The Result: When they fired the traffic cop and just used a Fixed Mix (checking both Keyword and Semantic search for every clue), the system got smarter.

Analogy: It's like a chef who decides to use a knife for everything because they see a vegetable on the counter. But sometimes you need a spoon! By using both tools for every dish, the meal turned out better than when the chef tried to be "smart" about which tool to pick.

2. The "Endless Loop" vs. "Two Steps"

The Setup: The system was designed to hunt for clues in a loop. It could theoretically search, find a clue, search again based on that clue, search again, and so on, up to 5 times.

The Surprise: The researchers found that after two searches, the detective had found almost everything they needed. Doing a third, fourth, or fifth search didn't really help.

Analogy: Imagine looking for your keys. You check the kitchen (Step 1). You don't find them, so you check the living room (Step 2). You find them! Checking the garage, the car, and the backyard (Steps 3, 4, 5) is a waste of time. You already have the answer. The paper found that 95% of the success happened in just the first two steps.

3. The "Question Breaker" and the "Editor"

The Setup:

Question Breaker: Before searching, the AI breaks one big hard question into three or four smaller, easier questions.
Editor: After finding 20 potential answers, the AI uses a special "Editor" to pick the best 5.

The Result: Both of these helped, but not as much as the first two factors.

The Breaker: It helped accuracy, but it made the process take twice as long (because the AI had to do extra thinking).
The Editor: It helped accuracy and took almost no extra time.
Analogy: The "Breaker" is like hiring a translator to rewrite a difficult question before asking it. It works well, but it costs time. The "Editor" is like a librarian who quickly scans a stack of books to pick the best one. It's fast and effective, so you should always keep the Editor, but maybe skip the Translator if you are in a huge rush.

The Bottom Line

The paper concludes that for a small, local AI model (running on your own computer, not a giant cloud server), simplicity wins.

Don't overthink the routing: Just use a mix of search tools for everything.
Don't over-search: Two rounds of searching are enough.
Keep the editor: It's cheap and effective.

By stripping away the complex, "adaptive" features that people assumed were necessary, the researchers built a system that was actually better, faster, and more reliable than the complicated version. They proved that for local AI, you don't need a Ferrari engine; a well-tuned bicycle often gets you to the destination first.

Technical Summary: Dissecting Agentic RAG via Component Ablation

Problem Statement
Agentic Retrieval-Augmented Generation (RAG) systems have emerged as a solution for Multi-Hop Question Answering (QA), utilizing iterative reasoning loops, query decomposition, and adaptive retrieval to chain facts across multiple documents. However, the specific contribution of each component remains poorly understood, particularly in resource-constrained environments relying solely on local Large Language Models (LLMs). Many existing designs assume that added complexity—such as adaptive retrieval routing and deep iterative loops—necessarily improves performance. This paper investigates whether these assumptions hold true when using a local 7B parameter model, aiming to isolate the value of individual components without relying on proprietary APIs or large-scale compute.

Methodology
The authors conducted a controlled component-level ablation study on a full agentic RAG pipeline using the Qwen2.5-7B-Instruct model, served locally via Ollama on an NVIDIA RTX A6000. The evaluation was performed on a fixed random sample of 5,000 questions from the HotpotQA distractor development set.

The system architecture consists of four stages:

Query Decomposition: An LLM breaks the input question into a sequence of sub-questions.
Iterative Retrieval Loop: A "plan-and-execute" design where the loop iterates over sub-questions. At each step, a retrieval strategy is selected, passages are retrieved, and a cross-encoder reranker refines the top candidates.
Adaptive Retrieval: A rule-based heuristic routes queries to Dense (BGE-small-en-v1.5), Sparse (BM25), or Hybrid (Reciprocal Rank Fusion) strategies based on the presence of named entities, dates, or numbers.
Synthesis: A final LLM call generates the answer from the accumulated evidence.

The study compared the full pipeline against a single-pass dense retrieval baseline and eight specific ablation conditions:

Depth variations: Retrieval loops capped at 1, 2, or 3 steps (vs. the default 5).
Component removal: Disabling query decomposition (no-decomp) or cross-encoder reranking (no-reranker).
Strategy fixation: Replacing the adaptive router with fixed Dense, Sparse, or Hybrid retrieval.

Statistical significance was assessed using paired tests (McNemar's exact test for Exact Match and Wilcoxon signed-rank test for F1) and 95% bootstrap confidence intervals.

Key Results
The full agentic pipeline achieved an Exact Match (EM) of 53.2% and F1 of 61.6%, significantly outperforming the single-pass baseline (EM 43.1%, F1 54.0%). However, the ablation study revealed counter-intuitive findings regarding component efficacy:

Fixed Hybrid Retrieval > Adaptive Routing: The "hybrid-only" variant (using fixed Reciprocal Rank Fusion for all queries) outperformed the full adaptive pipeline by +1.8 EM and +1.9 F1 ( $p < 0.001$ ). The rule-based router was found to be flawed; its heuristic (triggering on named entities) caused it to route 79.2% of sub-queries to BM25. Since multi-hop sub-questions inherently contain entities, the router over-relied on sparse retrieval, discarding the complementary dense signal.
Diminishing Returns on Iteration Depth: Reducing the retrieval loop from 5 steps to 2 steps captured 95% of the performance gains over the baseline. The difference between 2 steps and 5 steps was statistically negligible ( $\pm 0.0$ EM), while reducing to 1 step caused a significant drop (-7.1 EM). This suggests that the primary value comes from running a loop, not a deep loop.
Significant but Costly Components:
- Query Decomposition: Removing decomposition resulted in a -1.4 EM drop ( $p=0.004$ ) but also halved latency (from ~5.6s to ~2.5s) because it collapsed the multi-step loop into a single retrieval.
- Cross-Encoder Reranking: Removing the reranker caused a -1.7 EM drop ( $p < 0.001$ ) with negligible latency impact, making it a high-value, low-cost component.
Strategy Equivalence: Within the iterative loop, fixed Dense, Sparse, and Hybrid strategies performed nearly identically (EM ~53.0%), suggesting that repeated retrieval across sub-questions compensates for the blind spots of any single strategy.

Significance and Claims
The paper claims that for local 7B models, simplicity and fixed choices are often superior to adaptive complexity. The authors argue that the "agentic" gains in multi-hop QA are driven primarily by the existence of an iterative retrieval loop rather than sophisticated routing heuristics or excessive iteration depth.

Specifically, the study concludes:

Adaptive routing is unnecessary and potentially harmful in this context due to selection bias in the routing heuristics.
Two retrieval iterations are sufficient to capture the vast majority of performance gains, offering a more efficient deployment path.
Fixed hybrid retrieval (combining dense and sparse signals via RRF) is the most robust retrieval strategy for multi-hop QA under these constraints.

The authors emphasize that their findings are empirical, derived from a controlled ablation on a local model, and do not rely on proprietary APIs. They position their work as a guide for practitioners building agentic RAG systems under resource constraints, advocating for fixed hybrid retrieval and shallow iteration depths over complex adaptive architectures.

1. The "Smart Router" vs. The "Fixed Hybrid"

2. The "Endless Loop" vs. "Two Steps"

3. The "Question Breaker" and the "Editor"

The Bottom Line

Technical Summary: Dissecting Agentic RAG via Component Ablation

More like this