Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval

Technical Summary: Agent-Orchestrated Adaptive RAG

Problem Statement
Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding responses in external knowledge, yet conventional "Naive RAG" pipelines rely on static, single-step retrieval. This approach struggles with complex queries requiring reasoning across multiple documents or chaining intermediate results. While Agentic RAG systems propose dynamic solutions—such as query decomposition, iterative retrieval, and self-reflection—existing literature often evaluates these mechanisms on single domains or benchmarks. Consequently, it remains unclear how agentic enhancements perform across domains with differing structural characteristics and reasoning demands, particularly regarding the trade-offs between retrieval precision, coverage, and latency.

Methodology
The authors propose and evaluate an Agent-Orchestrated Adaptive RAG framework built on a fully local, privacy-first inference stack using Llama-3.1-8B-Instruct. The system architecture departs from fixed pipelines by introducing a central Orchestrator that dynamically routes queries based on a rule-based control logic. The system comprises four specialized agents:

1. Query Classifier: Categorizes requests to enable metadata-aware filtering (e.g., restricting retrieval to specific document types like runbooks) before similarity ranking.
2. Query Decomposer: Breaks complex, multi-hop queries into ordered sub-queries, retrieves evidence for each, and aggregates results.
3. Answer Evaluator: Critiques generated responses for relevance, citation accuracy, grounding, and hallucination.
4. Orchestrator: Manages the control flow, triggering decomposition for complex queries and initiating a bounded (max two iterations) self-reflective loop if the evaluator flags issues.

The evaluation utilizes two contrasting datasets:

• DevOps Knowledge Base: A structured, domain-specific corpus of 80 documents (approx. 10,000 words) covering standards, architecture, runbooks, and incident reports. Documents are preprocessed using IBM Docling to preserve structural hierarchy and enriched with YAML frontmatter for attribute-based filtering.
• MuSiQue: A compositional multi-hop reasoning benchmark designed to require evidence integration across independent documents.

Performance is measured using Overall Score, Critical Source Recall, Mean Reciprocal Rank (MRR), Success@5, Citation Accuracy, Topic Coverage, and End-to-End Latency.

Key Contributions

1. System Implementation: A complete, locally hosted Agent-Orchestrated Adaptive RAG system featuring dynamic routing, decomposition, and bounded self-correction.
2. Dual-Dataset Evaluation: A comparative study contrasting a structured, domain-specific environment (DevOps) against an open-domain multi-hop benchmark (MuSiQue) to isolate agentic behavior under different reasoning loads.
3. Empirical Trade-off Characterization: An analysis demonstrating that agentic enhancements are not universally beneficial; specifically, that query decomposition yields gains in structured domains but degrades ranking precision in multi-hop settings, and that reflection mechanisms incur substantial latency costs for inconsistent quality improvements.

Results
The study reveals divergent outcomes based on the dataset and the specific agentic mechanism employed:

• Query Decomposition:

  • DevOps (Structured): Decomposition yielded consistent gains. The overall score increased from 0.814 to 0.855, MRR improved from 0.556 to 0.722, and Success@5 reached 1.0. However, latency more than doubled (21s to 47.66s).
  • MuSiQue (Multi-hop): Decomposition degraded ranking precision. While citation accuracy reached 1.0 and topic coverage increased, MRR collapsed from 0.469 to 0.102, and Success@5 dropped from 1.0 to 0.063. The authors attribute this to the fragmentation of contextual signals required for ranking in multi-hop settings.

• Reflection Mechanism:

  • Across both datasets, the reflection loop resulted in mixed to negative quality gains relative to its cost. On MuSiQue, latency increased roughly sixfold (17s to 104s) while the overall score decreased from 0.721 to 0.666. On DevOps, the overall score also decreased (0.870 to 0.781) with no improvement in recall or ranking.

• Strategy Distribution:

  • The Orchestrator routed the majority of DevOps queries (69.2%) through standard retrieval, with decomposition used for only ~5%. On MuSiQue, decomposition was used for ~16% of queries, indicating that the system selectively invokes complex strategies rather than applying them reflexively.

Significance and Claims
The paper argues that agentic enhancements are not universally beneficial and must be applied selectively according to query and domain characteristics. The primary significance of the work lies in demonstrating that "uniformly aggressive reasoning pipelines" are suboptimal. Instead, the authors advocate for adaptive, cost-aware orchestration that applies expensive strategies (decomposition, reflection) only when warranted by the query type.

The findings suggest that while decomposition is effective for structured domains where reasoning chains map to well-defined relationships, it can harm performance in complex multi-hop environments by diluting ranking signals. Furthermore, the substantial latency penalties associated with reflection mechanisms currently outweigh their marginal quality gains. The authors conclude that future systems must prioritize cost-aware optimization and learning-based strategy selection over static, heuristic-driven agentic workflows.

Limitations
The authors note several constraints: the DevOps corpus is modest in scale and generated with LLM assistance; the evaluation query sets are small, leading to wide confidence intervals; the study relies on a single generator and embedding model; and the orchestrator uses rule-based heuristics rather than learned policies. Latency figures are also environment-dependent, measured under local quantized inference.