Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search

This paper presents a controlled study using the Budget-Constrained Agentic Search (BCAS) framework to quantify how search depth, retrieval strategies, and completion budgets impact the accuracy and cost of Agentic RAG systems across six LLMs and three benchmarks, offering practical configuration guidelines for budget-constrained deployments.

The Gist

Imagine you are hiring a research assistant (an AI) to answer a difficult question for you. You have two main constraints:

1. Time/Money: You can only ask them to look up information a few times (Search Budget).
2. Paper Length: You can only let them write a certain number of words before you stop them (Token Budget).

This paper is like a scientific taste test to figure out: "How do we get the best answer from our assistant without blowing our budget?"

The researchers built a testing ground called BCAS (Budget-Constrained Agentic Search). They didn't just ask the AI to "do its best." Instead, they forced it to work within strict limits, like a student with a limited number of library visits and a strict word count for their essay.

Here is what they discovered, explained through simple analogies:

1. The "Three-Visit" Rule (Search Depth)

The Finding: Accuracy goes up when you let the AI search more, but only up to a point. After about three searches, the extra effort barely helps.

• The Analogy: Imagine you are trying to find a specific recipe in a giant library.
  • 1 Search: You ask the librarian once. You might get the right book, or you might get a cookbook from the wrong decade.
  • 2-3 Searches: You ask again, maybe refining your question. Now you have a few good options. Your chances of finding the perfect recipe skyrocket.
  • 10 Searches: You keep asking. You now have 50 cookbooks. But you're just re-reading the same recipes. You aren't getting a better answer; you're just getting more of the same. You've hit the "diminishing returns" wall.

Takeaway: Don't let the AI search forever. Give it 3 chances to look things up, then let it write the answer.

2. The "Smart Librarian" vs. The "Keyword Matcher" (Retrieval Strategy)

The Finding: The best way to find information isn't just matching keywords (like a basic Google search) or just matching "vibes" (AI understanding). It's a hybrid approach, followed by a re-ranker (a second opinion).

• The Analogy:
  • Basic Search (BM25): Like a librarian who only finds books with the exact words you typed. If you ask for "fast cars," they ignore a book titled "Speeding Vehicles."
  • AI Search (Dense): Like a librarian who understands concepts. They find "Speeding Vehicles" because they know it's about fast cars.
  • The Hybrid + Re-ranker: This is the Super-Librarian. They use both methods to get a huge list of candidates, then a senior editor (the re-ranker) looks at that list and says, "Actually, this book is the most useful one. Put that at the top."
  • Result: This combination gave the biggest boost in accuracy, especially for tricky questions.

3. The "Essay Length" Trap (Token Budget)

The Finding: Giving the AI more space to write (more tokens) only helps if the question requires complex thinking (like combining facts from three different books). For simple questions, more writing space just makes the AI ramble.

• The Analogy:
  • Simple Question: "Who was the first president?"
    • If you give the AI a 1-page limit or a 10-page limit, the answer is still just "George Washington." Giving it more paper doesn't make the answer better; it just makes the AI write a long, boring biography when you only wanted a name.
  • Complex Question: "How did the invention of the printing press influence the spread of democracy in Europe?"
    • Here, the AI needs a lot of space to connect the dots between the printing press, the spread of ideas, and political changes. If you cut the paper short, the AI has to rush and miss the connections.
  • The Twist: Sometimes, giving the AI less writing space forces it to be efficient. It stops rambling and focuses on making more searches instead of writing a long, useless essay.

4. The "Small Dog" vs. The "Smart Dog" (Model Size)

The Finding: Smaller, cheaper AI models can actually beat huge, expensive models if you give them more search opportunities and better tools.

• The Analogy:
  • The Big Model (Expensive): Like a genius dog that knows a lot of facts already. It can answer well with just one look-up. But it's expensive to feed.
  • The Small Model (Cheap): Like a smart puppy. It doesn't know as much off the top of its head.
  • The Magic: If you give the puppy a map (planning) and let it run around the neighborhood three times (searches) to find the facts, it can catch up to the genius dog. The "cheap" option becomes the "best value" option because you aren't paying for the expensive model's brain; you're paying for the process of finding the answer.

The "Golden Rule" for Budgets

If you are building an AI system and have a limited budget, the paper suggests this order of operations:

1. First, buy more "Searches": Let the AI look things up 2 or 3 times. This gives the biggest bang for your buck.
2. Second, upgrade the "Search Engine": Use a smart hybrid search with a re-ranker (the Super-Librarian).
3. Third, only then, buy more "Writing Space": Only increase the word limit if you are asking very complex questions that require synthesizing lots of information.

Summary

The paper teaches us that being "Agentic" (smart and autonomous) isn't about letting the AI run wild. It's about strategic constraints. By forcing the AI to search a few times, use smart tools, and write concisely, you get better answers for less money. It's the difference between hiring a frantic intern who writes 50 pages of nonsense, and a focused researcher who checks three sources and gives you the perfect answer.

Technical Summary

Here is a detailed technical summary of the paper "Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search."

1. Problem Statement

Retrieval-Augmented Generation (RAG) has evolved from static "retrieve-then-generate" pipelines to Agentic RAG systems, where Large Language Models (LLMs) autonomously plan and execute multi-step search strategies. While current research focuses on maximizing answer accuracy, there is a critical gap in understanding how computational budget constraints (specifically limits on search calls and generation tokens) impact performance in real-world deployments.

Most existing studies assume generous tool usage or rely on bespoke training. There is a lack of controlled, model-agnostic measurements quantifying how design decisions (e.g., search depth, retrieval strategies, re-ranking) trade off against accuracy when strict budgets are enforced. The authors aim to fill this "measurement gap" by determining the optimal configuration for budgeted agentic pipelines.

2. Methodology: The BCAS Framework

To address this, the authors introduce Budget-Constrained Agentic Search (BCAS), a model-agnostic evaluation harness designed to simulate real-world cost constraints.

• Core Architecture: BCAS operates as a stateful loop where the LLM reasons, selects a tool (e.g., search_database), executes, and observes results.
• Budget Enforcement:
  • Search Budget: A hard cap (max_searches) is enforced. Once exhausted, the search tool is removed from the action space, forcing the model to answer with gathered information.
  • Token Budget: A cumulative cap (max_total_tokens) tracks input and output tokens. The loop terminates if this limit is reached.
  • Explicit Signals: The model is explicitly informed of its remaining budget at every step, encouraging deliberate planning.
• Experimental Setup:
  • Models: Six LLMs ranging from small (LLaMA 3.1 8B, Qwen 3 14B) to large/reasoning-focused (o4-mini, DeepSeek V3, GPT-4.1-mini, Gemma 3 27B).
  • Datasets: Three multi-hop QA benchmarks with varying complexity:
    • TriviaQA: Single-fact retrieval.
    • HotpotQA: Multi-hop reasoning requiring synthesis.
    • 2WikiMultihopQA: Complex entity relationships and reasoning.
  • Retrieval Components: The study ablates various Information Retrieval (IR) strategies:
    • Baseline: BM25 with phrase boosting.
    • Hybrid Search (HS): BM25 + Dense Vector Search (BGE-M3).
    • Re-ranking (RR): Cross-encoder re-ranking of top candidates.
  • Agentic Features: Optional Pre-planning (decomposing questions) and Reflection (reviewing progress/adjusting strategy).

3. Key Contributions

1. BCAS Evaluation Harness: A reproducible, open-source framework that standardizes budget constraints across different models and datasets without requiring fine-tuning.
2. Systematic Ablation Study: A controlled comparison of design knobs (search depth, retrieval type, planning, reflection) across six models and three datasets.
3. Quantified Trade-offs: Empirical data on how accuracy scales with search depth and token budgets, revealing non-linear relationships and diminishing returns.
4. Practical Configuration Guidelines: Evidence-based recommendations for deploying cost-effective agentic RAG systems.

4. Key Results

The study identifies three consistent patterns across models and datasets:

A. Search Depth vs. Accuracy (RQ1 & RQ3)

• Diminishing Returns: Accuracy improves reliably with additional searches up to a cap of roughly 3 searches. Beyond this, gains plateau.
• Model Capacity Narrowing: Iterative search significantly narrows the performance gap between small and large models. For example, on HotpotQA, a smaller model (Qwen 3 14B) with unlimited searches and planning (75.33%) outperformed a larger model (o4-mini) with a single search (70.17%).
• Strategy: Allocating budget to multiple search steps is generally more effective than allocating the same budget to a single, long generation.

B. Component Tuning (RQ2)

• Hybrid + Re-ranking is King: The combination of Hybrid Search (Lexical + Dense) with lightweight Re-ranking (RR) yielded the largest average accuracy gains (+9.29% on HotpotQA).
• Planning & Reflection: These features provided significant boosts (4–12 points) for smaller models (e.g., LLaMA 8B, Qwen 14B) but had minimal impact on larger, reasoning-native models like o4-mini.
• Model Dependency: Smaller models benefit more from external scaffolding (planning/reflection) and retrieval enhancements, whereas larger models rely more on their inherent reasoning capabilities.

C. Token Budget vs. Task Type

• Synthesis-Heavy Tasks: For HotpotQA, increasing the token budget from 4k to 16k resulted in a sharp accuracy increase. This suggests the bottleneck was the model's ability to synthesize multiple retrieved documents, not the retrieval itself.
• Retrieval-Heavy Tasks: For 2WikiMultihopQA and TriviaQA, token budget increases had minimal impact. These tasks are constrained by the difficulty of finding the correct evidence, not by the generation length.
• The "Constrained Efficiency" Phenomenon: Surprisingly, models with tight token budgets (500–2k) sometimes outperformed those with unlimited budgets when allowed multiple searches. Tight budgets forced models to make succinct, efficient tool calls, whereas unlimited budgets encouraged verbose, single-pass generations that consumed tokens without gathering new information.

D. The o4-mini Anomaly

The reasoning model o4-mini showed resistance to most external enhancements (planning/reflection added <1% gain), likely because it possesses built-in chain-of-thought processing. However, it showed a massive jump (25 points) on 2WikiMultihopQA when both planning and reflection were combined, suggesting reflection can complement its internal reasoning for complex strategy revision.

5. Significance and Recommendations

The paper shifts the focus from "building the best agent" to "optimizing the agent under constraints." The authors propose a practical deployment hierarchy for budget allocation:

1. Prioritize Search Depth: First, allocate budget to allow 2–3 iterative search steps. This yields the highest marginal return on accuracy.
2. Optimize Retrieval Quality: Implement Hybrid Search with Re-ranking. This provides the most consistent accuracy gains across model sizes.
3. Expand Token Budgets Last: Only increase generation token limits for tasks requiring heavy synthesis (like HotpotQA). For retrieval-heavy tasks, a moderate token budget combined with multiple searches is more cost-effective.

Conclusion:
The study demonstrates that budget-aware design is critical for practical Agentic RAG. By explicitly managing search and token constraints, practitioners can achieve high accuracy with significantly lower costs, often allowing smaller, cheaper models to outperform larger models constrained to single-pass retrieval. The BCAS framework provides the necessary tools to make these trade-off decisions data-driven rather than heuristic.