🤖 AI

RAISE: RAG Design as an Architecture Search Problem

This paper introduces RAISE, a comprehensive framework and benchmark that formulates RAG design as an architecture search problem to enable systematic, reproducible evaluation of hyperparameter optimization methods across diverse datasets and tasks.

Original authors: Zhen Chen, Yibing Liu, Weihao Xie, Yu Liang, Peilin Chen, Shiqi Wang

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhen Chen, Yibing Liu, Weihao Xie, Yu Liang, Peilin Chen, Shiqi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the perfect recipe for a gourmet sandwich. You have a lot of ingredients to choose from: the type of bread, the thickness of the slice, the kind of cheese, how much mustard to use, and whether to toast it.

In the world of Artificial Intelligence, this "sandwich" is called RAG (Retrieval-Augmented Generation). It's a system where a smart AI (like a chatbot) looks up facts in a giant library before answering a question. To make the AI work well, you have to tune many "knobs":

How big should the library pages be?
How many pages should the AI read?
Should it rewrite the question first to make it clearer?
Should it re-rank the pages to find the best ones?

The Problem: Guessing vs. Searching

Until now, people building these AI systems have mostly been guessing. They tweak the knobs based on gut feeling or trial-and-error. It's like trying to bake the perfect cake by randomly changing the amount of sugar and flour, hoping you get lucky. This makes it hard to know if one recipe is truly better than another, because everyone is using different ingredients and different ovens.

The Solution: RAISE (The "Sandwich Architect")

The authors of this paper introduce a new tool called RAISE (RAG Intelligence Search Engine). Think of RAISE as a super-organized kitchen that treats building an AI not as a guessing game, but as a scientific search problem.

Instead of just trying random recipes, RAISE sets up a controlled environment where:

The Ingredients are Fixed: Everyone uses the same library of documents and the same types of questions.
The Budget is Fixed: Every method gets the same number of "tasting attempts" (e.g., 30 tries).
The Judges are Fair: They use a standard scoring system to rate the sandwiches.

RAISE acts as a tasting panel that tests 13 different "search strategies" (different ways of tuning the knobs) to see which one finds the best recipe fastest.

The Big Discovery: "One Size Does Not Fit All"

The most surprising thing the paper found is that there is no single "best" strategy.

Imagine you have 13 different chefs.

Chef A is amazing at making sandwiches for a picnic (a specific type of question).
Chef B is the master of late-night snacks (a different type of question).
If you ask Chef A to make a late-night snack, they might fail. If you ask Chef B to make a picnic sandwich, they might struggle.

The paper shows that an optimization method that works perfectly on one dataset (like a trivia quiz) might perform terribly on another (like a complex science question).

The Analogy:
Think of it like shoes.

You wouldn't wear running shoes to climb a mountain.
You wouldn't wear hiking boots to swim in a pool.
You wouldn't wear sandals to play soccer.

The paper argues that we shouldn't just create a "Leaderboard" that says "Chef A is the best overall." Instead, we need to say, "Chef A is the best for this specific type of question, but Chef B is better for that one."

What They Actually Did

They built a framework (RAISE) that lets researchers plug in different search algorithms.
They tested these algorithms on 7 different types of questions (ranging from simple trivia to complex science and long documents).
They ran the tests three times with different random seeds to make sure the results weren't just luck.
They found that the "best" algorithm changes depending on the task. For example, some tasks need the AI to read more pages, while others need it to rewrite the question first.

The Takeaway

The paper doesn't claim to have found the "magic bullet" that makes all AI perfect. Instead, it provides a fair playing field (a benchmark) so that researchers can stop guessing and start systematically understanding which tuning strategy works for which specific problem.

It's a call to stop looking for a universal "best" AI recipe and start recognizing that the best recipe depends entirely on what you are trying to cook.

Technical Summary: RAISE: RAG Design as an Architecture Search Problem

Problem Statement
Retrieval-Augmented Generation (RAG) systems involve numerous coupled design choices, including query rewriting, chunking strategies, retrieval depth, reranking mechanisms, and context compression. In practice, these components are typically configured via heuristics or trial-and-error tuning. This approach hinders systematic evaluation, reproducibility, and fair comparison across different RAG systems. Existing efforts to optimize RAG configurations (e.g., AutoRAG, Self-RAG, AutoRAG-HP) utilize disparate pipelines, datasets, search spaces, and evaluation protocols, leaving the field without a shared framework for comparing optimization algorithms under controlled conditions.

Methodology
The authors formulate RAG design as an Architecture Search problem, specifically treating it as the selection of pipeline configurations that optimize end-to-end performance under a fixed evaluation budget. To enable controlled study, they introduce RAISE (RAG Intelligence Search Engine), a comprehensive framework and benchmark.

Framework Architecture: RAISE decouples the system into three components:
1. Pipeline Abstraction: A parameterized, modular directed acyclic graph (DAG) representing the RAG workflow. The search space $\Theta$ is the Cartesian product of configuration spaces for six modules: Query Rewriter, Chunker, Text/Visual Retriever, Reranker, Pruner, and Generator. This supports both text-only and multimodal (text-vision) pipelines.
2. Evaluation Layer: Maps configurations to task-level rewards. The objective function is a dataset-level average of generation quality, supporting lexical (ROUGE-L, BLEU, METEOR, token-F1), semantic, and model-based metrics.
3. Controller Interface: A standardized API through which 13 distinct optimization algorithms propose configurations ( $\theta_t$ ) and receive rewards ( $r_t$ ) from the environment. This treats RAG configuration as a black-box optimization problem.
Benchmark Environments: The framework is instantiated across seven public datasets representing diverse RAG challenges:
- Text: TriviaQA (long documents), HotpotQA (multi-hop), MS MARCO (retrieval/reranking), SQuAD v2 (extractive/hallucination control), LongBench-MF (long-context/cross-field), and LongBench-Qasper (scientific reasoning).
- Multimodal: ScienceQA (visual-textual grounding).
  Each environment uses a fixed proxy subset (100 QA pairs) to ensure computational feasibility while maintaining task structure.
Optimization Algorithms: The study evaluates 13 algorithms grouped into seven families: Random Search, Local Trajectory (Greedy, Coordinate Descent, Simulated Annealing, Iterative Local Search), SMBO/Bayesian (TPE), Estimation-of-Distribution (Cross-Entropy Method), Evolutionary (Regularized Evolution), Bandit (Thompson Sampling, UCB), and RL-style (GRPO, Dr. GRPO, Reinforce++).

Key Results
The authors conducted a controlled evaluation using three random seeds and a budget of 30 configuration evaluations per algorithm per environment.

Optimizer-Environment Interaction: No single optimization algorithm dominates across all environments. Performance is highly task-dependent. For instance, Greedy Search performed best on HotpotQA and TriviaQA, while Cross-Entropy Method (CEM) led on MS MARCO, and Random Search was competitive on ScienceQA. Aggregate rankings do not reflect universal superiority; instead, results must be interpreted as interactions between the search strategy and the specific RAG environment.
Module Sensitivity: Ablation studies reveal that the impact of specific pipeline modules varies by task structure.
- TriviaQA (long documents) is most sensitive to query rewriting and pruning.
- HotpotQA (multi-hop) relies heavily on retrieval depth.
- LongBench-MF is highly sensitive to retrieval top-k settings.
  This confirms that optimal pipeline choices are not globally transferable but depend on the evidence structure of the task.
Search vs. Random: Under matched budgets, explicit optimization methods generally outperform random sampling, though gains are often modest. Adaptive methods (e.g., Bandit, RL-style) tend to perform better in later trials by focusing on promising configurations.
Proxy Stability: The study found that proxy task sizes of 100–200 examples provide sufficient stability for ranking optimizers, with diminishing returns for larger subsets.

Significance and Claims
The paper claims that RAISE provides a common experimental substrate for fair, reproducible, and systematic research on RAG hyperparameter optimization. By separating the controller, search space, and environment, RAISE reduces confounds related to pipeline implementation and evaluation protocols.

The authors argue that RAG architecture search should be reported as optimizer–environment interactions rather than through a single universal leaderboard. They posit that the field requires standardized benchmarks similar to HPOBench or YAHPO Gym to advance the understanding of how different search biases perform under specific RAG constraints. The framework is designed to be extensible, allowing new optimization algorithms and benchmark tasks to be integrated without altering the core pipeline implementation.

Limitations
The authors acknowledge that their experiments instantiate a fixed search space over specific models (Qwen3 series) and discrete values. Alternative model families, retrieval backends, or continuous search spaces may alter algorithm behavior. Additionally, the reliance on lightweight proxy environments and a fixed budget (30 trials) facilitates broad comparison but does not substitute for full-dataset or large-budget studies. The primary objective uses an equal-weight lexical and token-level aggregate, treating semantic and model-based metrics as auxiliary signals.