💻 computer science

An Agent-Oriented Pluggable Experience-RAG Skill for Experience-Driven Retrieval Strategy Orchestration

This paper introduces Experience-RAG Skill, an agent-oriented pluggable module that dynamically orchestrates retrieval strategies by analyzing task contexts and consulting experience memory, thereby outperforming fixed pipelines and achieving competitive results across diverse heterogeneous tasks.

Original authors: Dutao Zhang, Tian Liao

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dutao Zhang, Tian Liao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef (the Agent) trying to cook a meal based on a customer's order. In the past, your kitchen had a strict rule: no matter what the customer asked for, you always used the same single knife and the same single cutting board.

If the customer ordered a delicate fish, you used the knife. If they ordered a tough steak, you still used the same knife. Sometimes it worked, but often the result was messy or inefficient.

This paper introduces a new tool called Experience-RAG Skill. Think of this not as a new knife, but as a smart sous-chef who stands between you and your entire kitchen.

Here is how it works in simple terms:

1. The Problem: One Size Doesn't Fit All

The authors noticed that different types of questions need different ways of finding answers.

Factoid questions (e.g., "Who was the first president?") are like cutting a tomato; you need a sharp, direct tool.
Multi-hop questions (e.g., "Who directed the movie that starred the actor who won the Oscar in 2020?") are like deboning a chicken; you need a more complex, multi-step approach.
Scientific questions are like filleting a fish; they require extreme precision.

Old systems tried to use one "fixed" search method for all of these. The authors say this is like using a butter knife to chop a steak—it just doesn't work well for everything.

2. The Solution: The Smart Sous-Chef

The Experience-RAG Skill is a pluggable layer (like a smart assistant) that sits between the main Agent and the pool of available search tools (the "Retriever Pool").

When you give it a request, it does four things:

Scene Analysis: It looks at the request and asks, "What kind of task is this? Is it simple? Is it complex? Is it scientific?"
Experience Memory: It checks a "notebook" of past experiences. It remembers, "Last time we had a complex question like this, the 'Hybrid Search' tool worked best."
Strategy Routing: Based on the scene and the memory, it picks the best tool from the kitchen (e.g., "Use the Dense Retriever" or "Use the Hybrid RRF").
Packaging: It gathers the evidence and hands it back to you in a neat, organized package, so you don't have to worry about how it was found, just that it was found.

3. The Results: A Better Meal

The authors tested this smart sous-chef on three different types of "orders" (datasets):

Simple fact questions.
Complex, multi-step reasoning questions.
Scientific fact-checking.

The Outcome:

The smart sous-chef (Experience-RAG Skill) performed better than using any single tool fixed in place.
It was almost as good as the most advanced "adaptive" systems currently on the market, but with a key difference: it treats the choice of tool as a reusable skill that the Agent can learn and use, rather than hard-coding it into the system's brain.

4. The Catch (Limitations)

The paper is honest about what it doesn't do yet:

The Kitchen is Small: The tests were done on a small, sampled set of ingredients (documents), not a massive, real-world library.
The Rules are Simple: Right now, the "memory" mostly follows simple rules (e.g., "If it's scientific, use Tool X"). They tried to make it learn on its own, but the simple rules worked better for now.
Fixed Menu: It can only choose from the tools currently in the kitchen. It can't go out and buy new tools on the fly yet.

Summary

In short, this paper argues that instead of forcing a robot to use the same search method for every job, we should give it a smart manager. This manager looks at the job, remembers what worked in the past, picks the right search tool, and delivers the answer. It's a small but powerful step toward making AI agents more flexible and efficient.

Technical Summary: Experience-RAG Skill for Experience-Driven Retrieval Strategy Orchestration

Problem Statement

Current Retrieval-Augmented Generation (RAG) systems frequently operate under the assumption that a single, fixed retrieval pipeline is sufficient for all tasks. However, this assumption fails in realistic agent settings where task requirements vary significantly. For instance, factoid question answering, multi-hop reasoning, and scientific claim verification exhibit distinct retrieval preferences; a dense retriever may suffice for direct facts, whereas hybrid retrieval often proves more stable for complex multi-hop or scientific evidence matching. The core challenge identified is not merely retrieving knowledge, but determining how retrieval should be organized and orchestrated for the specific current task.

Methodology

The paper proposes Experience-RAG Skill, an agent-oriented, pluggable retrieval orchestration layer positioned between the agent and a pool of retrievers. Rather than treating retrieval as a monolithic step, this skill encapsulates the selection process as a reusable agent capability.

Framework Architecture

The framework operates through a structured pipeline involving six modules:

Skill Interface: The unified entry point for the agent.
Scene Analyzer: Constructs a structured scene representation ( $s$ ) based on the query ( $q$ ), dialogue history ( $h$ ), and task metadata ( $m$ ).
Experience Memory ( $M$ ): Stores historical records of scene features, score vectors, the best-performing strategy, and the performance margin. Features include task type, domain, context length, question complexity, query style, and document structure.
Strategy Router: Selects the optimal retriever ( $r^*$ ) by consulting the experience memory based on the current scene.
Retriever Pool ( $R$ ): A candidate pool of retrievers (e.g., BM25, Dense, Hybrid RRF).
Result Packager: Returns standardized, structured evidence to the agent.

The operational flow is defined as:
$s = \text{Analyze}(q, h, m)$
$r^* = \text{Route}(s, M)$
$o = \text{RetrieveAndPackage}(q, r^*)$

Routing Mechanism

The current implementation utilizes rule-based routing as its strongest setting, mapping direct tasks to dense retrievers and multi-hop/scientific tasks to hybrid RRF (Reciprocal Rank Fusion). While learned routing variants were explored, they did not outperform the rule-based baseline under the current scale of experience data. The system prioritizes explicit experience reuse and structured evidence packaging over purely complexity-driven selection.

Key Contributions

The authors outline three primary contributions:

Formulation: They reframe retrieval strategy selection as an agent skill orchestration problem, distinct from embedding selection logic directly into the upper workflow.
Experience Memory Construction: They built a memory system that records comprehensive scene features, multi-retriever performance scores, and strategy margins, rather than just storing the "best" label.
Empirical Evaluation: They evaluated the skill on heterogeneous retrieval tasks, comparing it against both traditional fixed baselines and modern adaptive routing approaches.

Experimental Results

The system was evaluated on three public benchmarks: BeIR/nq, BeIR/hotpotqa, and BeIR/scifact, using 120 queries per dataset with sampled candidate corpora.

Performance Against Fixed Baselines

Experience-RAG Skill achieved the highest overall performance, outperforming all fixed single-method baselines:

nDCG@10: 0.8924 (vs. 0.8802 for the strongest fixed baseline, Hybrid RRF).
Recall@10: 0.9428.
MRR@10: 0.9006.
The results indicate that the skill's primary advantage lies in heterogeneous orchestration, effectively managing mixed workloads rather than dominating every single subtask individually.

Comparison with Modern Baselines

When compared to modern approaches including HyDE, LongRAG-style, RAPTOR-style, and Adaptive-RAG-style:

Adaptive-RAG-style showed a marginal lead (0.8934 nDCG@10 vs. 0.8924 for Experience-RAG Skill).
HyDE, LongRAG-style, and RAPTOR-style performed significantly weaker in the sampled-corpus, retrieval-only setting.
The paper notes that Experience-RAG Skill remains competitive with the strongest modern routing baselines while offering a clearer system boundary for agent-facing interaction.

Significance and Claims

The paper concludes that retrieval strategy selection can be productively encapsulated as a reusable agent skill rather than being hard-coded into the workflow. The study demonstrates that experience-driven orchestration is effective for heterogeneous retrieval workloads and remains competitive even when modern baselines are introduced into a fixed candidate pool.

The authors maintain a modest stance regarding the current work's scope:

The experiments rely on sampled corpora rather than full-scale datasets.
Learned routing has not yet surpassed rule-based baselines.
The implementation assumes a fixed candidate pool, excluding dynamic onboarding.
The workflow analysis is qualitative and does not yet constitute a full end-to-end interactive agent benchmark.

Consequently, the authors position this work as a short paper or preprint establishing the viability of the skill layer, with future work planned to address dynamic candidate onboarding, capability-level routing, and comprehensive end-to-end agent evaluation.