💬 NLP

Rethinking On-policy Optimization for Query Augmentation

This paper systematically compares prompting-based and reinforcement learning-based query augmentation, revealing that simple training-free methods often rival complex RL approaches, and consequently proposes a novel hybrid method called On-policy Pseudo-document Query Expansion (OPQE) that synergistically combines the flexibility of prompting with the targeted optimization of RL to achieve superior retrieval performance.

Original authors: Zhichao Xu, Shengyao Zhuang, Xueguang Ma, Bingsen Chen, Yijun Tian, Fengran Mo, Jie Cao, Vivek Srikumar

Published 2026-03-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhichao Xu, Shengyao Zhuang, Xueguang Ma, Bingsen Chen, Yijun Tian, Fengran Mo, Jie Cao, Vivek Srikumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific needle in a massive, chaotic haystack. In the world of computers, this "needle" is the answer you need, and the "haystack" is the internet or a giant database of documents. The tool you use to find it is called a Retriever.

Sometimes, the problem isn't that the haystack is too big; it's that your description of the needle (your Query) is too vague. "Find me something about dogs" is a bad query if you actually want to know about "training golden retrievers for agility competitions."

This paper is about teaching computers how to write better descriptions (queries) so they can find the right information faster. The authors compared two main ways to teach computers this skill and then invented a third, better way.

Here is the breakdown using simple analogies:

1. The Two Old Ways of Teaching

Method A: The "Smart Guess" (Prompting)

The Analogy: Imagine you ask a very smart, well-read librarian (a Large Language Model) to help you. You say, "I need info on dog training." The librarian doesn't just repeat your words; they imagine a whole article about dog training and say, "Here is a fake article that answers your question. Use this text to search the library."
How it works: The computer generates a "fake document" based on what it already knows, and uses that to search.
Pros: It's free, fast, and requires no training. You just ask the smart librarian.
Cons: Sometimes the librarian gets too chatty or adds weird words that confuse the search engine.

Method B: The "Drill Sergeant" (Reinforcement Learning / RL)

The Analogy: Imagine you hire a student and put them in a training camp. Every time they write a query, you send them into the library. If they find the needle, you give them a gold star (a reward). If they fail, you give them a thumbs down. Over thousands of tries, the student learns exactly how to phrase things to get gold stars.
How it works: The computer learns by trial and error, specifically tuned to get high scores on search metrics.
Pros: It can become a master at a specific type of search.
Cons: It takes a long time, costs a fortune in electricity (compute), and the student might only be good at that specific library. If you move them to a new library, they might forget everything.

2. The Big Surprise

The authors ran a massive experiment comparing these two. They wanted to see if the expensive "Drill Sergeant" method was actually worth the money.

The Result: In many cases, the "Smart Guess" (Method A) was just as good as, or even better than, the "Drill Sergeant" (Method B).

Why? Because the "Smart Guess" uses a super-powerful, pre-trained brain that already knows a lot. The "Drill Sergeant" tries to teach a smaller brain from scratch, which is hard work and often doesn't pay off unless the task is very specific.

The Lesson: Sometimes, just asking a smart AI to "imagine an answer" is better than spending weeks training a smaller AI to "learn how to search."

3. The New Hybrid: "The Best of Both Worlds" (OPQE)

The authors realized they didn't have to choose. They created a new method called OPQE (On-policy Pseudo-document Query Expansion).

The Analogy: Imagine you take the "Drill Sergeant" training camp, but instead of teaching the student to just write a short, punchy sentence (a rewritten query), you teach them to write that fake article (the pseudo-document) that the librarian would write.
How it works:
1. The computer starts with the "Smart Guess" ability (it knows how to write a good fake article).
2. Then, it uses the "Drill Sergeant" (Reinforcement Learning) to fine-tune how it writes that article to get the best search results.
The Magic: It combines the rich knowledge of the smart librarian with the targeted discipline of the drill sergeant.

The Final Verdict

If you have a powerful AI: Just ask it to generate a "fake document" to help you search. It's cheap and works great.
If you want the absolute best performance: Use the new OPQE method. It teaches the AI to generate those "fake documents" but optimizes them specifically to beat the search engine's scoring system.

In a nutshell: The paper proves that you don't always need to spend months training a robot to be a search expert. Sometimes, just asking a smart robot to "pretend to be the answer" works better. And if you really want to win, teach the robot to "pretend to be the answer" and give it a gold star every time it does it right.

1. Problem Statement

In information retrieval (IR), improving query effectiveness is crucial, especially when the retriever itself cannot be modified (e.g., black-box API services like OpenAI's embeddings or proprietary systems like PubMed). Two dominant paradigms have emerged for query augmentation using Large Language Models (LLMs):

Prompt-based Augmentation: Zero-shot or few-shot methods (e.g., HyDE, Query2Doc) where an LLM generates a "pseudo-document" or rewrites a query based on its parametric knowledge. These are training-free but lack explicit optimization for retrieval metrics.
Reinforcement Learning (RL) based Augmentation: Methods (e.g., DeepRetrieval) that use on-policy RL (like PPO) to fine-tune an LLM to rewrite queries, directly optimizing for retrieval rewards (e.g., NDCG, Recall). These are computationally expensive and require labeled data or reward signals.

The Gap: There has been no systematic, compute-aware comparison between these two approaches. It remains unclear whether the computational cost of RL training yields significant gains over simple, powerful prompting, or if RL is only beneficial for specific retriever types (sparse vs. dense).

2. Methodology

The authors conducted a rigorous empirical comparison and proposed a novel hybrid method.

A. Comparative Evaluation Setup

Baselines:
- SPQE (Simple Pseudo-document Query Expansion): A zero-shot prompting method where an LLM generates a hypothetical document ( $d_H$ ) answering the query. The augmented query is the concatenation $(q, d_H)$ . To ensure a fair "compute-aware" comparison, SPQE utilized larger, proprietary, or strong open-weight models (e.g., GPT-4o-mini, Qwen3-32B) to match the training compute budget of RL methods.
- DeepRetrieval (DR): An on-policy RL method (using PPO) that trains smaller LLMs (3B/7B parameters) to rewrite queries directly, incorporating reasoning steps before generating the final query.
Datasets & Tasks: Evaluation across three domains:
- Evidence-seeking: NQ, TriviaQA, SQuAD.
- Ad Hoc Retrieval: BEIR datasets (FEVER, HotpotQA, SciFact, etc.).
- Tool Retrieval: ToolRet (Web, Code, Customized).
Retrievers: Both sparse (BM25) and dense (E5-base-v2, Contriever-msmarco) retrievers were tested.

B. Proposed Method: OPQE

Motivated by the observation that prompting often outperforms RL, the authors introduced On-policy Pseudo-document Query Expansion (OPQE).

Core Idea: Instead of training the policy to rewrite the query ( $q \to q'$ ), the policy is trained to generate a pseudo-document ( $d_H$ ) that maximizes retrieval performance.
Mechanism:
1. The LLM policy generates a hypothetical document $d_H$ given query $q$ .
2. The augmented input is $(q, d_H)$ .
3. The model is optimized via on-policy RL (PPO) using retrieval metrics (e.g., NDCG, Recall) as rewards.
Rationale: This merges the generative structure and rich context of prompting (which aligns well with dense retrievers and mitigates vocabulary mismatch for sparse ones) with the targeted optimization of RL.

3. Key Contributions

Systematic Benchmarking: The first controlled comparison of prompting vs. RL-based query augmentation across diverse IR tasks and retriever types, accounting for compute budgets (inference cost for prompting vs. training cost for RL).
Key Finding: Simple, training-free prompting (SPQE) using powerful LLMs often matches or surpasses complex, RL-trained methods. RL gains are highly dependent on the retriever type (better for sparse/BM25, inconsistent for dense) and task.
Novel Hybrid Method (OPQE): Introduced a method that applies on-policy optimization to pseudo-document generation rather than direct query rewriting. This approach consistently achieved the best overall performance across benchmarks.

4. Results and Analysis

A. Prompting vs. RL (SPQE vs. DeepRetrieval)

Sparse Retrievers (BM25): SPQE consistently outperformed RL methods. The pseudo-document approach effectively mitigates vocabulary mismatch by introducing relevant terms.
Dense Retrievers: SPQE performed on par with or slightly better than RL. However, RL methods (DeepRetrieval) showed mixed results; in some cases (e.g., ToolRet with dense retrievers), RL actually degraded performance compared to the base, likely because the RL policy learned to generate keyword-heavy outputs that hurt semantic embeddings.
Compute Efficiency: SPQE with a 32B model on a single GPU outperformed or matched RL training of 3B/7B models requiring multi-GPU clusters.

B. Performance of OPQE

Overall Best: OPQE achieved the highest average scores across all settings.
- Dense Retrieval: OPQE-7B achieved an average NDCG@10 of 58.1 on Ad Hoc retrieval, outperforming both DeepRetrieval (57.5) and SPQE (56.6).
- Sparse Retrieval: OPQE closed the gap with SPQE and significantly outperformed standard RL query rewriting.
Training Dynamics: Analysis of reward trajectories (Figure 1) revealed that OPQE starts with a "warm start" (higher initial rewards) because the pseudo-document structure leverages the LLM's internal knowledge immediately. Standard RL starts from a lower baseline and struggles to learn effective term generation for sparse retrievers.

C. Tool Retrieval Specifics

In tool retrieval, where candidate tools are functionally similar, SPQE and OPQE significantly outperformed the base and RL baselines.
RL methods failed to improve dense retriever performance in this domain, suggesting that the "query rewriting" formulation is less effective than "pseudo-document generation" for complex semantic matching tasks.

5. Significance and Implications

Re-evaluating RL Necessity: The paper challenges the assumption that RL is always necessary for query augmentation. For many scenarios, leveraging the inherent knowledge of large, instruction-tuned LLMs via simple prompting is more cost-effective and robust.
Synergistic Approach: The study demonstrates that the best results come from combining the two: using the generative structure of prompting as the objective for RL optimization.
Practical Guidance:
- For sparse retrievers or resource-constrained settings, simple prompting with strong LLMs is a strong default.
- For dense retrievers or specific domains, RL is useful but should be applied to pseudo-document generation (OPQE) rather than direct query rewriting to avoid performance degradation.
- RL approaches require per-dataset training, whereas prompting is zero-shot and transferable, making prompting more viable for dynamic environments.

Conclusion: The paper establishes that while RL offers targeted optimization, the "pseudo-document" paradigm is a superior structural choice for query augmentation. By training an RL policy to generate these pseudo-documents (OPQE), researchers can achieve state-of-the-art retrieval performance that surpasses both pure prompting and traditional RL query rewriting.