Technical Summary: QUESTER (Query Specification for Generative Keyword-Based Retrieval)

Problem Statement

Traditional Information Retrieval (IR) faces a trade-off between efficiency and effectiveness. Bag-of-words (BoW) models like BM25 are highly efficient and scalable but suffer from the vocabulary mismatch problem, where user queries and relevant documents share no common terms. Conversely, neural IR models (e.g., dense retrievers like DPR or sparse models like SPLADE) mitigate this mismatch but require costly index construction and rebuilding, limiting their scalability and adaptability.

Generative Retrieval (GR) models attempt to internalize the index within model parameters to bypass indexing, but they often struggle with generalization on large collections or rely on arbitrary document identifiers that lack semantic grounding. Furthermore, existing Large Language Model (LLM)-based query rewriting methods often require large models, complex prompting, and stochastic sampling, leading to high inference latency and instability, particularly when using smaller LLMs.

Methodology

The authors propose QUESTER (QUEry SpecificaTion gEnerative Keyword-Based Retrieval), a framework that bridges Generative Retrieval and query reformulation. Instead of mapping queries to document metadata or generating arbitrary identifiers, QUESTER learns to generate explicit keyword-based search specifications (rewritten queries) that are executed by a standard, efficient lexical retriever (BM25).

Core Components

1. Policy Optimization via GRPO:
   The rewriting policy is a lightweight LLM (specifically Qwen3-4B) optimized using Group Relative Policy Optimization (GRPO). For each input query, the policy samples a group of candidate keyword queries. The policy is updated based on the relative performance of these candidates rather than absolute rewards, encouraging exploration.

2. Reward Signal (SoftNDCG with Distillation):
   To train the policy without direct access to ground-truth relevance labels for every candidate, the system uses a reward function based on SoftRank (Taylor et al., 2008).

   • SoftNDCG: Unlike hard ranking metrics, SoftNDCG computes the expected nDCG by assuming search engine scores follow a normal distribution. This provides a differentiable and stable gradient signal even when score differences are small.
   • Distillation: To address the sparsity and noise of user-click labels (e.g., in MS MARCO), the authors employ knowledge distillation. A powerful Cross-Encoder (CE) model assesses the relevance of retrieved documents, providing graded relevance scores that serve as the supervision signal for the GRPO training.

3. Inference:
   During inference, the model generates a single rewritten query using greedy decoding (temperature $\tau=0$). This deterministic output is then passed to a standard BM25 retriever. The system does not require rebuilding the index when the neural policy is updated.

Key Contributions

• Generative Keyword Specifications: The paper introduces a novel approach where generative models produce structured keyword queries rather than document identifiers, leveraging the efficiency of established lexical search engines.
• Efficient Training with Small LLMs: QUESTER demonstrates that a small, open-source LLM (4B parameters) fine-tuned with GRPO can achieve competitive performance, avoiding the computational costs associated with large proprietary models or complex multi-sampling strategies used in prior work.
• Robust Reward Design: The integration of SoftNDCG with Cross-Encoder distillation provides a stable and effective reward signal for reinforcement learning in IR, overcoming the limitations of sparse binary labels.
• Efficiency-Effectiveness Balance: The framework maintains the low latency of BM25 retrieval while significantly boosting effectiveness through intelligent query rewriting.

Experimental Results

The model was evaluated on the MS MARCO dataset (in-domain) and the BEIR benchmark (out-of-domain).

• In-Domain Performance: On MS MARCO Dev and TREC DL datasets, QUESTER significantly outperforms standard BM25 (+4.0 nDCG@10) and zero-shot prompting of the same LLM. It achieves performance comparable to or better than state-of-the-art query rewriting baselines (e.g., HyDE, LameR, MuGI) while using a much smaller model.
• Out-of-Domain Performance: On the BEIR benchmark, QUESTER outperforms neural IR baselines (SPLADEv2, ColBERTv2) and is competitive with other query rewriting methods, demonstrating strong generalization capabilities.
• Efficiency: QUESTER offers a favorable trade-off. While prompt-heavy methods like MuGI and LameR achieve high nDCG, they incur significant latency (>100 ms/query). QUESTER achieves similar quality with a latency of approximately 28 ms/query (excluding generation time), making it 4–7 times faster than its closest competitors in terms of retrieval latency.
• Ablation Studies:
  • Model Size: Performance scales with model size (0.6B < 1.7B < 4B), with the 4B model offering the best balance.
  • Supervision: Using Cross-Encoder derived labels is crucial; training without CE supervision results in lower performance.
  • SoftNDCG Parameters: A moderate standard deviation ($\nu=0.5$) in SoftRank yields the best results, balancing stability and discriminability.
  • KL Weight: Setting the KL divergence weight to 0 (omitting explicit KL penalty) encourages necessary exploration and yields the best performance.

Significance and Claims

The paper claims that QUESTER represents a significant step forward in balancing the efficiency of lexical retrieval with the effectiveness of generative query rewriting. By decoupling the generative model from the retrieval index, the approach avoids the high costs of index rebuilding associated with neural IR models.

The authors position QUESTER as a practical solution for scenarios requiring high efficiency and explainability (as the output is a human-readable keyword list) without sacrificing retrieval quality. They note that while the model lags slightly behind specialized dense neural models in-domain, its out-of-domain performance and efficiency make it a strong candidate for scalable search applications. The work suggests a future direction toward more structured query languages (e.g., Lucene syntax) and hybrid backends, but emphasizes that the current keyword-based approach already improves upon existing related works in terms of efficiency while maintaining comparable effectiveness.