SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval

` tags.

• Search: Queries generated within <search> tags.
• Retrieval: Retrieved documents wrapped in <information> tags.
• Answer: Final response in <answer> tags.

The framework encourages query decomposition, where complex queries are broken down into sub-queries (either sequentially or in parallel) to facilitate better retrieval.

B. Intrinsic Intermediate Rewards

SubSearch introduces three specific reward components calculated during the reasoning trace:

1. Answerability Reward ($R_{answerability}$):

   • Measures how well a generated sub-query is addressed by the retrieved documents.
   • Calculated as the cosine similarity between the sub-query embedding and the top-$k$ retrieved document embeddings.
   • Acts as a continuous proxy for information coverage, ensuring the model searches for relevant data.

2. Decomposition Reward ($R_{decomposition}$):

   • Encourages high-quality query splitting. It consists of two objectives:
     • Semantic Coverage: Ensures the aggregate of sub-queries maintains the same information content as the parent query (preventing "query drift").
     • In-group Splitability: Maximizes the relevance of a sub-query to the parent while ensuring uniqueness relative to sibling sub-queries (mutual exclusivity).
   • This is a weighted combination of coverage and splitability scores.

3. Format Reward ($r_{format}$):

   • A binary or scalar signal to ensure the model adheres to the required XML-like structure (<search>, <answer>, etc.), preventing training instability and early model collapse.

C. Reward Aggregation

The final reward signal is an adaptive residual aggregation of the intermediate rewards and the final outcome reward (Exact Match):
$$r = R_{answer} + \beta(1 - R_{answer}) \cdot \text{avg}(R_{intermediate}) + r_{format}$$

• Logic: If the model gets the final answer correct ($R_{answer}=1$), the intermediate rewards are ignored (or weighted down). If the model fails the final answer, the intermediate rewards act as a "residual" signal to guide the model toward better reasoning steps.
• Adaptive $\beta$: The weight of the intermediate rewards ($\beta$) evolves during training, prioritizing intermediate signals when the model is struggling to find the correct answer.

D. Optimization

The model is trained using Group Relative Policy Optimization (GRPO), a variant of PPO that does not require a separate critic model, making it computationally efficient.

3. Key Contributions

1. Intrinsic Process Rewards: A novel method to generate intermediate rewards purely from the model's internal state (embeddings) and retrieval results, removing the dependency on human annotations or external LLM judges.
2. Decomposition-Aware Training: Explicitly incentivizes the decomposition of complex queries into sub-queries, improving the agent's ability to handle multi-hop reasoning.
3. Adaptive Residual Aggregation: A mechanism to dynamically balance outcome and process rewards, ensuring that intermediate guidance is used primarily when the final outcome is uncertain.
4. Data Efficiency: Demonstrates that high-performance search agents can be trained without expensive supervised fine-tuning (SFT) on reasoning trajectories.

4. Experimental Results

The authors evaluated SubSearch on seven benchmarks, including open-domain QA (Natural Questions, TriviaQA, PopQA) and multi-hop QA (HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle).

• Performance: SubSearch consistently outperformed other SFT-free RL-based agents (e.g., Search-R1, ZeroSearch, R-Search) and standard baselines (CoT, RAG).
  • Multi-hop gains: Significant improvements were observed on complex datasets, e.g., +6.5 EM on HotpotQA, +7.7 EM on 2WikiMultiHopQA, and +13.5 EM on Bamboogle compared to Search-R1-base.
  • General QA: Also achieved gains on simpler datasets (e.g., +4.2 EM on Natural Questions).
• Ablation Studies:
  • Decomposition: Query decomposition via prompting significantly improved performance on multi-hop datasets (HotpotQA) but had less impact on factoid datasets (NQ), confirming the method's adaptability.
  • Reward Signals: Combining answerability and decomposition rewards yielded better results than using only outcome rewards or decomposition alone.
  • Aggregation: The adaptive residual aggregation method proved superior to simple weighted sums, preventing the model from being penalized for good reasoning if the final answer was correct by chance.
• Comparison to SFT+RL: SubSearch achieved competitive performance against methods like InForage and O2-Searcher, which rely on expensive human-annotated reasoning trajectories, proving the efficacy of the unsupervised intrinsic reward approach.

5. Significance and Future Work

• Autonomous Reasoning: SubSearch moves the field toward autonomous information-intensive reasoning by eliminating the "bottleneck" of human annotation for training process rewards.
• Robustness: By penalizing flawed intermediate steps, the framework produces more robust and interpretable reasoning traces, reducing the risk of reward hacking.
• Efficiency: The approach is more data-efficient and scalable than supervised process modeling.

Limitations & Future Directions:

• Retriever Dependency: The answerability reward is conditioned on the quality of the external retriever. Future work aims to jointly optimize the generator and the retriever.
• Computational Cost: Calculating intrinsic rewards (embeddings, similarity) adds computational overhead during training. Future work will explore optimizing these calculations.

In summary, SubSearch represents a significant step forward in training LLM agents for complex search tasks by shifting from outcome-based supervision to intrinsic, process-based guidance, enabling more reliable and scalable reasoning without the need for costly human annotations.