Technical Summary: SAAS – Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Search

1. Problem Statement

Agentic search systems, which enable Large Language Models (LLMs) to solve complex multi-hop questions through iterative reasoning and external retrieval, face a critical limitation in practice: over-search. Despite their effectiveness, these agents often fail to recognize their own knowledge boundaries. This manifests in two specific ways:

1. Unnecessary Search: Agents trigger search queries even when their internal parametric knowledge is sufficient to answer the question.
2. Redundant Search: Agents continue to search even after adequate external evidence has already been collected to form a correct answer.

The lack of self-awareness regarding these boundaries leads to severe over-search, incurring substantial inference latency and prohibitive computational costs. Existing approaches, including prompt-based routing and reinforcement learning (RL) with outcome-based rewards, suffer from three key limitations:

• They rely on static heuristics or fixed thresholds that cannot capture the agent's evolving capabilities during training.
• Outcome-only rewards provide no signal regarding when to initiate or terminate search, leaving the agent "blind" to search necessity.
• Naively adding search penalties often triggers reward hacking, where agents learn to avoid search entirely (even when necessary) to minimize penalties, rather than developing genuine boundary awareness.

2. Methodology: The SAAS Framework

The authors propose SAAS (Self-Aware Reinforcement Learning for Agentic Search), a novel RL framework designed to cultivate dynamic self-awareness. SAAS regulates search behavior without compromising accuracy through three core components:

(i) Search Boundary Modeling

Instead of relying on static annotations, SAAS dynamically identifies the search boundary for each question under the evolving policy. It does this by contrasting two groups of rollouts generated by the current policy $\pi_\theta$:

• Search-Disabled Group ($G_d$): The agent attempts to answer using only parametric knowledge.
• Search-Enabled Group ($G_e$): The agent is allowed to interact with a search engine.

By comparing the success rates of these groups, the system classifies each question $q$ into one of three states:

• NOSEARCH: The model can reliably answer without external evidence (indicated by a sufficient number of correct trajectories in $G_d$).
• NEEDSEARCH: The model requires external evidence (indicated by $G_d$ failing but $G_e$ succeeding).
• UNDETERMINED: The boundary is unclear based on current rollouts.

(ii) Boundary-Aware Reward Module

SAAS translates the identified search boundary into trajectory-level penalties to suppress unnecessary and redundant searches selectively:

• For NOSEARCH questions: A zero-tolerance penalty is applied to any search action ($R_{search} = -\alpha N_i$), discouraging the agent from searching when it shouldn't.
• For NEEDSEARCH questions: Search itself is not penalized. Instead, the system calculates $N_{min}$, the minimum number of search actions required by correct trajectories in the search-enabled group. Only search actions exceeding this threshold are penalized ($R_{search} = -\alpha \max(0, N_i - N_{min})$). This targets redundant searches specifically.
• For UNDETERMINED questions: No additional constraints are applied to avoid restricting reasoning.

To ensure stable training, the framework employs group-wise advantage normalization within the GRPO (Group Relative Policy Optimization) algorithm to handle distributional mismatches between the search-disabled and search-enabled groups.

(iii) Stage-Wise Optimization Strategy

To prevent reward hacking (where agents avoid necessary searches before learning tool-use competence), SAAS employs a sequential curriculum:

• Stage 1 (Capability Acquisition): The agent is trained using only outcome-based rewards (answer correctness) to learn basic reasoning and tool-use capabilities without search constraints.
• Stage 2 (Efficiency Refinement): Once the agent demonstrates competence, the boundary-aware reward is activated to refine search behavior, suppressing over-search while preserving accuracy.

3. Key Contributions

The paper claims the following contributions:

1. Identification of Limitations: The authors identify that existing agentic search models lack dynamic self-awareness, leading to over-search, and that static constraints fail to adapt to evolving model capabilities.
2. SAAS Framework: They propose a novel RL framework that dynamically models the agent's search boundary by contrasting search-disabled and search-enabled rollouts.
3. Boundary-Aware Optimization: SAAS establishes a mechanism to translate this boundary awareness into trajectory-level penalties that suppress unnecessary and redundant searches without triggering reward hacking, achieved through a stage-wise optimization strategy.
4. Empirical Validation: Extensive experiments demonstrate that SAAS consistently mitigates over-search while maintaining or improving answer accuracy across various benchmarks and model backbones.

4. Experimental Results

The authors evaluated SAAS on seven open-domain QA benchmarks (TriviaQA, PopQA, NQ, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle) using Qwen2.5-3B and Qwen2.5-7B models.

• Accuracy vs. Efficiency: SAAS achieved the best average accuracy on Qwen2.5-3B (45.8%), surpassing the strongest baseline (HiPRAG) by 2.2%, while reducing the average search count from 1.89 (HiPRAG) to 1.13. On Qwen2.5-7B, it achieved 48.7% accuracy with an average search count of 0.97, compared to 2.19 for HiPRAG.
• Over-Search Reduction:
  • Question-Level Over-search Ratio (QOR): SAAS reduced the average QOR to 45.9% (compared to 52.5% for RFT and 100% for StepSearch/HiPRAG), indicating the model successfully leverages parametric knowledge when sufficient.
  • Step-Level Over-search Ratio (SOR): SAAS achieved the lowest average SOR of 6.3% (compared to 14.8% for RFT and 24.3% for StepSearch), demonstrating its ability to stop searching once sufficient evidence is collected.
• Ablation Studies: Removing stage-wise optimization led to a significant drop in accuracy (from 45.8% to 40.9%), confirming that early search regularization hinders tool-use learning. Removing on-policy boundary modeling also degraded performance, validating the need for dynamic boundary tracking.

5. Significance and Claims

The paper posits that the significance of SAAS lies in its ability to dynamically model the agent's evolving search boundary. Unlike prior methods that rely on static heuristics or fixed penalties, SAAS aligns search decisions with the model's actual, changing competence.

The authors claim that this approach allows agents to:

1. Distinguish necessity: Recognize when internal knowledge is sufficient to avoid unnecessary tool calls.
2. Determine sufficiency: Recognize when collected evidence is enough to stop redundant retrieval.
3. Stabilize training: Avoid the "reward hacking" common in naive penalty-based RL by prioritizing capability acquisition before efficiency refinement.

Ultimately, the paper argues that dynamic search boundary awareness is essential for building more efficient and effective search-augmented reasoning systems, reducing computational costs without sacrificing the quality of reasoning. The authors note that while the current evaluation focuses on text-based retrieval, the core mechanism is modality-agnostic and could be extended to multimodal reasoning in future work.