Technical Summary: Adaptive Information Control for Search-Augmented LLM Reasoning

Problem Statement

Search-augmented reasoning agents, which interleave multi-step reasoning with external retrieval, face a critical bottleneck: uncontrolled information acquisition. While existing approaches rely on outcome-based reinforcement learning (RL) using sparse terminal rewards (e.g., final answer correctness), this supervision provides limited guidance for intermediate decisions regarding when to retrieve, how much to retrieve, and at what granularity to expose evidence.

Consequently, agents often exhibit suboptimal behaviors:

1. Over-retrieval: Accumulating redundant or noisy evidence, leading to context saturation and degraded decision quality.
2. Premature Stopping: Terminating search too early when additional evidence would be beneficial.
3. Training Instability: Sparse rewards make optimization sensitive to errors in long reasoning trajectories, particularly when starting from weaker base models.

Current methods typically address retrieval quality, stopping behavior, or evidence expansion in isolation, failing to treat information acquisition as a unified control problem under RL.

Methodology: DEEPCONTROL

The authors propose DEEPCONTROL, an adaptive information-control framework that regulates information acquisition along two complementary axes: extent (whether to continue retrieval) and resolution (how much detail to expose). The framework operates during online RL training and is designed to be internalized by the policy for autonomous operation at test time.

1. Information Utility

The core of the framework is Information Utility, a state-dependent estimate of the marginal value of retrieved evidence. It combines two metrics:

• Novelty: Measures the degree to which newly retrieved leaf nodes overlap with previously retrieved evidence (computed via semantic similarity in a shared embedding space).
• Effectiveness: Measures the improvement in the model's likelihood of generating the gold answer after injecting new evidence.
  The utility $U(e_l)$ for a search step $l$ is defined as:
  $$U(e_l) = \rho \cdot \text{Novelty}(e_l | s_{t_l}) + (1 - \rho) \cdot \text{Effectiveness}(e_l | u, s_{t_l})$$
  where $\rho$ balances the two components. This utility serves as a dense, intermediate training signal, contrasting with sparse outcome rewards.

2. Dual-Axis Control Mechanisms

The framework implements control through two specific mechanisms guided by the utility signal:

• Granularity Control (Resolution):
  Instead of injecting raw, full-length retrieved passages, the system uses a hierarchical selective expansion strategy. Retrieved sources are modeled as trees (e.g., Document $\to$ Section $\to$ Paragraph). The agent initially receives only root-level summaries. It then performs explicit expand actions to reveal finer-grained content. During training, the controller identifies high-utility leaf nodes and traces their ancestors to generate target expansion sets, supervising the model to prioritize high-value evidence while limiting context growth.

• Search Continuation Control (Extent):
  The system intervenes in the agent's decision to stop or continue searching based on utility thresholds:

  • Termination: If utility remains below a threshold $\delta_{stop}$ for $m_{stop}$ consecutive steps, a "Stop searching" signal is injected.
  • Continuation: If utility remains high ($\ge \delta_{cont}$) but the agent attempts to terminate while its confidence in the gold answer is low, a "Continue search" signal is injected.

3. Annealed Control-Forcing RL

To ensure the policy internalizes these behaviors rather than relying on external intervention at test time, DEEPCONTROL employs an annealed control-forcing scheme:

• Rollout Modes: During training, the agent alternates between a "Controlled" mode (where the controller injects signals) and an "Autonomous" mode (where the agent acts alone).
• Curriculum: The probability of the controlled mode ($p$) is gradually reduced (annealed) over training epochs. Early training uses frequent control to stabilize learning and guide exploration; later stages reduce control, forcing the policy to internalize effective acquisition behaviors.
• Reward Design: A composite reward combines an outcome-based F1 score with penalties for tool-usage violations and control non-compliance, ensuring the agent learns to follow the control signals during the training phase.

Key Contributions

1. Information Utility: A novel, state-dependent measure of the marginal value of retrieved evidence that combines novelty and effectiveness, serving as a practical signal for intermediate control.
2. Unified Control Framework: The introduction of two complementary control mechanisms—retrieval continuation control (regulating extent) and granularity control (regulating resolution)—treated as a unified problem under RL.
3. Annealed Internalization: A training strategy that gradually removes external control, enabling the policy to internalize effective information acquisition behaviors for reliable test-time performance without external intervention.

Experimental Results

The authors evaluated DEEPCONTROL on seven benchmarks covering General QA (NQ, TriviaQA, PopQA) and Multi-hop QA (HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle) using Qwen2.5-3B and Qwen2.5-7B models.

• Performance Gains: DEEPCONTROL consistently outperformed strong baselines, including Search-R1 (a state-of-the-art search-augmented RL method).
  • On Qwen2.5-7B, it improved average Exact Match (EM) by +9.4 points over Search-R1.
  • On Qwen2.5-3B, it improved average EM by +8.6 points.
• Training Stability: The framework demonstrated higher training rewards and stability compared to vanilla PPO and GRPO, particularly in preventing reward collapse and action-format degeneration.
• Behavioral Improvements: Analysis showed reduced "insufficient retrieval" and "missing supporting evidence" errors. The agent learned to adapt its search depth to task difficulty (e.g., fewer steps for simple QA, more for multi-hop).
• Robustness: The method remained effective when the underlying retriever was changed from E5 (dense) to BM25 (sparse), indicating the control mechanism is not tied to a specific retrieval backend.

Significance and Claims

The paper claims that adaptive information control is a critical, previously under-addressed component of search-augmented reasoning. The authors argue that simply improving retrieval quality or reasoning capacity is insufficient if the agent cannot effectively regulate how and when it consumes information.

By formulating information acquisition as a control problem regulated by utility, DEEPCONTROL addresses the "context saturation" and "redundant evidence" issues that plague current agents. The work demonstrates that explicit, adaptive control during training—specifically through the annealing of control signals—allows models to internalize sophisticated search strategies, leading to significant gains in reasoning accuracy and training stability without requiring external control at inference time. The authors position this as a step toward more robust, resource-efficient agents capable of operating in complex information environments.