Technical Summary: Optimizing Agentic Reasoning with Retrieval via Synthetic Semantic Information Gain Reward

1. Problem Statement

Large Reasoning Models (LRMs) have demonstrated the ability to integrate external knowledge retrieval into complex reasoning chains. However, optimizing the retrieval process within these agentic systems remains a fundamental challenge. Existing approaches suffer from three primary limitations:

1. Scalability Issues: Supervised fine-tuning (SFT) relies on human-annotated demonstrations of retrieval actions, which are costly and fail to capture the nuanced value of retrieval in open-ended scenarios.
2. Signal Sparsity: Reinforcement learning (RL) methods often rely solely on final-answer correctness (e.g., Exact Match). This results in sparse, delayed rewards that cannot distinguish between retrieval actions that are equally distant from the final answer or guide intermediate steps effectively.
3. Lack of Theoretical Grounding: While some works incorporate intermediate reasoning quality, they often rely on heuristic process supervision without a formal probabilistic framework guaranteeing that local retrieval decisions contribute to global uncertainty reduction.

The core problem addressed is how to measure and reward the intrinsic value of each retrieval action to guide an agent toward effective information gathering without requiring manual intermediate annotations.

2. Methodology: InfoReasoner

The authors propose InfoReasoner, a unified framework that incentivizes effective information seeking via a synthetic semantic information gain reward. The methodology bridges theoretical information theory with practical model optimization.

2.1 Theoretical Framework: Information Gain as Uncertainty Reduction

The reasoning process is modeled as a Partially Observable Markov Decision Process (POMDP). The authors redefine Information Gain (IG) not merely as data acquisition, but as the reduction of uncertainty over the model's belief states.

• Belief State ($b_t$): Defined as the posterior probability distribution over a latent task variable (e.g., the correct answer or its semantic class) given the history of observations.
• Uncertainty Functional ($U$): A mapping from belief states to non-negative real values (e.g., Shannon entropy), satisfying minimality, concavity, and expected monotonicity.
• Information Gain ($IG_t$): Defined as $U(b_t) - U(b_{t+1})$.
• Key Properties: The framework proves that under ideal Bayesian updates, IG is non-negative, telescopes additively (local gains sum to global uncertainty reduction), and is monotonic with respect to information channel quality. This establishes IG as a principled reward signal that aligns local retrieval steps with the global goal of reducing epistemic uncertainty.

2.2 Practical Implementation: Synthetic Semantic Information Gain

To make this theory tractable without manual annotations, InfoReasoner instantiates the reward using the model's own output distributions:

1. Semantic Clustering: The model samples multiple answer sequences under different contexts (with and without retrieved evidence). These sequences are clustered into semantic equivalence classes using bidirectional textual entailment (via a pretrained NLI model). This overcomes the limitations of token overlap or simple embedding similarity.
2. Belief Estimation: The probability distribution over these semantic classes approximates the belief state $b_t$.
3. Reward Computation:
   • Entropy-based IG ($\hat{IG}^{ent}_t$): Measures the reduction in semantic entropy when evidence is provided.
   • Gold-Class Gain ($r_t$): A more direct training signal computed as the log-probability increase of the correct semantic class ($c^\star$) after retrieval: $r_t = \log p(c^\star | \text{with evidence}) - \log p(c^\star | \text{without evidence})$.
   • This reward provides dense supervision for intermediate retrieval steps while remaining anchored to final-answer correctness.

2.3 Policy Optimization

The retrieval policy is optimized using Group Relative Policy Optimization (GRPO). The total reward $R_i$ for an output is a weighted sum of the final Exact Match (EM) score and the cumulative information gain rewards from intermediate steps:
$$R_i = \mathbb{I}(y_i = y^\star) + \lambda \cdot \frac{1}{T_i} \sum_{t} r_t(x_t, a_{t,i})$$
where $\lambda$ balances the incentive for accurate reasoning against the intermediate supervision for informative retrieval.

3. Key Contributions

1. Theoretical Framework: A redefinition of information gain as uncertainty reduction over belief states, establishing formal guarantees (non-negativity, telescoping additivity, monotonicity) for its use as a reward signal in agentic reasoning.
2. Practical Reward Mechanism: An output-aware training reward that computes synthetic semantic information gain directly from model outputs using bidirectional textual entailment, eliminating the need for manual intermediate retrieval annotations.
3. Empirical Validation: Demonstration of consistent performance improvements across diverse reasoning benchmarks, validating the dense semantic information gain as a practical training signal.
4. Scalable Optimization: A GRPO-based approach enabling efficient policy learning for agentic reasoning with retrieval.

4. Experimental Results

The authors evaluated InfoReasoner on seven question-answering benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, Bamboogle), as well as MATH500 (tool-integrated) and WebDetective (long-horizon search).

• Performance Gains: InfoReasoner achieved state-of-the-art results at both 3B and 7B parameter scales.
  • 3B Scale: InfoReasoner-3B achieved an average accuracy of 34.6%, significantly outperforming Search-R1-3B-Instruct (30.1%) and standard RAG (27.0%). It even surpassed several 7B-scale baselines.
  • 7B Scale: InfoReasoner-7B reached 39.1% average accuracy, outperforming AutoRefine-7B-Base (36.9%) and Search-R1-7B-Instruct (33.7%).
  • Generalization: On MATH500, InfoReasoner-7B reached 85.8% (vs. 84.6% for SimpleTIR-7B). On WebDetective, it improved Pass@1 from 26.5% to 30.0% over Search-R1-3B-Base.
• Ablation Studies:
  • The information gain coefficient $\lambda$ was critical; $\lambda=0.6$ yielded optimal performance. Setting $\lambda=0$ (degenerating to Search-R1) dropped performance by 4.7 percentage points.
  • Over-emphasizing IG ($\lambda=1.0$) degraded performance, suggesting a need to balance epistemic exploration with pragmatic exploitation.
• Training Dynamics: Analysis showed that InfoReasoner initially explores diverse retrieval strategies (indicated by higher entropy loss) before converging to a more certain policy. This exploration leads to a more robust retrieval strategy and higher final accuracy compared to baselines that converge prematurely. Additionally, InfoReasoner generated responses approximately 30% shorter than baselines, indicating more efficient information usage.

5. Significance and Claims

The paper claims that InfoReasoner provides a theoretically grounded and scalable path toward optimizing agentic reasoning with retrieval. Its significance lies in:

• Bridging Theory and Practice: It successfully translates the abstract concept of information gain into a computable, dense reward signal that does not require human annotation.
• Solving Signal Sparsity: By providing dense credit for retrieval steps, it overcomes the feedback delay inherent in final-answer-only RL, enabling more efficient learning of retrieval policies.
• Trustworthy Search: By grounding the model's motivation in "uncertainty reduction," the framework encourages agents to actively verify knowledge rather than hallucinating, which is critical for high-stakes applications.
• Efficiency: The framework promotes resource-efficient system designs by encouraging the model to search only when it lacks confidence, minimizing unnecessary API calls.

The authors emphasize that their approach moves beyond heuristic supervision to a formal probabilistic framework where local retrieval decisions are mathematically guaranteed to contribute to the global reduction of epistemic uncertainty.