Technical Summary: EviSD – Evidence-Conditioned Self-Distillation for Search-Augmented Agents

Problem Statement

Search-augmented language agents utilize external retrieval to interleave reasoning with information gathering. While recent outcome-based reinforcement learning (RL) methods, such as Group Relative Policy Optimization (GRPO), enable training via verifiable final answers, they suffer from a fundamental limitation in credit assignment. Outcome-level rewards broadcast the same trajectory advantage to every generated token. Consequently, the system cannot distinguish between:

• Effective search queries that retrieved decisive evidence.
• Redundant or vague searches that provided no value.
• Intermediate reasoning tokens versus terminal answer tokens.

This "outcome-only" feedback fails to localize instance-specific decisions, making it difficult to optimize the specific actions (search or answer) that led to success or failure, particularly in multi-turn scenarios where early queries dictate the quality of subsequent reasoning.

Existing approaches attempt to refine credit granularity through retrieval-specific rewards or step-level estimators, but they often lack a mechanism to determine what instance-specific information should guide a sampled action. Conversely, on-policy self-distillation (OPSD) methods introduce privileged teacher contexts but often rely on trajectory-derived hindsight or task-level skills, which may not directly identify the specific evidence required for a given query.

Methodology: EviSD

The authors propose EviSD (Evidence-Conditioned Self-Distillation), a framework that uses instance-level supporting evidence as privileged information to refine outcome-derived credit. The method operates under two core design decisions:

1. Instance-Level Privileged Context Construction

EviSD treats supporting evidence (annotated in training data) as privileged information for search actions and golden answers for answer actions.

• For Search Actions: The teacher model is conditioned on the annotated supporting evidence ($E_q$) from the current QA instance. This provides a view of what a useful query should uncover without prescribing a specific query string.
• For Answer Actions: The teacher is conditioned on the golden answer ($C_q$).
• Action Alignment: The privileged context is prepended to the original context only when the student generates a search or answer action. For other tokens (reasoning, delimiters), the context remains unchanged.

2. Action-Localized, Outcome-Anchored Credit Modulation

Rather than optimizing a separate distillation loss, EviSD translates the teacher-student gap into a bounded correction to the GRPO advantage.

• Teacher-Student Contrast: The same policy acts as both student (sampling from the original context) and teacher (re-scoring the same action under the privileged context). The signal is the detached log-likelihood ratio ($\delta$) between the two views.
• Localization: The modulation applies only to the content tokens of generated search and answer actions ($m^{act}$). Reasoning tokens and retrieved passages retain their original GRPO credit.
• Modulation Function: The privileged signal modulates the magnitude of the outcome-derived advantage ($A$) via a bounded function:
  $$\Delta A^{priv} = \lambda |A| \tanh(\tau \delta) m^{act}$$
  $$\hat{A} = A + \Delta A^{priv}$$
  Here, $\lambda$ controls the maximum relative correction, and $\tau$ controls sensitivity. Crucially, for $\lambda < 1$, the correction cannot reverse the sign of the original advantage. The outcome reward remains the anchor for the update direction; privileged information only refines the strength of the credit for specific action tokens.

Training and Inference

• Training: The model samples trajectories, computes outcome rewards, constructs privileged contexts for search/answer turns, re-scores actions to generate the teacher-student gap, and updates the policy using the modulated advantage $\hat{A}$ within the standard clipped GRPO objective.
• Inference: The deployed agent operates exactly as before. Privileged contexts and the teacher view are removed, requiring no changes to the inference pipeline or additional latency.

Key Contributions

1. Evidence-Conditioned Framework: EviSD introduces a novel use of instance-level supporting evidence as privileged teacher context for search actions, complemented by golden answers for answer actions.
2. Action-Localized Modulation: The paper proposes a credit modulation rule that converts the detached teacher-student gap into a bounded correction for GRPO advantages. This approach requires no auxiliary distillation loss and avoids inference-time changes.
3. Performance and Robustness: The method achieves state-of-the-art results across seven QA benchmarks and three model backbones (Qwen2.5-3B, 7B, and Qwen3-1.7B), outperforming the strongest baselines by 1.3–2.3 macro-average Exact Match (EM) points.

Experimental Results

The authors evaluated EviSD on seven datasets (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, Bamboogle) covering single-hop and multi-hop reasoning.

• Overall Performance: EviSD achieved the highest macro-average EM in all evaluated settings. On the Qwen2.5-7B backbone, it outperformed the second-best method (SDAR) by 1.3–2.3 points.
• Search Efficiency: EviSD demonstrated higher success rates with fewer searches (1.87 searches per trajectory vs. 2.16 for SDAR), indicating more precise guidance of search actions.
• Ablation Studies:
  • Privilege Source: Replacing instance-specific evidence with answer-only context reduced performance by 2.1 EM points, confirming the value of evidence for supervising search actions.
  • Localization: Applying modulation to the full response (rather than just action spans) reduced performance by 7.1 points, validating the necessity of localizing guidance to action tokens.
  • Training Dynamics: EviSD maintained a smooth on-policy success rate curve and lower KL divergence compared to competitors, suggesting stable policy improvement without abrupt deviations.
• Sensitivity: The method showed limited sensitivity to the quality of privileged evidence; replacing gold evidence with relevant retrieved documents resulted in negligible performance degradation.

Significance and Claims

The paper claims that EviSD addresses the critical gap in outcome-based RL for search agents by localizing credit assignment without sacrificing the scalability of outcome rewards. By treating supporting evidence as privileged information, the framework guides the agent to identify what information is necessary for a query without dictating the query itself.

The authors emphasize that their approach is distinct from prior self-distillation methods because it:

1. Uses instance-specific evidence rather than trajectory-derived hindsight or reusable skills.
2. Modulates outcome credit rather than optimizing a separate distillation objective.
3. Preserves the update direction determined by the outcome reward while refining the magnitude of credit for specific actions.

Ultimately, EviSD demonstrates that refining credit assignment through evidence-conditioned self-distillation can significantly improve search-augmented reasoning capabilities across different model scales and generations, all while maintaining a deployment-ready inference architecture.