Technical Summary: KbSD – Knowledge Boundary Aware Self-Distillation for Behavioral Calibration in Agentic Search

Problem Statement

Agentic search systems empower Large Language Models (LLMs) to dynamically retrieve information, yet current reinforcement learning (RL) approaches face a fundamental bottleneck: reward sparsity in knowledge boundary calibration. While existing methods can determine when to retrieve, they struggle to guide the agent on how to reconcile conflicting knowledge sources (parametric memory vs. retrieved evidence) or when to abstain.

The core issue is that binary or sparse terminal rewards only penalize undesirable outcomes (e.g., hallucinations) without providing guidance on the intermediate reasoning process. This is particularly problematic across four distinct knowledge-boundary quadrants defined by internal certainty and retrieval quality:

1. Integrated: Both sources reliable.
2. External: Retrieval reliable, internal uncertain.
3. Refusal: Both sources unreliable.
4. Internal: Internal reliable, retrieval unreliable.

Sparse rewards fail to decompose the cognitive steps required to distinguish principled abstention from uninformed guessing (Quadrant 3) or to guide the integration of concordant evidence (Quadrant 1). Furthermore, a uniform training objective is ill-suited for these heterogeneous states, as they exhibit different distributional characteristics (e.g., concentrated integration vs. diverse refusal patterns).

Methodology: KbSD Framework

The authors propose KbSD (Knowledge Boundary Self-Distillation), a framework that addresses reward sparsity through dense token-level supervision and quadrant-adaptive optimization. The pipeline consists of three stages:

1. Boundary-Oriented Data Synthesis

To train models near their knowledge limits, the authors synthesize challenging multi-hop queries from seed datasets (HotpotQA, 2WikiMultiHopQA) using two operators:

• Fact Injection: Adds compositional depth by introducing new fact hops, forcing the model to combine evidence.
• Information Fuzzing: Replaces precise conditions with underspecified descriptions to increase retrieval difficulty and ambiguity.

2. Knowledge Boundary Quantification & Quadrant Mapping

For each query, the system quantifies the knowledge state using:

• Parametric Certainty ($\mu$): The fraction of correct answers generated by the model without retrieval.
• Semantic Stability ($\sigma$): The consistency of the model's internal responses (semantic similarity).
• Retrieval Quality ($\hat{\rho}$): The estimated sufficiency of external evidence.

These signals map each query to one of four behavioral quadrants (Integrated, External, Refusal, Internal), defining the target behavioral norm.

3. Hint-Augmented Self-Distillation

KbSD employs an information-asymmetric self-distillation design:

• Teacher Model: A copy of the student model that receives privileged hints ($h(q)$) containing parametric certainty, stability, retrieval quality, and the target quadrant. The teacher generates calibrated reasoning demonstrations (e.g., how to inspect evidence before refusing).
• Student Model: Trained on the original input without hints, learning to internalize the teacher's calibrated reasoning patterns through token-level supervision.

4. Quadrant-Adaptive Distillation Objectives

To address the heterogeneous optimization geometry of the four quadrants, KbSD applies specific divergence objectives:

• Quadrant 1 (Integrated): Uses Reverse KL (mode-seeking) because correct reasoning is concentrated around a single integration pattern.
• Quadrant 3 (Refusal): Uses Forward KL (mass-covering) because valid refusal behaviors are diverse; the model should not collapse to a single template.
• Quadrants 2 & 4 (External/Internal): Uses Pareto-optimal Bidirectional KL, adaptively weighting forward and reverse KL to balance precision (source selection) and coverage (robustness to noise).

This distillation loss is jointly optimized with GRPO-based Reinforcement Learning, which provides sparse trajectory-level rewards for final task correctness, while distillation provides dense process-level supervision.

Key Contributions

1. Identification of Reward Sparsity: The paper identifies the lack of process-level supervision as a critical bottleneck in knowledge boundary calibration, particularly for distinguishing between different reasoning strategies in heterogeneous knowledge states.
2. Information-Asymmetric Self-Distillation: A novel framework that constructs a hint-augmented teacher to generate dense, token-level reasoning demonstrations without requiring a larger external model. This bridges the gap between sparse behavioral labels and the full cognitive process.
3. Quadrant-Adaptive Optimization: A distillation strategy that aligns the optimization geometry with the specific knowledge state, utilizing Reverse KL, Forward KL, and Pareto-optimal Bidirectional KL to handle concentrated, diverse, and asymmetric distributions respectively.

Experimental Results

Experiments were conducted on nine benchmarks (including HotpotQA, NQ, TriviaQA, FRAMES, and GAIA) using Qwen2.5 models (3B and 7B).

• Performance: KbSD consistently achieved the lowest Unreliability Rate (defined as $1 - (\text{F1} + \text{Refusal Rate})$) across all benchmarks and model scales, outperforming strong baselines including Search-R1, AEPO, and the sparse-reward baseline KbPO.
• Scale Efficiency: Improvements increased with model scale. On Qwen2.5-7B, the average unreliability rate dropped from 57.99% (KbPO) to 41.18% (KbSD).
• Generalization: Despite training only on synthesized multi-hop QA data, KbSD generalized effectively to out-of-domain agentic reasoning tasks (FRAMES, GAIA).
• Ablation Studies: Removing the distillation loss or using fixed (non-adaptive) KL objectives significantly degraded performance, confirming the necessity of both dense supervision and quadrant-adaptive divergence.
• Case Studies: Analysis showed KbSD successfully learned to abstain on invalid questions (Quadrant 3) where sparse-reward baselines hallucinated, and correctly integrated evidence in Quadrant 1. However, a residual challenge remains in Quadrant 4 (Internal), where models occasionally over-rely on retrieval even when internal knowledge is sufficient.

Significance and Claims

The paper claims that KbSD fundamentally shifts the paradigm of agentic search training from relying solely on outcome-based sparse rewards to leveraging dense, process-level supervision derived from self-distillation.

The authors assert that the primary bottleneck in calibrating agentic behavior is not model capacity, but the sparsity of process-level supervision. By making latent behavioral targets explicit through a hint-augmented teacher and adapting the optimization objective to the specific knowledge state, KbSD enables models to learn how to calibrate trust between internal and external knowledge. The results suggest that this approach is particularly effective in "challenging quadrants" (where sparse rewards are least informative), significantly reducing hallucination rates while maintaining helpfulness on tractable tasks. The work positions knowledge boundary awareness as a domain-agnostic skill that can be learned through this self-distillation mechanism.