🤖 AI

Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?

This paper reveals that Feedback-Augmented Self-Distillation often fails in retrieval-interleaved search agents due to a "decoding collapse" phenomenon and inconsistent supervision signals, which can be effectively mitigated by employing an Exponential Moving Average (EMA) teacher to stabilize the learning process.

Original authors: Fan Yang, Rui Meng, Yuxin Wen

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fan Yang, Rui Meng, Yuxin Wen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly confused, robot how to solve a mystery. The robot has a superpower: it can ask a librarian for help whenever it gets stuck. This is the world of "search agents"—computer programs that think, ask questions, and look up answers in a giant library of text to solve problems. Usually, to teach these robots, we use a "teacher" robot that is already an expert. The student robot watches the teacher, copies its moves, and tries to get better. But what if we don't have a super-expert teacher? What if the only teacher we have is the student robot itself, looking back at its own past attempts? This is called "self-distillation." It's like a student trying to learn from their own homework, hoping that if they remember how they solved a problem correctly once, they can do it again without help. Scientists are very excited about this because it could let robots learn from their own mistakes and successes without needing a human to grade every single step. But does this trick actually work when the robot has to search for information along the way? That is the big question this paper tackles.

The researchers in this study decided to test a specific version of this idea called "Feedback-Augmented Self-Distillation" (FA-SD). They set up an experiment where their AI agent, a robot named Qwen, tried to solve tricky questions by searching the internet. The idea was simple: if the robot got a question right in one attempt, they would show that successful attempt back to the robot as a "hint" or "feedback" to help it solve the same question again. They hoped this would act like a super-teacher, guiding the robot to make better choices. However, the results were surprising and a bit disappointing. Instead of getting smarter, the robot started to fail in a very specific, weird way.

The team discovered that the robot wasn't actually learning to think better. Instead, it fell into a trap they call "decoding collapse." Imagine a student taking a test who, instead of reading the question, just memorizes the shape of the answer sheet. They write down the same fancy-looking paragraphs and search queries over and over again, regardless of what the question actually is. To an outside observer, the answers look different and complex, but they are actually empty of real meaning. The robot was doing exactly this. It found a "template" for how to look like it was searching and thinking, and it just repeated that template. Because the robot was just copying a pattern rather than truly understanding the question, the "teacher" signal it was trying to learn from became useless. The math used to teach the robot (called KL divergence) couldn't tell the difference between a smart answer and a fake one, so the robot didn't improve.

The researchers also found that the problem wasn't that the "teacher" robot was too weak. In fact, when they gave the robot the "hint" (the successful past attempt) in its prompt, it could solve the problem much better than when it was left alone. The issue was that the robot couldn't "internalize" or absorb that skill. It was like a student who could ace a test if they had the answer key in front of them, but failed completely when the key was taken away. The robot couldn't translate the "hint" into a permanent skill.

To fix this, the team tried to make the "teacher" more stable. They realized that because the robot was learning and changing every second, the "teacher" version of itself was also changing, which made the lessons confusing. They tried two solutions: one where the teacher was frozen in time (a "fixed reference"), and another where the teacher was an "average" of the robot's past selves (called an "Exponential Moving Average" or EMA). The "frozen" teacher helped a little, but the "average" teacher was the real hero. By smoothing out the changes, the EMA teacher gave the robot a steady, consistent guide. This allowed the robot to recover from its earlier failures and actually learn. In the end, the robot using the EMA teacher scored an average of 0.206 on seven different tests, which was a big jump from its starting score of 0.114.

However, the paper also warns that this "hint" trick doesn't work for everyone. When they tried to use this same "feedback" idea with a powerful, external teacher robot (one that wasn't the student itself), it actually made things worse. It seems that adding extra hints to a teacher that is already strong just creates confusion. The study suggests that while self-teaching is a powerful idea, it's very fragile. If the robot starts copying templates instead of thinking, the whole system breaks. But with the right kind of steady guidance, like the EMA teacher, it is possible to help these search agents learn to think for themselves, even if they start out by just pretending to know the answer.

Technical Summary: Why Feedback-Augmented Self-Distillation Fails to Improve Retrieval-Interleaved Search Agents

Problem Statement

Retrieval-interleaved search agents, which alternate between reasoning and external tool use (search), face challenges in training due to the sparsity of outcome-level rewards. While Reinforcement Learning (RL) and dense supervision signals (e.g., process rewards) have been proposed, On-Policy Self-Distillation (OPSD) offers a promising alternative by using a single Large Language Model (LLM) as both student and teacher. In OPSD, the teacher is conditioned on "privileged information" (such as successful rollouts) to provide dense token-level supervision to the unconditioned student.

However, it remains unclear whether this paradigm extends effectively to retrieval-interleaved agents. Unlike standard language modeling, search agents generate trajectories where model actions determine subsequent retrieved observations (environment outputs), creating a mismatch between standard token-level distillation and the long-horizon search decisions required for effective retrieval. This paper investigates whether Feedback-Augmented Self-Distillation (FA-SD)—which uses successful rollouts as privileged demonstrations to condition the self-teacher—can provide reliable supervision for these agents.

Methodology

1. Feedback-Augmented Self-Distillation (FA-SD)

The authors instantiate FA-SD for search agents. The process involves:

Rollout Generation: Sampling a group of trajectories for a given question.
Privileged Feedback: If a successful trajectory exists within the group, it is used as feedback ( $f$ ) to augment the prompt for the "self-teacher."
Distillation Objective: The unconditioned student policy ( $\pi_\theta$ ) is trained to match the token distribution of the feedback-augmented self-teacher ( $\pi_\theta(\cdot | x, f)$ ). The loss function minimizes the divergence (e.g., KL divergence) between the student and the stop-gradient self-teacher, masking retrieved observations and focusing on assistant-generated tokens.

2. Diagnosing Failure Modes

To understand why FA-SD fails, the authors decompose the instability of the supervision signal into two components:

Model Inconsistency: The self-teacher and student share the same continuously updated parameters, causing the teacher target to drift during training.
Prompt Inconsistency: The teacher is conditioned on feedback demonstrations, while the student is not. This creates a distributional mismatch that may not transfer cleanly.

3. Proposed Mitigations

To address Model Inconsistency, the authors evaluate two regularization strategies for the self-teacher:

Fixed Reference Teacher: The teacher is frozen to the student's initial parameters at the start of training.
Exponential Moving Average (EMA) Teacher: The teacher parameters are updated as a moving average of the student parameters ( $\theta_{EMA} = \alpha \theta_{EMA} + (1-\alpha)\theta_t$ ), slowing the drift of the teacher target.

4. External Teacher Extension

The authors also test whether the feedback-augmentation trick generalizes to Multi-Rollout On-Policy Distillation (MOPD) with an external teacher (FA-MOPD), where the external teacher is conditioned on successful rollouts while the student remains unconditioned.

Key Findings and Results

1. Failure of Standard FA-SD

Across seven open-domain QA benchmarks (NQ, HotpotQA, etc.), standard FA-SD variants (unclipped and PPO-style clipped) fail to sustain improvement.

Training Dynamics: Pass rates remain near zero for most of the training, with only a transient rebound before regressing.
Decoding Collapse: The authors identify a specific failure mode termed decoding collapse. Models begin to rely on recurring reasoning-and-search output templates. These trajectories appear diverse but are largely agnostic to the input question. Consequently, the KL-based distillation signal becomes uninformative because the model is not learning to solve the specific question but rather mimicking a generic template. Standard aggregate metrics often miss this because they do not verify input-specificity.

2. The Self-Teacher is Capable, but Internalization Fails

Experiments show that the feedback-augmented self-teacher is often stronger than the unconditioned student at inference time. This rules out the hypothesis that the failure is due to a weak teacher. Instead, the issue is an internalization failure: the student cannot reliably absorb the token-level distillation signal derived from the privileged feedback into its unconditioned policy.

3. Impact of Regularization

Stabilization: Both Fixed Reference and EMA teachers stabilize training by preventing the teacher target from drifting too rapidly.
Performance: The EMA teacher yields the best results. It allows the model to recover from temporary regressions and achieves a higher final performance (average score of 0.206 across benchmarks) compared to the baseline (0.114).
Warm-up Phase: The EMA approach requires a warm-up phase where performance may temporarily regress before the supervision signal stabilizes and improves.

4. Generalization to External Teachers

When the feedback-augmentation trick is applied to MOPD with a strong external teacher (FA-MOPD), performance degrades compared to standard MOPD. This suggests that privileged feedback introduces prompt inconsistency or unnecessary conditioning even when a strong external teacher is present, and does not straightforwardly generalize to external-teacher OPD.

Key Contributions

Instantiation of FA-SD: The paper defines and implements Feedback-Augmented Self-Distillation for retrieval-interleaved search agents, utilizing successful rollouts as privileged demonstrations.
Identification of Decoding Collapse: The authors characterize a failure mode where models produce input-agnostic trajectories via recurring templates, rendering the distillation signal uninformative. This phenomenon is distinct from simple performance regression and is often missed by standard metrics.
Analysis of Supervision Inconsistency: The study decomposes the instability of FA-SD into model inconsistency (drifting teacher) and prompt inconsistency (conditioning mismatch). It demonstrates that stabilizing the teacher via EMA or fixed references is crucial for effective training.
Negative Results on Generalization: The paper shows that while strong external teachers provide stable early supervision in MOPD, adding privileged feedback to them (FA-MOPD) does not improve performance, indicating that feedback augmentation introduces specific inconsistencies in the search-agent setting.

Significance

The paper provides a critical diagnostic of why self-distillation, a promising technique for dense supervision, fails in the complex setting of retrieval-interleaved agents. It highlights that decoding collapse and supervision inconsistency are primary barriers, not just the lack of a strong teacher. By demonstrating that EMA regularization can stabilize the self-teacher and recover performance, the work offers a practical pathway for applying self-distillation to agentic tasks. Furthermore, it cautions against the naive application of feedback augmentation to external-teacher frameworks, suggesting that the benefits of privileged information are highly sensitive to the specific architecture and conditioning of the distillation setup.