Technical Summary: AMR-SD

Problem Statement

The alignment of Large Language Models (LLMs) for complex reasoning tasks increasingly relies on Reinforcement Learning with Verifiable Rewards (RLVR). Standard algorithms, such as Group Relative Policy Optimization (GRPO), estimate advantages from sparse sequence-level outcomes but apply a uniform advantage scalar to all tokens within a rollout. This creates a severe credit-assignment bottleneck, where critical logical deductions and superficial stylistic choices receive identical reward signals, hindering the optimization of reasoning tasks.

To address this, the community has explored On-Policy Self-Distillation (OPD), where the model acts as its own teacher conditioned on privileged information (e.g., ground-truth solutions). However, existing approaches suffer from three primary limitations:

1. Information Leakage and Over-Conditioning: Directly exposing the teacher to raw oracle solutions induces over-conditioned distributions and implicit answer leakage, degrading the student's intrinsic reflective capabilities.
2. Signal Dampening: Because the privileged context narrows the teacher's output distribution onto specific oracle tokens, the teacher's probabilities for student-generated tokens are often systematically lower. This causes continuous scaling multipliers to be predominantly less than 1, dampening the primary reward signal.
3. Late-Stage Collapse: The combination of information asymmetry and signal dampening often leads to training collapse in later stages, where the model fails to generalize or stabilize.

Methodology: AMR-SD

The authors propose Asymmetric Meta-Reflective Self-Distillation (AMR-SD), a framework designed to resolve the credit-assignment bottleneck while preventing information leakage and signal dampening. The method operates through three sequential phases:

1. Socratic Rescoring via Meta-Reflection

Instead of conditioning the teacher directly on raw ground-truth solutions, AMR-SD introduces a Meta-Reflection phase. The model generates concise, natural-language "Socratic" targets based on the trajectory outcome:

• Hints: For successful trajectories, the model generates a hint that distills latent reasoning principles without revealing the final answer.
• Critiques: For failed trajectories, the model generates a targeted critique that diagnoses logical or syntactic divergences.
• Mechanism: These reflections act as a low-bandwidth bottleneck between diagnostic sources (verifier outcomes, peer rollouts) and the token-rescoring teacher. This prevents the teacher from seeing raw oracle traces, thereby mitigating over-conditioning and answer leakage while preserving essential guidance.

2. Causal Information Gain (CIG) Quantification

To translate these reflections into token-level advantage modulations, the authors introduce Causal Information Gain (CIG). CIG measures the pointwise log-likelihood ratio between the context-aware Socratic teacher (conditioned on the reflection) and the student policy:
$$ICIG_t = \log \left( \frac{\pi_{\theta_{sg}}(a_t | s_t, c_i)}{\pi_{\theta}(a_t | s_t)} \right)$$
where $c_i$ is the generated reflection. To ensure numerical stability and bounded Lipschitz continuity, this value is symmetrically clamped to a range $[-\kappa, \kappa]$.

3. Asymmetric Token-Level Advantage Modulation

The core innovation lies in how CIG is applied. Rather than continuously scaling the advantage, AMR-SD employs an asymmetric, ReLU-gated threshold ($\tau$):

• Sparsity: The mechanism acts as a strict gatekeeper. It preserves the baseline environmental reward by default and only triggers adjustments when the CIG signal exceeds the threshold $\tau$.
• Asymmetry: The modulation is decoupled for positive and negative rollouts using distinct coefficients ($\lambda$ for hints, $\gamma$ for critiques).
  • For successful rollouts ($A_i \ge 0$), the advantage is amplified only if the student underperformed the teacher ($CIG_t > \tau$).
  • For failed rollouts ($A_i < 0$), the penalty is amplified only if the student was erroneously over-confident ($CIG_t < -\tau$).
• Temporal Annealing: To prevent overfitting to degraded reflection signals in late training stages, the modulation coefficients ($\lambda, \gamma$) are linearly decayed to zero, smoothly shifting the optimization focus entirely to the pure environmental reward.

Key Contributions

1. Meta-Reflection Paradigm: The introduction of a self-generated reflection bottleneck (hints/critiques) that replaces direct oracle conditioning. This reduces information asymmetry and improves the calibration of token-level credit signals without discarding privileged information sources.
2. Causal Information Gain (CIG): A novel metric combined with an asymmetric, threshold-gated (ReLU) mechanism. This enables sparse, precise token-level advantage modulations that filter out distributional noise while preserving the base reinforcement signal.
3. AMR-SD Framework: A complete training framework integrating Meta-Reflection, CIG, and temporal annealing. It achieves robust long-horizon stability and successfully prevents the late-stage training collapse observed in prior self-distillation methods.

Experimental Results

The authors evaluated AMR-SD across scientific reasoning, mathematical reasoning, and tool-use benchmarks using Qwen3-8B and Qwen2.5-7B-Instruct backbones.

• Scientific Reasoning & Tool Use: On the SciKnowEval benchmark (Chemistry, Physics, Biology, Materials) and ToolAlpaca, AMR-SD achieved the highest or near-highest scores at convergence (step 75). Notably, it significantly outperformed GRPO and RLSD on Biology tasks (e.g., 63.7 vs. 56.5 for GRPO on Qwen3-8B) and demonstrated superior stability, avoiding the performance regression seen in baselines like RLSD and SDPO.
• Mathematical Reasoning: On AIME 24/25, AMC 23, and HMMT benchmarks, AMR-SD achieved the best average score (62.7), outperforming GRPO (60.3) and RLSD (57.2). The gains were most pronounced on complex multi-step tasks (e.g., +2.9 on HMMT over GRPO), suggesting better generalization for complex reasoning chains.
• Training Dynamics: Unlike baselines that exhibit early gains followed by plateaus or collapse, AMR-SD showed a distinct convergence profile: it sometimes lagged in early convergence (trading speed for stability) but achieved superior long-run optimization and avoided the severe policy collapse seen in SDPO.
• Ablation Studies: Removing Meta-Reflection (reverting to direct ground-truth conditioning) reduced accuracy by 3.8 points. Removing the CIG threshold or the asymmetric ReLU gate also degraded performance, confirming the necessity of sparsity and directional filtering.

Significance and Claims

The paper claims that AMR-SD provides a highly robust and scalable solution for aligning LLMs on complex reasoning and agentic tasks. By effectively filtering distributional noise and utilizing temporal annealing, the framework guarantees long-horizon stability and prevents late-stage training collapse.

Crucially, the authors argue that AMR-SD improves reasoning efficiency by reducing unnecessary verbosity while successfully preserving critical cognitive explorations (e.g., self-correction markers). This confirms that introducing a reflection bottleneck successfully mitigates the vulnerabilities inherent in direct answer conditioning, translating training stability into superior out-of-domain accuracy.

Limitations Acknowledged:
The authors note that AMR-SD's effectiveness is contingent on the quality of the model's self-generated meta-reflections. On models with weak introspective capabilities (e.g., non-thinking mode models), the reflection quality may be insufficient to yield reliable CIG signals, causing the method to underperform compared to GRPO. Additionally, the meta-reflection phase introduces a moderate inference overhead during training rollouts.