HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation

HEAL is an RL-free distillation framework that overcomes the "Teacher Ceiling" of standard rejection sampling by synergizing entropy-guided repair, uncertainty filtering, and progressive curriculum learning to transfer reasoning capabilities from Large Reasoning Models to smaller students.

The Gist

Imagine you are trying to teach a brilliant but slightly stubborn math genius (the Teacher) how to explain their thought process to a bright but inexperienced student (the Student).

Usually, the way we do this is called "Rejection Sampling." The Teacher tries to solve a hard problem. If they get it right on the first try, we write down their steps. If they get stuck or make a mistake, we throw that problem away and try a new one.

The Problem: The Teacher is great, but they aren't perfect. For the hardest, "corner-case" problems, the Teacher often gets stuck. Because we throw away all the problems where the Teacher failed, the Student never gets to see how to solve the really hard stuff. The Student hits a "ceiling" and can never get smarter than the Teacher's best successful attempts.

The Solution (HEAL): This paper introduces a new method called HEAL (Hindsight Entropy-Assisted Learning). Instead of throwing away the hard problems, HEAL acts like a wise mentor who helps the Teacher "fix" their mistakes so the Student can learn from them.

Here is how HEAL works, broken down into three simple steps using a Garden Analogy:

1. GEAR: The "Garden Rescue" (Guided Entropy-Assisted Repair)

Imagine the Teacher is walking through a dense forest (the problem) trying to find a treasure. Suddenly, they hit a wall and stop. In the old method, we would say, "Okay, this path is dead, let's move on."

With GEAR, we have a special radar (Entropy) that detects exactly where the Teacher got confused. It's like a GPS that says, "You were doing great until you turned left at the big oak tree; that's where you got lost."

• The Fix: We give the Teacher a tiny hint (a "nudge") right at that moment of confusion, saying, "Actually, the path goes right here."
• The Result: The Teacher can now finish the path and write down the correct steps for a problem they originally couldn't solve. We turn "waste" data into valuable lessons.

2. PURE: The "Logic Inspector" (Perplexity-Uncertainty Ratio Estimator)

Now, imagine the Teacher, after getting the hint, writes down the solution. But sometimes, when people are given the answer, they cheat. They might write: "The answer is 42 because the answer is 42." This looks like a solution, but it's actually a "shortcut" with no real logic.

PURE is like a strict editor or a logic inspector. It checks the Teacher's notes step-by-step.

• The Check: It asks, "Did you actually think through this step, or did you just jump to the conclusion because you knew the answer?"
• The Fix: If the logic is shaky or looks like a cheat, PURE throws that specific note away. This ensures the Student only learns real reasoning, not magic tricks.

3. PACE: The "Curriculum Coach" (Progressive Answer-guided Curriculum Evolution)

Finally, we have to teach the Student. If you throw a beginner into a PhD-level thesis, they will panic and forget everything they already know.

PACE is the coach who organizes the lessons into three levels:

• Level 1 (Foundation): The Student practices on easy problems the Teacher solved perfectly on their own.
• Level 2 (Expansion): The Student practices on medium problems where the Teacher needed a little hint to finish.
• Level 3 (Frontier): The Student tackles the hardest "rescued" problems (the ones the Teacher was stuck on until GEAR helped).
• The Result: The Student builds a strong foundation first, so when they face the hardest problems, they are ready and don't get overwhelmed.

Why This Matters

In the past, students were limited by how many problems the Teacher could solve perfectly without help. HEAL breaks that limit. By helping the Teacher fix their own stuck moments, we create a much richer library of lessons.

The Bottom Line:
HEAL proves that you don't need a perfect Teacher to teach a perfect Student. You just need a smart system that knows how to help the Teacher when they get stuck, filter out the cheating, and teach the student in the right order. The result? The student learns to solve problems that were previously thought to be "too hard" for them.

Technical Summary

Here is a detailed technical summary of the paper "HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation."

1. Problem Statement: The "Teacher Ceiling"

Current methods for distilling reasoning capabilities from Large Reasoning Models (LRMs) to smaller student models primarily rely on Rejection Sampling. In this standard paradigm, a teacher model generates multiple reasoning trajectories for a given problem, and only those resulting in the correct answer are retained for training the student.

The Core Limitation:
This approach treats the teacher as a static filter. When the teacher encounters complex "corner-case" problems where it fails to independently find a valid solution, these problems are discarded as "unsolvable."

• Consequence: This creates an artificial "Teacher Ceiling." The student model is deprived of the most challenging data, limiting its learning potential to the teacher's successful exploration capabilities rather than its latent reasoning potential.
• Observation: Even with large sampling budgets (e.g., $N=64$), teachers fail to generate valid trajectories for a significant portion (e.g., 13%) of hard problems. The paper argues that these failures often represent "reasoning dead-ends" rather than a total lack of capability; the teacher simply needs a "nudge" to navigate the correct search space.

2. Methodology: The HEAL Framework

The authors propose HEAL (Hindsight Entropy-Assisted Learning), an RL-free framework inspired by the educational theory of the Zone of Proximal Development (ZPD). The framework aims to bridge the gap between a model's unaided capability and its guided capability through three synergistic modules:

A. GEAR (Guided Entropy-Assisted Repair)

• Goal: To synthesize valid reasoning trajectories for problems where the teacher initially fails.
• Mechanism:
  • Entropy Dynamics: Instead of random intervention, GEAR monitors the entropy gradients of the teacher's generation.
  • Breakpoint Detection: It identifies the "Critical Reasoning Breakpoint" ($t^*$), defined as the step with the maximum entropy surge within the first third of the trajectory. This marks the moment of "cognitive dissonance" where the teacher's logic diverges.
  • Targeted Repair: The system backtracks one step from the peak and injects a hindsight hint (conditioned on the ground-truth answer or specific intermediate steps) to guide the teacher back to a valid path.
• Significance: This transforms "waste data" (failed attempts) into valuable training signals by repairing broken logic chains exactly where the model struggles.

B. PURE (Perplexity-Uncertainty Ratio Estimator)

• Goal: To filter out "spurious shortcuts" (logical cheating) introduced by conditioning on the answer.
• Problem: When models are given the answer, they may generate syntactically coherent but logically disconnected rationales (e.g., jumping straight to the conclusion without derivation).
• Mechanism:
  • Suspicion Ratio ($R_t$): PURE calculates a ratio between the Step-wise Perplexity (PPL) and the Answer Uncertainty (NLL) at each step.
  • Logic Check: High PPL combined with low uncertainty indicates a "shortcut" where the model is forcing logic to fit a known answer.
  • Filtering: Trajectories with high "Anomaly Scores" (global maximum of the suspicion ratio) are pruned. This ensures only logically sound, step-by-step derivations are retained for distillation.

C. PACE (Progressive Answer-guided Curriculum Evolution)

• Goal: To organize the training data to prevent catastrophic forgetting and ensure stable learning.
• Strategy: A three-stage curriculum learning approach:
  1. Foundation Alignment: Train exclusively on standard, independently solved trajectories ($D_{base}$).
  2. Latent Expansion: Introduce globally hinted trajectories ($D_{hint}$) mixed with base data to expand the solution space.
  3. Frontier Breakthrough: Integrate the entropy-repaired trajectories ($D_{repair}$) to tackle the hardest corner cases.
• Significance: This progressive exposure allows the student to consolidate foundational skills before attempting complex, repaired reasoning paths.

3. Key Contributions

1. HEAL Framework: An interventionist distillation framework that applies ZPD principles to actively repair failed teacher trajectories, effectively breaking the "Teacher Ceiling."
2. GEAR Module: A novel entropy-guided mechanism that detects reasoning dead-ends and injects targeted hindsight hints, enabling the recovery of valid paths for intractable problems.
3. PURE Metric: A rigorous filtering protocol that decouples genuine reasoning breakthroughs from spurious shortcuts, ensuring high pedagogical quality in the distilled data.
4. PACE Strategy: A three-stage curriculum distillation strategy that stabilizes training and maximizes the transfer of complex reasoning capabilities.

4. Experimental Results

The authors evaluated HEAL on four challenging benchmarks: MATH 500, AIME 2024, AIME 2025, and OlympiadBench, using Qwen2.5-14B and Qwen3-4B as student models with Qwen3-32B as the teacher.

• Performance Gains:
  • On Qwen2.5-14B, HEAL achieved an average accuracy of 61.68%, outperforming standard SFT by 10.69% and the strongest baseline (Curriculum SFT) by a significant margin.
  • On AIME 2024, HEAL reached 53.63%, a 17.36% improvement over the best baseline.
  • HEAL demonstrated consistent improvements across both Base and Instruct models, whereas other methods (like LIMO) showed inconsistent performance depending on the model architecture.
• Ablation Studies:
  • Removing GEAR (relying only on global hints) caused a drastic regression (~11% drop on AIME 2024), proving that fine-grained, entropy-guided repair is essential for hard problems.
  • Removing PURE led to severe performance degradation, confirming that filtering out logical shortcuts is critical.
  • Removing PACE (using monolithic data mixing) caused a sharp decline, highlighting the necessity of progressive curriculum learning.

5. Significance and Impact

• Unlocking Latent Potential: HEAL demonstrates that teacher models possess latent reasoning capabilities that are inaccessible via standard rejection sampling. By recovering these trajectories, the student model can surpass the performance limits imposed by the teacher's independent success rate.
• Educational Theory in AI: The work successfully translates human pedagogical concepts (ZPD, scaffolding, curriculum learning) into technical mechanisms for LLM distillation.
• RL-Free Efficiency: Unlike Reinforcement Learning (RL) approaches which can be unstable and computationally expensive, HEAL offers a stable, efficient, and controllable pathway to achieve competitive reasoning performance in smaller models.
• Complementarity: The framework provides a high-quality "cold-start" initialization for future RL training, offering a complementary advantage to existing RL-based methods.

Limitations: The method relies on the availability of ground-truth answers (limiting it to closed-ended tasks like math/coding) and assumes the problem lies within the teacher's ZPD. It also incurs offline computational overhead for calculating perplexity and NLL during the filtering phase.