Technical Summary: Critique-Guided Distillation for Robust Reasoning via Refinement

Problem Statement

Supervised Fine-Tuning (SFT) remains the dominant approach for adapting Large Language Models (LLMs) to downstream tasks. However, standard SFT often results in models that imitate expert outputs without internalizing the underlying reasoning processes, leading to poor out-of-distribution generalization, increased hallucination, and difficulty scaling to harder problem instances.

Alternative paradigms leveraging critique and revision (e.g., Self-Refine, Reflexion) show promise but incur substantial inference costs due to multi-pass prompting. Critique Fine-Tuning (CFT) attempts to move critique generation into the training phase but introduces significant drawbacks: training models to generate critiques often leads to output-format drift and catastrophic forgetting of general capabilities (e.g., instruction following). Furthermore, existing distillation methods often condition on generic teacher explanations of correct solutions rather than addressing specific student errors, or rely on implicit supervision signals (like token probabilities) that lack actionable diagnostic information.

Methodology: Critique-Guided Distillation (CGD)

The authors propose Critique-Guided Distillation (CGD), a training framework that decouples critique consumption from critique generation. The core intuition is that a student model can internalize self-correction capabilities by training on teacher-provided critiques of its own errors, without needing to generate those critiques itself.

Training Procedure

The CGD pipeline consists of three stages per training sample:

1. Student Answer Generation: The student model ($S_{\theta_{init}}$) generates an initial, potentially flawed response ($y'$) to a prompt ($x$).
2. Critique Generation: A teacher model ($T_\phi$) analyzes $y'$ and generates a textual critique ($c$) identifying specific errors, misconceptions, or gaps in reasoning.
3. Refined Answer Generation: The teacher model produces a gold-standard, refined answer ($\hat{y}$) conditioned on the prompt, the student's initial answer, and the critique.

The student is then fine-tuned on the augmented dataset $((x, y', c), \hat{y})$ using a standard language modeling objective to minimize the negative log-likelihood of the refined answer:
$$\mathcal{L}(\theta) = \mathbb{E}_{(x,y',c,\hat{y})}[-\log S_\theta(\hat{y} \mid x, y', c)]$$

Inference Behavior

Crucially, critiques are used only during training. At inference time, the model receives only the prompt ($x$) and generates a refined answer in a single forward pass. The model does not generate critiques, nor does it require them as input. This design ensures zero architectural overhead and maintains the standard inference format, avoiding the format drift observed in CFT.

Key Contributions

1. Novel Training Framework: Introduction of CGD, which trains models to consume explanatory feedback to refine their own responses, rather than training them to generate critiques.
2. Decoupling of Roles: The framework separates the roles of critique generation (teacher) and critique consumption (student), allowing the student to internalize error-aware reasoning without the inference-time latency of self-correction loops.
3. Preservation of General Capabilities: Unlike CFT, which degrades instruction-following abilities, CGD preserves general capabilities while significantly boosting reasoning performance.
4. Cross-Family Generalization: The method demonstrates robustness across diverse model families (LLaMA, Qwen, S1.1, Mixtral, OLMo) and scales, achieving state-of-the-art results even with open-source teachers.
5. Empirical Analysis: Comprehensive ablation studies characterizing the role of critique specificity, relevance, and correctness, as well as diagnostic analyses of internal attention mechanisms and gradient norms.

Experimental Results

The authors evaluated CGD across five model families and multiple benchmarks, comparing it against Standard SFT, Distilled SFT, and Critique Fine-Tuning (CFT).

• Mathematical Reasoning: CGD consistently outperforms baselines. On LLaMA3.1-8B, it achieved a +5.4% average improvement on math reasoning benchmarks over CFT, with specific gains of +15.0% on AMC23 and +12.2% on MATH-500. On the S1.1-3B model, gains reached +7.2% over CFT.
• General Capabilities: While CFT caused a catastrophic drop in instruction-following performance (IFEval accuracy dropped from 76.9% to 55.6%), CGD maintained performance (76.1%), effectively preserving general capabilities.
• Out-of-Distribution Transfer: Despite being trained on data containing no code, CGD models transferred to code generation tasks (HumanEval), achieving a +4.9% improvement over the base model, outperforming CFT.
• Reasoning Quality vs. Search: On challenging competition benchmarks (AIME24), CGD showed substantial gains at low sampling budgets (Pass@1 improved from 3.3% to 16.7%), indicating improved per-sample reasoning quality rather than reliance on extensive search. The models also produced significantly longer reasoning chains (4.4x longer on AIME).
• Ablation Insights:
  • Critique Necessity: Removing the critique from the training context caused performance degradation of up to 114% on AMC23, confirming the critique acts as a critical learning scaffold.
  • Specificity and Relevance: Specific, relevant critiques outperformed generic placeholders. Irrelevant critiques (specific but wrong for the problem) actively misled the model, performing worse than generic placeholders.
  • Robustness: Models trained on a balanced mixture of correct and incorrect student answers (50/50) showed the highest robustness, developing an internal ability to distinguish valid from invalid guidance.

Significance and Claims

The paper positions CGD as a practical and compute-efficient intermediate training paradigm for reasoning-centric tasks. Its primary significance lies in achieving substantial reasoning improvements without introducing inference-time overhead or architectural changes.

The authors claim that CGD successfully addresses the limitations of both standard SFT (shallow imitation) and CFT (format drift and capability degradation). By treating critiques as a training-time-only semantic scaffold, CGD enables models to internalize the transformation from error to correction. The results suggest that this approach offers a stable, supervised alternative to Reinforcement Learning (RL) for reasoning tasks, with the authors noting that CGD and RL may be complementary rather than competing approaches. The method is highlighted as particularly effective for resource-constrained settings, achieving strong performance with significantly lower compute costs compared to RL-based reasoning methods.