Technical Summary: CANON (Consensus-Anchored Self-Distillation)

Problem Statement

Large language models (LLMs) have demonstrated improved reasoning capabilities through Reinforcement Learning (RL) with verifiable rewards. However, this standard approach presupposes access to gold answers for verification, which are unavailable in unsupervised or label-free settings. While "self-consistency" (sampling multiple solutions and selecting the majority vote) serves as a reliable unsupervised proxy for accuracy, existing methods that leverage this signal are limited in their efficiency and information density. Current label-free approaches typically reduce the consensus signal to:

1. A filter for selecting solutions for supervised fine-tuning (SFT).
2. A source of pairwise preferences for alignment.
3. A scalar pseudo-reward for RL.

These methods discard the rich, token-level information contained within the agreeing solutions. Furthermore, RL-based approaches often require hundreds of optimization steps backed by fresh generations, resulting in high computational costs. The core challenge addressed by this paper is how to transform the model's own consensus into a dense, token-level supervision signal without gold labels, enabling efficient self-distillation that rivals gold-conditioned methods.

Methodology: CANON

The authors propose CANON (Consensus-Anchored self-distillatiON), a label-free post-training method that converts majority votes into dense supervision. The framework operates in three steps per unlabeled prompt:

1. Consensus Extraction: Given a prompt $x$, the model samples $N$ solutions (rollouts). The majority answer is identified, and the most confident solution within the majority set (highest mean token log-probability) is selected as the consensus solution ($s_x$).
2. Teacher Construction: A frozen snapshot of the model ($\bar{\pi}$), taken at the start of the training epoch, acts as the teacher. This teacher is conditioned on the prompt $x$ and the consensus solution $s_x$ (inserted via a template). Crucially, the teacher does not generate; it only scores.
3. Dense Distillation: The student model ($\pi_\theta$) generates its own rollouts $y^{(i)}$ conditioned only on the prompt. The student is trained to match the teacher's next-token distributions over these same rollouts. The objective minimizes the full-vocabulary Jensen–Shannon Divergence (JSD) between the student's distribution and the teacher's distribution at every token position:
   $$L(\theta) = \mathbb{E}_{x \sim D} \frac{1}{N} \sum_{i=1}^{N} \frac{1}{|y^{(i)}|} \sum_{t=1}^{|y^{(i)}|} D_{JS} \left( \pi_\theta(\cdot | x, y^{(i)}_{<t}) \parallel \bar{\pi}(\cdot | x, s_x, y^{(i)}_{<t}) \right)$$

Key Distinctions:

• Dense Supervision: Unlike scalar rewards, CANON provides a full next-token distribution for every token in every rollout, including those that disagree with the consensus.
• Frozen Teacher: The teacher is a static snapshot, preventing degenerate co-adaptation between teacher and student.
• Efficiency: The method requires only a single generation pass (the same pass used for inference-time self-consistency) and one epoch of training, avoiding the repeated generation cycles of RL.

Key Contributions

1. CANON Framework: A novel label-free method that distills a consensus-anchored teacher into the model, turning majority votes into dense token-level supervision.
2. Efficiency and Performance: A controlled comparison demonstrating that CANON dominates existing label-free alternatives (such as self-consistency fine-tuning and test-time RL) on the accuracy-compute frontier, achieving gains comparable to gold-conditioned teachers at a fraction of the compute cost.
3. Transferability: Evidence that improvements transfer to held-out and harder benchmarks, matching the performance of training methods that utilize gold labels.
4. Mechanism Analysis: An analysis showing that gains are not merely due to distribution sharpening (concentrating probability on existing answers) but involve genuine capability expansion, solving problems previously unsolved in multiple attempts.

Experimental Results

The authors evaluated CANON on mathematical (AMC 2023, AIME 2024/2025, MATH500) and scientific (GPQA-Diamond) reasoning benchmarks using various model families (Qwen, SmolLM, Gemma, LFM).

• Transductive Performance (Test-Time Training): On the AMC benchmark, CANON improved pass@1 by 10.5 points (from 66.0 to 76.5) using only 1.2 GPU-hours. This significantly outperformed label-free RL baselines (e.g., TTRL, SCRL), which required 6–10 GPU-hours and achieved gains of only 4–5 points. CANON approached the performance of an "oracle" teacher conditioned on gold-verified solutions (76.9 vs. 76.5).
• Inductive Transfer: When trained on unlabeled AMC prompts and evaluated on held-out benchmarks (AIME 2024/2025, GPQA), CANON maintained significant gains, outperforming label-free RL and matching gold-reward RL. For instance, on AIME 2024, CANON improved avg@32 by 7.0 points over the base model, whereas TTRL improved by 4.2 points.
• Compute Efficiency: CANON achieved results comparable to gold-reward GRPO (which required 19.2 GPU-hours) in just 1.2 GPU-hours (roughly 1/16th the compute).
• Coverage and Capability: Post-training analysis revealed that the model's majority vote accuracy increased, and pass@32 improved significantly on hard benchmarks. The model solved problems it previously never solved in 32 attempts, indicating that the gains extend beyond simple probability concentration.

Significance and Claims

The paper claims that CANON demonstrates that a model's own consensus can effectively serve as privileged context for self-distillation, a role traditionally filled only by gold labels. By anchoring a teacher to the consensus solution, the method extracts dense supervision signals that are both computationally efficient and highly effective.

The authors emphasize that the improvements are not merely a result of "sharpening" the output distribution on already-solvable problems. Instead, the method enables the model to solve problems it previously failed, suggesting a genuine expansion of reasoning capability. The work establishes that for small instruct models on domains with extractable answers (like math and multiple-choice science), consensus-based self-distillation can match or approach the efficacy of gold-supervised training while operating entirely without labels.

Limitations Noted by the Authors:

• The method requires an extractable final answer to form a consensus, limiting it to verifiable-output domains.
• It offers little benefit on "saturated" model-benchmark pairs where the model already achieves high accuracy.
• It can reinforce errors if the samples collude on a confidently wrong answer, though this failure mode was observed in only ~2.7% of analyzed prompts.
• The transductive regime assumes access to test prompts before answering, which may not suit interactive use cases, though inductive results suggest the assumption can be relaxed.