Technical Summary: Self-Verified Distillation

Problem Statement

The paper addresses the challenge of further improving post-trained Large Language Models (LLMs) for reasoning tasks without access to external teachers, tool feedback, or ground-truth solutions. While many successful post-training approaches rely on supervised data from human-labeled traces, teacher-generated answers, or verifier training on known correct solutions, these resources are expensive, difficult to scale, or unavailable for the most difficult reasoning problems (e.g., unsolved math or science questions).

The core question investigated is: Can an already post-trained reasoning model further improve using only itself? Specifically, can a model generate candidate solutions to unlabeled seed questions, reliably verify which of those solutions are correct without external ground truth, and then use only the verified solutions to refine its own capabilities?

The primary difficulty lies in the "generator-validator consistency" problem: a naive self-training approach risks reinforcing the model's own mistakes if the model generates an incorrect solution, incorrectly judges it as valid, and then trains on that flawed reasoning.

Methodology: Self-Verified Distillation

The authors propose Self-Verified Distillation, a post-training refinement algorithm that constructs a self-curated supervised dataset from unlabeled seed questions. The process consists of three stages:

1. Generation: Starting from a post-trained model $p_\theta$, the system samples $n$ candidate solutions for each unlabeled seed question. These candidates are not assumed to be correct.
2. Self-Verification: Candidates are filtered using a prompt-based, multi-stage self-verification pipeline inspired by the Unsolved Questions (UQ) benchmark. Unlike standard answer-matching, this pipeline uses "oracle-free" validators to screen answers. The verification is a three-stage cascade:
   • Cycle-Consistency: The model infers the question from the candidate answer and checks if it matches the original seed question.
   • Factuality: The model checks for factual, arithmetic, or logical errors in the response.
   • Correctness: The model performs a total correctness check to ensure the solution fully and accurately solves the problem.
   • Consensus Mechanism: For each stage, the verifier is called $v$ times. A candidate is accepted only if it passes all three stages in all $v$ repeated judge calls (unanimous voting).
3. Training: The model is trained via Supervised Fine-Tuning (SFT) on the dataset of accepted, self-generated solutions.

Key hyperparameters include $n$ (number of generations per question, controlling exploration) and $v$ (verification strength, controlling stringency).

Key Contributions

1. Self-Verified Distillation Algorithm: The paper introduces a method to improve reasoning models starting solely from unlabeled seed questions, eliminating the need for external teachers, tool feedback, or ground-truth solutions.
2. Controlled Study of Synthetic Data Construction: The authors systematically ablate the data-construction pipeline, demonstrating that:
   • Simply training on unfiltered self-generated data can hurt performance (particularly in coding).
   • Stronger verification (higher $v$) and broader generation (higher $n$) are required to produce high-quality training data.
   • A multi-stage UQ-style verifier (cycle-consistency + factuality + correctness) significantly outperforms a simple "is this correct?" prompt under the same verification strength.
3. Training-Time vs. Test-Time Compute: The paper introduces UQ-TTC, a baseline that applies the same verification procedure only at inference time (sampling multiple candidates and verifying them for every test query). The authors show that Self-Verified Distillation (amortizing compute during training) achieves competitive or superior performance compared to UQ-TTC while requiring only a single inference call at test time.

Experimental Results

The method was evaluated on Qwen3 models (0.6B, 4B, and 8B) across three domains: Math, Science, and Coding.

• Performance Gains: Self-Verified Distillation improved aggregate held-out pass@1 scores across all model scales and domains.
  • Qwen3-4B: Achieved the most significant gains: +16.7 points in Math (AIME26/HMMT), +11.1 points in Science (GPQA Diamond/HLE), and +8.3 points in Coding (LCBv5/LCBv6).
  • Scaling: Gains were observed in 0.6B and 8B models as well, though the 0.6B model showed smaller and less consistent improvements (e.g., a slight drop on HLE), suggesting model capability affects the quality of self-curated data.
• Ablation Findings:
  • Verification Strength: Increasing $v$ (e.g., from 1 to 5) generally improved results, but only when paired with sufficient generation ($n$). High $v$ with low $n$ resulted in too few accepted samples for effective training.
  • Filtering Necessity: In coding, training on unfiltered generations performed worse than the initial model, whereas filtered data improved performance.
  • Verifier Structure: The full UQ-style multi-stage verifier yielded a mean improvement of +8.4 points in math, compared to +4.9 points for a simple correctness prompt.
• Efficiency: Compared to the UQ-TTC baseline (which requires up to 168 inference calls per test problem), Self-Verified Distillation achieved better or comparable results with only 1 inference call per test problem.

Significance and Claims

The paper claims that Self-Verified Distillation demonstrates that post-trained reasoning models can further improve themselves using only unlabeled prompts and their own internal verification capabilities.

• Lightweight Refinement: The method serves as a lightweight additional refinement stage that does not require expensive external supervision or tool integration.
• Compute Amortization: It shows that generation and verification compute can be effectively amortized into model parameters during training, offering a more efficient alternative to repeated test-time search and verification.
• Data Quality over Quantity: The results emphasize that the quality of self-curated data depends heavily on the verification pipeline; unfiltered self-training is insufficient and potentially harmful, while rigorous, multi-stage self-verification yields reliable training signals.

The authors note limitations, including the risk of reinforcing systematic mistakes if the verifier is imperfect, and the dependency on the difficulty and diversity of seed questions matching the model's capability. They suggest future work should focus on better seed-question distributions and mitigation strategies for verification errors.