Technical Summary: On the Generalization Gap in Self-Evolving Language Model Reasoning

Problem Statement

Recent advances in Large Language Models (LLMs) have demonstrated significant progress in complex reasoning tasks, often driven by post-training techniques like Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). However, these methods typically rely on human-labeled data or verifiable external rewards. A growing area of interest is Self-Evolution (SE), where models improve using supervision signals generated solely by themselves.

The central question addressed in this work is: Under a strict closed-loop setup, where the SE algorithm has access only to an unlabeled prompt set and a base model, how close can internally generated supervision come to oracle-supervised training?

While prior work suggests SE can improve performance, it also highlights risks such as model collapse and diminishing returns. This paper seeks to empirically quantify the "generalization gap" between self-evolved models and those trained with ground-truth (oracle) supervision, isolating the effects of internal signals without external tools, symbolic environments, or human intervention.

Methodology

Framework and Setup

The authors propose a unified offline self-evolution framework where a single base model $M$ is instantiated in two roles:

1. Generator ($G$): Proposes candidate solutions.
2. Verifier ($V$): Evaluates the quality of these solutions.

The framework operates on an unlabeled prompt set $D$. All reasoning traces, solutions, rewards, and preference labels must be derived from $M$ itself. The study evaluates four representative strategies within this framework:

1. Single-Round Verification (SimpleSE): The verifier judges static candidate responses. To mitigate noise, a thresholded majority voting mechanism is used: a response is labeled positive if the verifier's correctness rate exceeds a threshold $\tau$, negative if below $1-\tau$, and discarded otherwise.
2. Multi-Turn Revision with Feedback (RevisionSE): The verifier provides textual feedback on incorrect solutions, and the generator revises its output. Preference pairs are extracted when a revision turns an incorrect solution into a correct one.
3. Iterative Training: The generator-verifier game is applied repeatedly. The model is fine-tuned on the generated preference data, and the updated model is used for the next round of generation and verification.
4. Curriculum Learning: Training proceeds from easier instances to harder ones (e.g., Knights and Knaves problems with fewer people to those with more), rather than random mixing.

Datasets and Models

• Primary Benchmark: Knights and Knaves (KK) logical reasoning tasks. This dataset is chosen for its deterministic solutions, controlled difficulty (parameterized by the number of people), and clean testbed for easy-to-hard generalization.
• Secondary Benchmarks: OpenThoughts3 (mathematical reasoning), GSM8K, MATH500, MATHHard, and TabMWP to assess robustness beyond deterministic logic.
• Models: The study primarily uses the Gemma-3-it family (1B, 4B, 12B, and 27B as an upper bound) and Qwen-2.5-Instruct. Instruction-tuned variants are used to ensure the model can follow system prompts for generation and verification.
• Training Objective: The generated preference pairs $(y_w, y_l)$ are used to fine-tune the policy via Direct Preference Optimization (DPO).
• Oracle Baseline: A setting where ground-truth solutions are used to label examples as correct/incorrect, serving as the upper bound for performance.

Key Contributions

1. Systematic Analysis of Closed-Loop SE: The paper provides a unified comparison of SE methods under a strict closed-loop constraint, isolating their effects relative to oracle-supervised preference optimization. It evaluates multiple strategies (verification, revision, iteration, curriculum) on the KK benchmark.
2. Effect of Internal Supervision Quality: The study demonstrates that the quality of the verifier signal is critical. Multi-turn feedback (RevisionSE) yields the best performance, particularly for larger models (12B), where it nearly matches oracle-supervised training. However, this comes at a high computational cost.
3. Bootstrapping and Generalization: The authors investigate "easy-to-hard" transfer. While iterative training and curriculum learning enhance the ability to solve harder examples (e.g., KK instances with 6–8 people), the ability to improve out-of-distribution (OOD) generalization remains limited compared to oracle supervision.
4. Quantification of the Persistent Gap: Even with increasingly structured and computationally intensive SE strategies, a performance gap of approximately 8–13% relative to oracle supervision persists in most settings. This gap is observed on both controlled KK data and real-world OpenThoughts3 settings.

Results

Performance on Knights and Knaves (KK)

• Base Model: Gemma 3 4B achieves 31.0% accuracy.
• Oracle Supervision: Training with ground-truth signals raises accuracy to 53.3%.
• Self-Evolution Strategies:
  • Single-round verification (SimpleSE): 40.7%.
  • Multi-turn feedback (RevisionSE): 42.2%.
  • Iterative training: 44.1%.
  • Curriculum learning: 44.8%.
• Scaling Effects:
  • 1B Model: Shows negligible improvement (0.6%) or even degradation with RevisionSE, highlighting a capability threshold where small models introduce noise that corrupts learning.
  • 4B Model: Achieves significant gains but remains ~8–13% below oracle.
  • 12B Model: RevisionSE achieves 52.8% accuracy, nearly matching the oracle baseline of 53.6%. This suggests that larger models possess sufficient intrinsic verification and revision capabilities to close the gap.

Performance on Real-World Benchmarks

On OpenThoughts3 and other reasoning benchmarks (GSM8K, MATH500), self-improvement gains are modest. For example, on MATH500, SimpleSE improves Gemma 4B by only 1.6%. The paper notes that for open-ended problems, internal verifiers struggle to distinguish plausible but flawed reasoning paths from correct ones, leading to smaller gains compared to the deterministic KK tasks.

Computational Trade-offs

The study finds that scaling the number of verifier passes ($n_2$) is generally more cost-effective than scaling the number of candidate generations ($n_1$). Moderate confidence thresholds (e.g., $\tau = 0.7$) offer a good balance between precision and recall.

Significance and Claims

The paper claims to characterize when closed-loop SE can help and how internally generated supervision remains insufficient under a minimal formulation.

• Limitations of Internal Signals: The primary finding is that under a strict closed-loop constraint, internal feedback alone is not a general substitute for verifiable rewards or curated ground-truth solutions. While SE can "sharpen" a model's confidence in its existing solution modes (improving Pass@1), it does not reliably teach the model to solve much harder problems without external signals.
• Capability Threshold: Self-improvement is highly dependent on model scale. Smaller models (e.g., 1B) often fail to self-evolve effectively due to verifier noise, whereas larger models (12B+) can leverage multi-turn feedback to approach oracle performance.
• Generalization Gap: A persistent gap exists between self-evolved models and oracle-trained models, particularly for OOD generalization and open-ended reasoning tasks. The paper suggests that while SE is useful for distilling latent verification capabilities, it cannot fully replace the need for high-quality external supervision in complex reasoning domains.

In summary, the work provides a rigorous empirical baseline showing that while self-evolution offers measurable improvements, it currently plateaus below the performance ceiling achievable with oracle supervision, unless the model is sufficiently large to act as its own reliable verifier and reviser.