Technical Summary: Better, Faster: Harnessing Self-Improvement in Large Reasoning Models

Problem Statement

Large Reasoning Models (LRMs) have demonstrated significant capabilities in complex reasoning tasks by leveraging explicit long chain-of-thought (CoT) trajectories. However, obtaining high-quality reasoning data is costly and time-consuming, leading to the emergence of "self-improvement" paradigms where models iteratively train on their own generated data. The authors identify two critical shortcomings in existing self-improvement methods when applied to complex reasoning tasks:

1. Data Imbalance: Self-generated datasets are heavily skewed toward simple queries. Challenging queries, which are crucial for improving reasoning capabilities, often yield few or no correct solutions, leading to a scarcity of high-value training samples.
2. Overthinking: Models frequently generate solutions with redundant, repetitive, or unnecessary reasoning steps. Existing methods that attempt to filter these often rely on simple length-based metrics, which can inadvertently penalize deep reasoning and degrade performance.

These issues result in suboptimal reasoning performance and, in severe cases, model collapse where performance degrades due to iterative training on low-quality, self-generated data.

Methodology: HSIR

To address these limitations, the authors propose HSIR (Harnessing Self-Improvement in large Reasoning models), a framework integrating two simple yet effective approaches: VeriExit sampling and Intrinsic Diversity (InDiv) scoring.

1. Verify-then-Exit (VeriExit) Sampling

To mitigate data imbalance, VeriExit aims to salvage valuable reasoning from failed attempts rather than discarding them.

• Mechanism: The method analyzes "failed" solutions (those with incorrect final answers) to identify intermediate reasoning steps that correctly arrive at the ground-truth answer (e.g., explicitly stating "answer is {yi}").
• Process: Upon detecting such a step, the reasoning trajectory is truncated, and an exit prompt is inserted. The model is then prompted to resample the final answer from this verified intermediate state.
• Benefit: This strategy efficiently converts partially correct failed attempts into correct training samples for difficult queries without the high inference overhead of generating entirely new trajectories from scratch.

2. Intrinsic Diversity (InDiv) Score

To combat overthinking, the authors introduce a metric that quantifies reasoning redundancy using the model's internal states, avoiding reliance on external models or simple length penalties.

• Mechanism: InDiv analyzes the hidden representations of intermediate reasoning steps at the middle layer of the LLM. It computes the cross-covariance matrix of these representations and derives eigenvalues.
• Calculation: The score is calculated as a weighted sum of the logarithms of these eigenvalues, where weights are determined by the self-attention scores of the tokens.
• Logic: Repetitive or redundant reasoning steps result in highly correlated hidden representations, leading to smaller eigenvalues and lower InDiv scores. Conversely, diverse and concise reasoning yields higher scores.
• Application: Solutions with InDiv scores below a specific threshold are filtered out, ensuring the training data consists of diverse and concise reasoning paths.

Integration into Training Paradigms

HSIR is designed to be compatible with various post-training paradigms:

• Supervised Fine-Tuning (SFT) & Preference Learning (DPO): HSIR filters the self-generated dataset before training.
• H-GRPO (Reinforcement Learning): The authors extend the method to the Group Relative Policy Optimization (GRPO) framework. They propose H-GRPO, which incorporates the InDiv score as an additional reward signal alongside accuracy and format rewards. This encourages the model to generate diverse and concise reasoning paths during the reinforcement learning process.

Key Contributions

1. Problem Identification: The paper empirically reveals and analyzes the dual problems of data imbalance and overthinking in LRM self-improvement, demonstrating how they lead to performance degradation in complex tasks.
2. HSIR Framework: The proposal of a unified framework utilizing VeriExit to recover correct solutions from failed attempts and InDiv to filter redundant reasoning, effectively balancing data quality and diversity.
3. H-GRPO Algorithm: The extension of these principles into a reinforcement learning setting (H-GRPO), using intrinsic diversity as an external reward to alleviate overthinking without relying on length-based penalties.
4. Comprehensive Evaluation: Extensive experiments across multiple model sizes (Qwen2.5, Qwen3, Phi-3.5, Mistral, LLaMA3) and reasoning benchmarks (MedQA, GSM8K, ARC, CSQA, StrategyQA).

Experimental Results

The authors report significant improvements in both reasoning performance and efficiency:

• Performance Gains: HSIR consistently outperforms baseline self-improvement methods (e.g., STaR, ReSTEM, IRPO). On the MedQA task, HSIR-DPO achieved an average performance gain of +10.9% over baselines.
• Efficiency Improvements: By filtering overthinking solutions, HSIR significantly reduces inference overhead. The method reduced the number of output tokens by up to 42.4% relative to baseline methods, indicating more concise reasoning.
• Generalization: Models trained with HSIR demonstrated better out-of-distribution (OOD) generalization on tasks like Medbullets, MedXpertQA, and MATH compared to models trained with standard iterative self-improvement, which often suffered from performance degradation over iterations.
• Robustness: The improvements were consistent across different model architectures and sizes, and the method remained effective even when using self-consistency pseudo-labels in the absence of ground-truth answers.

Significance and Claims

The paper claims that HSIR offers a more effective path for self-improvement in LRMs by ensuring the training data is both accurate (via VeriExit) and efficient/diverse (via InDiv). The authors emphasize that their approach:

• Avoids the pitfalls of model collapse by curating high-quality data.
• Solves the "overthinking" problem without sacrificing deep reasoning capabilities, a common issue with length-penalty methods.
• Provides a scalable solution that enhances both the "better" (accuracy) and "faster" (efficiency) aspects of reasoning models.

The authors maintain a modest stance regarding limitations, acknowledging that InDiv may misjudge necessary logical steps in rigorous mathematical derivations as redundant, and that VeriExit cannot solve queries where no valid intermediate steps exist. They position their work as a step toward more robust and efficient self-improvement training, rather than a complete solution to all reasoning challenges.