Technical Summary: Klear-Reasoner

1. Problem Statement

Despite the success of large-scale reinforcement learning (RL) in reasoning models (e.g., OpenAI O1, DeepSeek-R1), reproducing high-performance reasoning models remains challenging due to incomplete disclosure of training details. Existing 7–8B parameter models often lag behind distilled versions of larger models in mathematical and coding reasoning tasks.

Key technical bottlenecks identified in current approaches include:

• Data Inefficiency: Uncertainty regarding the optimal balance between data quality and diversity, and the impact of incorrect reasoning samples (noise) on training.
• Clipping Limitations in RL: Standard clipping mechanisms in Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) suppress critical exploration signals (high-entropy tokens) and ignore suboptimal trajectories (negative samples) by discarding their gradients. This leads to premature termination of exploration and delayed convergence on negative examples.

2. Methodology

The Klear-Reasoner framework employs a two-stage post-training workflow: Long Chain-of-Thought Supervised Fine-Tuning (long CoT SFT) followed by Reinforcement Learning (RL).

2.1 Long CoT Supervised Fine-Tuning (SFT)

• Data Strategy: The authors adopt a "quality-over-diversity" approach. Instead of aggregating massive, diverse datasets, they curate a compact set of high-quality data sources (e.g., OpenThoughts, NuminaMath, OpenCodeReasoning).
• Correctness Filtering: Contrary to the belief that only correct data should be used, the study finds that for difficult samples, including incorrect reasoning paths (mixed data) does not harm performance and may even aid exploration. Consequently, they retain all responses from difficult samples without strict accuracy filtering.
• Teacher Model: Responses are generated using DeepSeek-R1-0528, which is identified as the optimal teacher model for distillation.

2.2 Gradient-Preserving Clipping Policy Optimization (GPPO)

To address the limitations of standard clipping, the authors propose GPPO, a novel RL algorithm that preserves gradient information from all tokens, even those outside the clipping bounds.

• Mechanism: Unlike traditional clipping which sets gradients to zero for out-of-bound tokens, GPPO decouples the forward pass (clipping) from the backward pass (gradient computation).
  • For tokens with high importance ratios ($r_t > 1+\epsilon$), gradients are preserved but constrained to a bounded value (e.g., $1+\epsilon_h$) rather than being discarded.
  • For tokens with low importance ratios ($r_t < 1-\epsilon$), gradients are similarly preserved and constrained.
• Benefits:
  • Enhanced Exploration: Retains gradients from high-entropy tokens, allowing the model to explore critical decision points without premature suppression.
  • Faster Convergence: Enables the model to learn from suboptimal (negative) trajectories immediately, rather than waiting for them to fall within the clipping window.
  • Stability: The gradients are mathematically proven to be bounded and mild, preventing training instability.
• Loss Function: The final objective combines the GPPO loss with a token-level SFT loss on positive examples ($L = L_{GPPO} + \alpha L_{SFT}$) to act as a regularizer and prevent reward hacking.

2.3 Reward Design and Data Filtering

• Soft Rewards for Code: Instead of binary pass/fail rewards, the system uses a soft reward mechanism based on the test case pass rate (e.g., passing 4/16 cases yields a reward of 0.25). This mitigates reward sparsity and provides dense learning signals.
• Data Filtering:
  • Code: Prompts with low estimated pass rates (pass@16 < 0.5) are filtered to remove noisy test cases that might penalize correct code.
  • Math: Zero-advantage groups (where all responses in a batch are either all correct or all incorrect) are filtered out to prevent dilution of optimization signals.

3. Key Contributions

1. GPPO Algorithm: A new policy optimization method that retains gradient signals from clipped tokens, balancing training stability with enhanced exploration and faster learning from negative samples.
2. Data Curation Insights: Empirical evidence showing that for difficult reasoning tasks, a small set of high-quality data is superior to large diverse datasets, and that incorrect samples in hard tasks can be beneficial for exploration.
3. Comprehensive Training Pipeline: A fully disclosed workflow integrating long CoT SFT with GPPO, soft rewards, and rigorous data filtering, achieving state-of-the-art results with an 8B parameter model.
4. Soft Reward Mechanism: Demonstrating that proportional rewards based on test case pass rates significantly improve code reasoning performance compared to binary rewards.

4. Experimental Results

Klear-Reasoner-8B was evaluated on multiple benchmarks, outperforming or matching larger and more complex models.

Key Benchmarks (Avg@64 for Math, Avg@8 for Code):

• AIME 2024: 90.5% (vs. DeepSeek-R1-0528-Distill-8B at 86.0%)
• AIME 2025: 83.2% (vs. DeepSeek-R1-0528-Distill-8B at 76.3%)
• LiveCodeBench V5: 66.0%
• LiveCodeBench V6: 58.1%

Comparisons:

• The model significantly outperforms other 7–8B open-source reasoning models (e.g., OpenReasoning-Nemotron-7B, AceReason-Nemotron-1.1-7B).
• It achieves performance comparable to models using 64K/96K inference budgets while being trained on a 32K context window.
• Ablation studies confirm that GPPO outperforms standard GRPO with Clip-Higher and the concurrent CISPO method in both stability and final accuracy.

5. Significance

The Klear-Reasoner report provides a critical blueprint for the community on how to build high-performance reasoning models without relying on massive parameter counts or undisclosed training secrets.

• Democratization of Reasoning: It demonstrates that an 8B model can rival much larger models through optimized training strategies rather than scale alone.
• Algorithmic Innovation: GPPO offers a principled solution to the "clipping problem" in RL, suggesting that discarding gradient information is suboptimal. This insight is likely to influence future RLHF/RLAIF research.
• Data Philosophy: The findings challenge the "more data is better" dogma, advocating for high-quality, consistent reasoning patterns and the strategic inclusion of difficult, noisy data to foster robust exploration.

In summary, Klear-Reasoner establishes a new state-of-the-art for efficient reasoning models by combining rigorous data curation with a novel, gradient-preserving optimization algorithm.