Technical Summary: PCL-Reasoner-V1.5

Problem Statement

Mathematical reasoning remains a rigorous benchmark for Large Language Models (LLMs), demanding precise logical deduction and structured problem-solving. While recent breakthroughs like DeepSeek-R1 have demonstrated that Reinforcement Learning (RL) combined with Chain-of-Thought (CoT) can elicit superior reasoning, the prevailing paradigm relies heavily on online RL methods (e.g., Group Relative Policy Optimization or GRPO). These online approaches introduce significant technical challenges, including training instability, reward collapse, and high computational overhead due to the tight coupling of inference and training loops. Furthermore, the complex orchestration required for real-time rollouts and distributed gradient updates increases engineering complexity and experimentation costs.

Methodology

The authors introduce PCL-Reasoner-V1.5, a 32-billion-parameter LLM built upon the Qwen2.5-32B architecture. The model is refined through a two-stage pipeline: Supervised Fine-Tuning (SFT) followed by a novel Offline RL approach.

1. Supervised Fine-Tuning (SFT)

• Base Model: Qwen2.5-32B.
• Data: 666K samples from the mathematical subset of AM-DeepSeek-R1-0528-Distilled. Data was filtered to remove sequences exceeding 32K tokens.
• Training Configuration: Global batch size of 128, learning rate of $6 \times 10^{-5}$ (cosine decay), and 4 epochs. The authors utilized data packing to concatenate variable-length samples into 32K sequences, eliminating padding redundancy.
• Prompting: A specific template was used to enforce a CoT structure, requiring the model to output reasoning within <thought> tags and the final answer within <answer> tags.

2. Offline Reinforcement Learning

Unlike online RL, which iteratively samples and trains, this approach decouples inference from training.

• Data Curation: The dataset was derived from Nemotron-Post-Training-Dataset-v1. The authors filtered for high-difficulty tasks by retaining questions where the average inferred answer length was >32K tokens and not all 8 candidate answers were correct. This resulted in 6,068 difficult questions.
• Inference Phase: The SFT model (PCL-Reasoner-V1) generated 8 candidate answers per question using sampling-based decoding (top-k=40, top-p=0.95, temperature=0.6) on Huawei Ascend 910C NPUs.
• Verification & Filtering: A separate verifier model (Qwen3-32B) assigned binary rewards ($R=1$ for correct, $R=-1$ for incorrect).
  • Questions with zero reward variance (all correct or all incorrect) were excluded to mimic GRPO's internal mechanism.
  • Positive samples exceeding 48K tokens were discarded to prevent length bias.
  • The final dataset contained 30,215 samples (14,512 positive, 15,703 negative).
• Training Objective: The policy model was trained to minimize the RL loss $L(\theta) = -\sum R_i \pi_\theta(y_i|x_i)$, where $\pi_\theta$ is defined as the geometric mean of token-level probabilities. Importance sampling was explicitly omitted based on prior findings of its ineffectiveness in LLM RL settings.
• Infrastructure: Training utilized 8 compute nodes with 8 Ascend 910C NPUs each, employing tensor parallelism (degree 8) and pipeline parallelism (degree 4) via the MindSpeed-LLM framework.

Key Contributions

1. Offline RL Paradigm: The paper proposes and validates offline RL as a stable and efficient alternative to online RL (GRPO) for reasoning tasks. It argues that decoupling inference from training eliminates the feedback loops causing instability and allows for "infer once, train many" workflows, significantly reducing experimentation costs.
2. State-of-the-Art Performance: PCL-Reasoner-V1.5 achieves new state-of-the-art results among models post-trained on Qwen2.5-32B.
3. Hardware Efficiency: All experiments were successfully conducted on Huawei Ascend 910C NPUs, demonstrating the viability of this hardware for large-scale reasoning model training.
4. Open Release: The authors have released the model, training data, and code (including data processing and evaluation scripts) to facilitate reproducibility.

Experimental Results

The model was evaluated on the AIME 2024 and AIME 2025 benchmarks (30 problems each).

• Accuracy (pass@1):
  • AIME 2024: 90.9%
  • AIME 2025: 85.6%
  • These results surpass other Qwen2.5-32B derivatives (e.g., DeepSeek-R1-Distill-Qwen-32B at 72.6% on AIME 2024) and compete with much larger models like OpenAI-o3 (91.6%) and DeepSeek-R1-0528 (91.4%).
• Behavioral Analysis:
  • Response Length: RL training significantly increased the average response length (from ~22.5K to ~30.5K tokens on AIME 2024), suggesting the model engages in more extensive reasoning.
  • Long-CoT Capability: Disaggregated analysis shows that while the SFT model struggles with problems requiring long CoTs ($\ge$32K tokens), the RL-trained model shows dramatic accuracy improvements specifically in this range, indicating enhanced capacity for sustained, complex reasoning.

Significance and Claims

The authors claim that offline RL is a robust paradigm for advancing LLM reasoning, offering distinct advantages over online methods:

• Stability: By training on a fixed static dataset, the method avoids the reward collapse and divergence often seen in online RL due to dynamic inference-training interplay.
• Efficiency: Decoupling allows for independent optimization of inference (using high-throughput frameworks like vLLM) and training, maximizing hardware utilization and reducing latency overhead from synchronization.
• Simplicity: The workflow simplifies engineering by removing the need for complex real-time orchestration frameworks.

The paper concludes that while offline RL is bounded by the quality of its static dataset and cannot iteratively discover new reasoning paths like "R1-Zero" style self-correction, it is highly effective for refining models with strong baselines. The work challenges the necessity of online RL for high-performance reasoning, demonstrating that a stable, computationally efficient offline approach can yield competitive, state-of-the-art results.