Technical Summary: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR), or "Zero RL," has emerged as a powerful paradigm for eliciting Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs) without human-annotated data. However, existing research is largely constrained to small-scale models due to computational limits. Consequently, the training dynamics, emergent capabilities, and scalability of Zero RL at the trillion-parameter level remain unexplored. Furthermore, naive applications of Zero RL often suffer from critical limitations: poor readability of reasoning traces, token redundancy (uncontrolled length growth), and a lack of adaptive reasoning depth, where models fail to adjust their computational effort based on problem difficulty.

Methodology

The authors present Ring-2.5-1T-Zero, a stable and efficient training pipeline designed to elicit high-quality reasoning from a 1-trillion parameter Mixture-of-Experts (MoE) model (Ling-2.5-1T-Base) from scratch. The approach emphasizes minimalism, introducing only necessary algorithmic and system optimizations to stabilize training at scale.

1. Multi-Stage Training Pipeline

The pipeline consists of four distinct phases designed to progressively build and refine reasoning capabilities:

• First-Stage RL (Reasoning Elicitation): Trains directly from a pretrained base model lacking reasoning capabilities. It utilizes a clipped importance-sampling policy gradient to aggressively amplify low-probability reasoning tokens. To prevent numerical instability caused by discrepancies between the training engine (Megatron) and inference engine (SGLang), the authors implement a training-inference ratio correction, using training-engine logits for the numerator of the importance ratio. A token-level loss is used to encourage longer generations, coupled with a KL divergence penalty to prevent policy drift.
• Self-Distillation (Compression & Stabilization): To address verbosity and the accumulated training-inference gap, the model undergoes self-distillation. The first-stage expert model generates rollouts, from which the shortest correct traces are selected and further refined by filtering redundant segments. This refined corpus is used to fine-tune the base model, effectively resetting the model with concise, high-quality reasoning traces and eliminating numerical errors.
• Second-Stage RL (Sustained Optimization): Resumes RL training using sample-level loss normalization instead of token-level loss. This prevents uncontrolled length growth while maintaining stability. The KL penalty is removed to allow for further exploration, as the distilled model provides a robust starting point.
• Third-Stage RL (Adaptive Depth): Introduces tier-based training where questions are partitioned into Low, Medium, and High difficulty tiers, each with specific system prompts and maximum token budgets. This equips the model with adaptive response modes, allowing it to allocate reasoning depth dynamically based on problem complexity.

2. Infrastructure Optimizations

Scaling to 1T parameters required specific system-level interventions:

• Mixed-Precision Control: To mitigate numerical instability in the importance ratio calculation (which involves exponentiation), the attention softmax and LM head are computed in FP32, while the rest of the model remains in BF16. This prevents small rounding errors from exploding into macroscopic training collapse.
• Context Parallelism Optimization: To handle long context windows efficiently, the authors tailored context parallelism for the model's hybrid architecture. They employ All-to-All communication for Multi-Head Latent Attention (MLA) layers and a single AllGather operation for Lightning Attention layers, breaking the latency bottlenecks of standard ring attention.

Key Contributions

1. Stable Trillion-Scale Zero RL: The paper demonstrates that with minor, targeted algorithmic and system improvements (clipped importance sampling, ratio correction, mixed-precision control), Zero RL can be successfully scaled to 1T parameters, achieving competitive performance on mathematical benchmarks without human-annotated data.
2. Structured CoT Evaluation Framework: Moving beyond final-answer accuracy, the authors propose a three-dimensional evaluation framework: Comprehensibility (logical flow and readability), Reproducibility (transferability to weaker models via distillation), and Efficiency (token count for correct solutions).
3. Empirical Validation of the "Bitter Lesson": The work provides strong evidence that massive computation and scale can outperform human-engineered heuristics. The 1T model autonomously develops advanced cognitive strategies that previously required complex, hand-crafted pipelines.

Experimental Results

Evaluated on seven challenging mathematical benchmarks (including AIME 2024-2026, HMMT, and IMOAnswerBench), Ring-2.5-1T-Zero achieves competitive performance:

• Performance: The model reaches 93.2% pass@1 accuracy on AIME 2024 and 91.4% on AIME 2026 (High tier), comparable to frontier proprietary models.
• Scaling Effects: The 1T model exhibits significantly higher sample efficiency and a higher performance ceiling compared to its 104B counterpart (Ring-2.5-flash-Zero).
• CoT Quality: Under the proposed evaluation framework, the model demonstrates superior comprehensibility (winning pairwise comparisons against baselines), reproducibility (distilled student models outperform those distilled from DeepSeek-R1 with significantly less data), and efficiency (using ~6,368 tokens on average for correct solutions, less than half of other models).

Emergent Behaviors

A striking finding is the spontaneous emergence of complex cognitive strategies without explicit supervision:

• Anthropomorphism: The model exhibits simulated emotional states (e.g., "brain fart," "genius idea") and meta-commentary, mimicking human cognitive friction.
• Structured Formatting: It naturally organizes thoughts into explicit steps (e.g., "Step 1," "Step 2") and summaries, improving readability.
• Self-Verification: The model autonomously cross-checks intermediate derivations and re-evaluates assumptions to ensure logical consistency.
• Parallel Reasoning: It branches into alternative strategies (Tree-of-Thought) within a single rollout, comparing outcomes before committing to an answer.
• Context Anxiety: As the model approaches token limits, it strategically aborts complex chains to force a heuristic guess, prioritizing structural completeness over rigorous derivation to secure a non-zero reward.

Significance and Claims

The paper claims to provide a profound validation of the "bitter lesson" in AI: massive computation and scale ultimately overtake human-engineered heuristics. By scaling Zero RL to the trillion-parameter regime, the authors show that:

1. Scaling dictates capability: Larger models possess the latent capacity to self-discover complex logical deductions and advanced reasoning strategies purely through trial-and-error.
2. Training Dynamics: The learning process follows a sequential "discovery" phase (unlocking dormant reasoning pathways) followed by a "sharpening" phase (refining the policy within established boundaries).
3. Redundancy of Hand-Crafted Rules: Advanced behaviors like structured formatting, self-verification, and parallel reasoning emerge naturally at scale, rendering complex, hand-crafted reward designs and pipelines unnecessary.

The authors conclude that their work offers a reproducible, open playbook for scaling RL to trillion-parameter models, lowering engineering barriers and providing deeper insights into the emergent behaviors of self-evolving reasoning systems.