Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

1. Problem Statement

Rotary Positional Embedding (RoPE) is the standard mechanism for encoding relative positional information in modern Transformer-based Large Language Models (LLMs). However, a critical design choice remains largely unexplored: what fraction of the hidden dimensions within an attention head should undergo the rotary transformation?

• Inconsistency: Major model families adopt varying strategies (e.g., GPT-J/NeoX use 25%, LLaMA/Qwen use 100%, Nemotron uses 50%, and recent Qwen3-Next reverts to 25%).
• Memory Bottleneck: The RoPE cache (storing pre-computed sine/cosine values) scales linearly with sequence length and head dimension. For long-context models (e.g., 1M+ tokens), this cache consumes significant VRAM, potentially becoming a bottleneck, especially on edge devices or in multi-GPU setups.
• Knowledge Gap: There is no systematic study on how reducing the fraction of rotated dimensions (Partial RoPE) affects training dynamics, convergence, and final model performance.

2. Methodology

The authors conducted a systematic empirical study pretraining models from scratch to isolate the effects of Partial RoPE.

• Architectures Tested:
  • Sequential Attention: Based on Llama-3.2-1B and Llama-3.1-8B.
  • Parallel Attention: Based on Pythia-1B.
• Datasets:
  • FineWeb: A large-scale web dataset (100B tokens).
  • FineWeb-Edu: A higher-quality subset filtered for educational content.
• Experimental Variables:
  • RoPE Fractions: 0% (NoPE), 1%, 4%, 10%, 25%, 50%, 75%, and 100% (Full RoPE).
  • Sequence Lengths: 1024, 2048, 4096, and 8192 tokens.
  • Model Scales: 1B and 8B parameters.
• Evaluation Metrics:
  • Training Dynamics: Loss convergence curves and stability.
  • Benchmarks: EleutherAI LM Evaluation Harness (ARC, LogiQA, LAMBADA, PIQA, etc.) and perplexity on LAMBADA.
• Stability Analysis: Investigated loss spikes in NoPE configurations using random seeds, learning rate variations, and the application of QK-Norm (Query-Key Normalization).

3. Key Findings & Results

A. The "10% Threshold" for Convergence

The most significant finding is that applying RoPE to a small fraction of dimensions is sufficient to achieve performance comparable to full RoPE.

• Convergence: Models with 10% or more rotated dimensions converge to the same final loss as 100% RoPE.
• Diminishing Returns: Increasing the fraction from 10% to 100% yields negligible improvements in final loss or benchmark accuracy.
• Memory Savings: Reducing the RoPE application to 10% results in a ~10x reduction in RoPE cache memory usage, which is critical for long-context windows (e.g., 1M+ tokens).

B. Robustness Across Variables

The "10% threshold" trend holds consistently across:

• Model Sizes: Both 1B and 8B parameter models.
• Architectures: Both sequential and parallel attention designs.
• Data Quality: Observed in both general web data (FineWeb) and high-quality educational data (FineWeb-Edu), though higher-quality data generally resulted in lower overall loss.
• Sequence Lengths: Trends remained stable across 1k to 8k token contexts.

C. The Danger of NoPE (0% RoPE)

• Instability: Models trained with NoPE (0% RoPE) exhibit unstable learning trajectories, often characterized by severe loss spikes (divergence) after tens of billions of tokens, particularly in parallel architectures and at longer sequence lengths (8192).
• Mitigation: These spikes can be mitigated by applying QK-Norm, which stabilizes training but results in a higher final loss band compared to Partial RoPE.
• Conclusion: While QK-Norm helps, applying even a minimal amount of RoPE (e.g., 10%) is a more effective strategy for stability than relying solely on normalization.

D. Benchmark Performance

• MCQ Tasks: On 9 out of 10 multiple-choice benchmarks, all RoPE variants (10%–100%) performed similarly.
• Perplexity (LAMBADA): Perplexity dropped sharply when moving from <10% RoPE to ≥10% RoPE, after which it plateaued. This confirms that the "loss bands" observed in training correlate with generalization capabilities.

4. Key Contributions

1. Systematic Characterization: The first comprehensive study mapping the relationship between the fraction of rotated dimensions and model convergence/stability.
2. Practical Efficiency Guideline: Establishes that 10% Partial RoPE is the "sweet spot," offering full RoPE performance with a 10x reduction in memory overhead for the position cache.
3. Stability Insights: Identifies that NoPE configurations are prone to catastrophic loss spikes in large-scale, long-context training and demonstrates that minimal RoPE application is a superior stabilization strategy compared to QK-Norm alone.
4. Open Source: The authors released all training code and artifacts to ensure reproducibility.

5. Significance and Impact

This work provides actionable guidance for model designers and engineers:

• Long-Context Deployment: For models targeting massive context windows (100k–1M+ tokens), switching to 10% Partial RoPE can drastically reduce VRAM requirements without sacrificing model quality, enabling deployment on resource-constrained hardware.
• Training Stability: It highlights the risks of "NoPE" approaches in large-scale training and suggests a minimal RoPE application as a robust safety net against training divergence.
• Design Standardization: It challenges the current lack of consensus in the industry, suggesting that the specific fraction (25% vs. 100%) matters less than ensuring a minimum threshold (10%) is met, allowing for more efficient model architectures.

In summary, the paper argues that Partial RoPE is not just a compromise but an optimization: a small fraction of rotary dimensions unlocks the full potential of positional encoding while significantly improving memory efficiency and training stability.