Technical Summary: LeRoPE: Learnable RoPE Frequencies Improve Language Modeling

Problem Statement

Rotary Positional Encodings (RoPE) are the standard positional encoding mechanism in modern large language models (LLMs). RoPE operates by rotating query and key vectors in two-dimensional frequency bands, where the rotation rate for each band is determined by a fixed geometric sequence defined by a base hyperparameter ($b$).

Prior research has identified limitations in this fixed-frequency approach:

1. Semantic vs. Positional Confusion: Recent work indicates that the slowest frequency bands (those with the lowest rotation rates) often carry semantic information rather than positional information, as they rotate negligibly over typical context windows.
2. Suboptimal Frequency Allocation: Existing improvements, such as Partial RoPE (p-RoPE) or adjusting the base $b$, involve manually selecting which bands to rotate or how to scale them. These methods fix the frequency allocation a priori before training, potentially preventing the model from discovering the optimal frequency spectrum for a given task and context length.

The authors posit that since the RoPE operation is differentiable with respect to its frequencies, these frequencies should be treated as learnable parameters rather than fixed hyperparameters.

Methodology: LeRoPE

The paper proposes LeRoPE (Learned RoPE), a modification to the standard RoPE mechanism where the frequency of each band is rescaled by a learned scalar.

Core Mechanism

Instead of using a fixed frequency $\theta_m$ for band $m$, LeRoPE introduces a learnable scalar $\alpha_m$ per frequency band. The effective frequency $\hat{\theta}_m$ is defined as:
$$\hat{\theta}_m = e^{\alpha_m} \theta_m$$
where $\theta_m$ is the standard RoPE frequency derived from the base $b$.

Key Design Choices:

• Logarithmic Parametrization: Frequencies are parametrized in log space ($\alpha_m$). This ensures that updates to $\alpha_m$ result in multiplicative changes to the frequency, providing better conditioning when scaling frequencies down toward zero and avoiding unstable gradient dynamics near zero.
• Initialization: $\alpha_m$ is initialized to 0, making LeRoPE mathematically equivalent to standard RoPE at the start of training.
• Parameter Sharing: The same set of learned frequencies is shared across all layers and attention heads. For a head dimension $d_h$, this adds $d_h/2$ learnable parameters to the model (e.g., 32 parameters for $d_h=64$).
• Optimization: The authors omit weight decay on the LeRoPE parameters and apply independent gradient clipping to them to ensure stable learning of the frequency scalars.

Theoretical Analysis

The paper derives the gradient of the loss with respect to the frequency parameters. They show that the gradient update for a frequency $\theta_m$ corresponds to the derivative of a surrogate loss function $P_m(\theta)$, which is the real component of the discrete-time Fourier Transform of the offset-aggregated gradient signal. This formulation suggests that the model learns frequencies by aligning them with the spectral characteristics of the attention logits' gradient signals across relative token offsets.

Experimental Setup

The authors validate LeRoPE through a rigorous scaling study:

• Model Scale: A "ladder" of decoder-only transformers ranging from 52M to 2.52B parameters.
• Architecture: Based on the Enoki scaling ladder (Charles et al., 2025; Ferbach et al., 2026), with fixed head dimension (64) and co-scaled depth/width.
• Training Data: Pre-trained on the C4 corpus with a sequence length of 2048 tokens.
• Baselines:
  • Standard RoPE ($b=10,000$).
  • Partial RoPE (p-RoPE, $p=0.75$), which removes rotation from the slowest 25% of bands.
• Evaluation Metrics: Validation perplexity on C4, compute multipliers (FLOPs required to match a baseline's loss), and "Needle-in-a-Haystack" (NIAH) retrieval accuracy.

Key Results

1. Performance Across Scales

LeRoPE consistently outperforms both standard RoPE and p-RoPE across all model sizes (52M to 2.5B).

• Perplexity: LeRoPE achieves lower validation perplexity at every scale.
• Compute Efficiency: At the largest scale (2.5B), standard RoPE requires 3.4% more compute (FLOPs) to match the loss achieved by LeRoPE. This advantage is persistent, with the compute multiplier for RoPE plateauing around 1.03–1.04 relative to LeRoPE.
• Robustness: The improvement holds across multiple random seeds. For the 217M model, the gap between LeRoPE and RoPE (0.0066 nats) is 3 to 7 times larger than the standard deviation of seed variability.

2. Emergence of a Dominant Positional Band

Analysis of the learned frequencies reveals a consistent pattern across scales and seeds:

• Frequency Suppression: The model learns to suppress (drive toward zero) the frequencies of the slowest bands, effectively removing their rotation.
• Dominant Band: A single frequency band emerges with a large norm and a strong positional contribution to the attention logits. This "dominant positional band" has a wavelength ($\lambda$) that scales linearly with the training context length ($L_{train}$), specifically $\lambda \approx 2.2 \times L_{train}$.
• Mechanism: While standard RoPE distributes positional information across many bands, LeRoPE concentrates the positional signal into this specific dominant band, while the remaining bands appear to handle other signal types or cancel out.

3. Ablation and Generalization

• Frozen Frequencies: Training a model with frequencies frozen to the values learned by a separate LeRoPE run captures 63.6% of the performance gain, indicating that the discovery of the optimal frequency set is a primary driver of success, though joint training dynamics contribute the remainder.
• Code Pretraining: Experiments on the StarCoder-Python dataset show similar gains (0.0104 nats improvement over RoPE) and similar frequency patterns, suggesting the method generalizes beyond natural language.
• Extrapolation: While LeRoPE degrades more sharply than RoPE under naive extrapolation (due to the dominant band's sinusoidal contribution becoming positive outside the training window), it remains compatible with standard context-extension techniques like NTK-by-parts and YaRN. When combined with these methods, LeRoPE outperforms RoPE and p-RoPE at sequence lengths up to 32k tokens.

Significance and Claims

The paper claims that LeRoPE offers a simple, parameter-efficient, and universally applicable improvement to the positional encoding of language models.

• Learnability over Heuristics: The work demonstrates that treating frequency bands as learnable parameters allows the model to automatically discover a more efficient spectral allocation than manually tuned heuristics (like p-RoPE or fixed base adjustments).
• Efficiency: The method provides a consistent "free" performance boost, reducing the compute budget required to reach a target performance level by approximately 3.4% at scale.
• Mechanistic Insight: The emergence of a single dominant positional band provides new insight into how models utilize frequency spectra, suggesting that a specific, learned frequency is sufficient to encode the necessary positional signal for in-distribution tasks, while slower bands are better utilized for semantic content.

The authors remain modest regarding the scope, noting that experiments are limited to models up to 2.5B parameters and that the gains on overtrained, larger-scale models remain to be fully demonstrated. However, the consistency of the results across the scaling ladder and different datasets (C4 and Code) suggests the method is robust and likely beneficial for larger foundation models.