GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a critical paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs), particularly in domains like mathematics, coding, and science. Unlike Supervised Fine-Tuning (SFT), which primarily injects knowledge by modifying principal weight directions, RLVR operates as a constrained optimization process that amplifies latent reasoning behaviors through reward-induced sampling bias.

Existing Parameter-Efficient Fine-Tuning (PEFT) methods face significant limitations in the RLVR regime:

• Geometric Mismatch: Standard Low-Rank Adaptation (LoRA) and its variants (e.g., PiSSA) are designed for SFT. They often initialize adapters based on the principal components of pre-trained weights. However, RLVR updates tend to occur in "off-the-principal" directions (low-magnitude, orthogonal to pre-trained features). Forcing updates onto principal components creates a geometric mismatch, leading to instability or suboptimal performance.
• Efficiency Bottlenecks: While some sparse fine-tuning methods align better with RLVR's update patterns, they rely on unstructured sparsity. Modern hardware (GPUs) struggles with unstructured sparse computations, resulting in poor practical efficiency and high overhead compared to dense operations.

The core challenge is to design a PEFT method that respects the specific optimization geometry of RLVR (anisotropic, compressible, and off-principal) while maintaining hardware-efficient dense computation.

2. Methodology: GeoRA

The authors propose GeoRA (Geometry-Aware Low-Rank Adaptation), a framework that explicitly aligns low-rank adaptation with the geometric structure of RLVR updates.

A. Core Insight

The authors hypothesize that the effective update subspace in RLVR is anisotropic and compressible, rather than isotropic. It exhibits a structured low-rank form that differs from the principal components used in SFT-oriented methods.

B. Architecture and Initialization

GeoRA modifies the standard LoRA architecture in two key ways:

1. Geometry-Constrained Initialization (SVD-based):

   • Instead of initializing adapters randomly or from the principal components of the full weight matrix $W$, GeoRA first constructs a geometry-constrained matrix $W_{Geo}$.
   • $W_{Geo}$ is derived by masking the original weights to select a stable, low-magnitude subspace. This involves two priors:
     • Spectral Prior ($M_{Spec}$): Selects entries from the rank-$r$ approximation of $W$ that have low magnitude (suppressing high-curvature/high-energy directions).
     • Euclidean Prior ($M_{Euc}$): Selects entries in the original weight space with low magnitude (capturing parameter plasticity).
   • The union of these masks defines $W_{Geo}$.
   • SVD Decomposition: GeoRA performs Singular Value Decomposition (SVD) on $W_{Geo}$ to extract the top-$r$ singular components. The trainable adapters $A_{Geo}$ and $B_{Geo}$ are initialized to approximate this geometry-aware subspace.

2. Frozen Residual Anchor:

   • To ensure the model's output remains unchanged at initialization and to preserve pre-trained capabilities, a Residual Matrix ($W_{res}$) is computed: $W_{res} = W - \frac{\alpha}{r} B_{Geo}A_{Geo}$.
   • During the forward pass, $W_{res}$ is frozen. The model computes $h = W_{res}x + \frac{\alpha}{r} B_{Geo}A_{Geo}x$.
   • This acts as a "structural anchor," preventing the erosion of pre-trained representations while allowing the trainable adapter to explore the specific RLVR update manifold.

C. Forward Pass

The forward computation remains dense and hardware-efficient:
$$h = W_{res}x + \frac{\alpha}{r} B_{Geo}A_{Geo}x$$
This avoids the overhead of unstructured sparse computation while restricting optimization to the geometry-aligned subspace.

3. Key Contributions

1. Novel Framework: GeoRA is the first geometry-aware low-rank adaptation framework explicitly designed for RLVR, bridging the gap between PEFT efficiency and RLVR optimization dynamics.
2. Theoretical Insight: The paper demonstrates that effective RLVR updates are directional and admit a compressible low-rank structure, distinct from the isotropic assumptions of standard LoRA or the principal-component focus of PiSSA.
3. Mechanism Design: The introduction of a frozen residual anchor combined with SVD-based initialization from a masked subspace ensures stability and prevents catastrophic forgetting.
4. Comprehensive Evaluation: Extensive experiments across multiple model scales (1.5B to 32B) and domains (Math, Medicine, Coding) validate the method's superiority.

4. Experimental Results

The authors evaluated GeoRA on Qwen and Llama models (1.5B to 32B parameters) using the GRPO algorithm.

• In-Distribution (Math Reasoning):
  • GeoRA consistently outperformed strong baselines (LoRA, PiSSA, MiLoRA, SparseFT) on challenging benchmarks like AIME24/25, OlymMATH, and MATH-500.
  • On the Qwen3-8B model, GeoRA achieved 23.75 on AIME24, surpassing Full Fine-Tuning (23.33) and PiSSA (22.50).
• Out-of-Distribution (Generalization):
  • GeoRA showed superior generalization on OOD tasks (HumanEval, GPQA, MMLU, TruthfulQA), indicating less catastrophic forgetting compared to other methods.
  • It maintained higher capability retention on instruction following and truthfulness tasks.
• Cross-Domain Performance:
  • GeoRA extended effectively to Medical (MedQA, PubMedQA) and Coding (LiveCodeBench, MBPP) RLVR tasks, consistently outperforming LoRA and competing with Full Fine-Tuning.
• Training Stability & Efficiency:
  • Stability: GeoRA reached peak performance faster and maintained stable reward trajectories under aggressive learning rates, whereas PiSSA and LoRA often suffered from collapse or slower convergence.
  • Efficiency: GeoRA reduced trainable parameters by 99.5% compared to Full Fine-Tuning. Unlike SparseFT, it achieved actual hardware speedups (19.9% faster per iteration) by utilizing dense matrix operations.
• Ablation Studies:
  • Removing the geometry-aware initialization (replacing with random or tail-r initialization) caused significant performance drops, confirming the importance of the specific initialization strategy.
  • Both the Spectral and Euclidean masks were shown to be complementary and necessary for optimal results.

5. Significance and Conclusion

GeoRA represents a significant advancement in the post-training of LLMs for reasoning tasks. By recognizing that RLVR operates on a different geometric manifold than SFT, the authors successfully adapted the low-rank paradigm to fit this regime.

• Scientific Impact: It challenges the assumption that principal components are always the best target for adaptation, highlighting the importance of "off-principal" updates in reinforcement learning.
• Practical Impact: It offers a solution that is both computationally efficient (dense, GPU-friendly) and highly effective (state-of-the-art performance with minimal parameters), making it a viable strategy for scaling RLVR on large models without the prohibitive cost of full fine-tuning or the inefficiency of unstructured sparsity.

The paper concludes that aligning parameter-efficient methods with the intrinsic optimization geometry of the learning task is crucial for achieving stable, generalizable, and efficient model adaptation.