🤖 machine learning

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

This paper proposes geometry-preserving orthonormal initialization for Low-Rank Adaptation (LoRA) in Reinforcement Learning with Verifiable Rewards (RLVR), theoretically demonstrating that this approach minimizes the gap to full fine-tuning, stabilizes training, and outperforms existing variants like PiSSA and MiLoRA on mathematical reasoning benchmarks.

Original authors: Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart library (a Large Language Model) that has already read almost everything in the world. You want to teach it a new, specific skill, like solving complex math problems.

There are two ways to do this:

Full Fine-Tuning: You rewrite the entire library's catalog and reorganize every single book. This is powerful but requires a massive amount of energy and storage (like needing a whole new warehouse).
LoRA (Low-Rank Adaptation): Instead of rewriting everything, you just add a small, sticky-note index card to the front of the library. This card contains the new instructions. It's cheap, fast, and doesn't require a new warehouse.

The Problem: The "RLVR" Workout

Recently, a new way of teaching these models called RLVR (Reinforcement Learning with Verifiable Rewards) became popular. Think of this not as a classroom lecture, but as a high-intensity gym workout.

In a classroom (Supervised Fine-Tuning), the teacher just corrects your homework.
In the gym (RLVR), the model tries many different answers, gets a score (reward) based on whether it's right or wrong, and learns from the experience.

The problem is that the "sticky-note index cards" (LoRA) that worked perfectly in the classroom started failing in the gym. Some advanced versions of these cards (called PiSSA and MiLoRA) actually made the model collapse, like a weightlifter trying to lift a barbell that was too heavy right from the start. The training became unstable, and the model stopped learning.

The Investigation: Why Did the Cards Break?

The researchers looked at why these advanced cards failed. They found two main culprits:

The "Heavy" Start: The advanced cards tried to grab the "most important" parts of the library's existing knowledge (the heavy, popular books) and amplify them immediately. In the gym setting, this was like trying to sprint before you've even stretched. It caused the model to move too fast and too far, breaking the rules of the workout.
The Wrong Direction: The gym (RLVR) needs the model to explore new directions, not just reinforce what it already knows. The old cards were too focused on the "main" paths, causing the model to get stuck or spin out of control.

The Solution: The "Geometry-Preserving" Index Card

The researchers realized that to survive the gym, the index card needs to be orthonormal.

The Analogy:
Imagine the library's knowledge is a 3D space.

Standard LoRA is like drawing a random line on a piece of paper. It works, but it's a bit messy.
PiSSA/MiLoRA are like drawing a line that tries to hug the heaviest books on the shelf. In the gym, this line is too "stiff" and snaps.
The New Method (LoRA-RLPO/RLMO): The researchers designed a new index card that is perfectly rigid and structured (orthonormal). It doesn't try to amplify the heavy books; instead, it aligns perfectly with the library's existing geometry without adding extra weight.

Think of it like a dance partner.

The old advanced cards tried to lead the dance with too much force, causing the partner (the model) to stumble.
The new method is a partner who moves in perfect sync with the music (the library's existing structure) but doesn't push too hard. It preserves the shape of the dance floor so the model can move freely without falling over.

What They Did

They created two new types of index cards:

LoRA-RLPO: Aligns with the "main" paths but keeps the structure light and balanced.
LoRA-RLMO: Aligns with the "minor" paths (the less obvious routes) but also keeps the structure light and balanced.

The Results

When they tested these new cards in the "gym" (math reasoning benchmarks):

Stability: The training didn't crash. The model stayed calm and steady.
Performance: The model learned faster and got better scores than the standard method.
The "Why": They proved mathematically that by keeping the index card "orthonormal" (perfectly structured) and not amplifying the weight of the existing knowledge, the model stays close to the ideal "full rewrite" performance without needing the massive energy cost.

Summary

The paper says: "If you want to train a smart AI using the new 'gym' method (RLVR), don't use the fancy, heavy index cards that worked in the classroom. Instead, use our new, perfectly structured, lightweight cards. They keep the model stable, prevent it from breaking, and help it learn math problems much better."

They also noted that this works for different sizes of models and even for coding tasks, but the core discovery is about how to start the training so it doesn't explode.

Technical Summary: Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

Problem Statement

While Low-Rank Adaptation (LoRA) and its variants have become standard for parameter-efficient fine-tuning under the Supervised Fine-Tuning (SFT) paradigm, their behavior under Reinforcement Learning with Verifiable Rewards (RLVR) remains poorly understood. Recent observations indicate that SVD-based initialization strategies, specifically PiSSA (Principal Singular values and Singular vectors Adaptation) and MiLoRA (Minor LoRA), which excel in SFT, often underperform standard LoRA and exhibit training instability in RLVR settings. These methods initialize adapters using the top-r (PiSSA) or bottom-r (MiLoRA) singular components of pretrained weights, often incorporating singular value scaling. The paper posits that the distinct optimization dynamics of RLVR—characterized by implicit KL-divergence constraints that limit policy drift—render SFT-optimized initialization principles ineffective or even detrimental. The core challenge is identifying an initialization strategy that stabilizes RLVR training while maintaining the efficiency of low-rank adaptation.

Methodology

The authors propose a theoretical analysis of LoRA optimization dynamics in RLVR to derive a principled initialization strategy.

Theoretical Analysis

The paper establishes that the approximation gap between LoRA and full fine-tuning is minimized when the low-rank matrix $A$ is initialized with orthonormal rows and $B$ is initialized to zero.

Optimization Dynamics: Under the assumption of $L$ -smooth loss and bounded gradients, the paper proves (Theorem 5.2) that the error between the LoRA trajectory and the full fine-tuning trajectory is controlled by the term $\|I_n - A_0^\top A_0\|_2$ . This term achieves its minimum value of 1 if and only if $A_0$ has orthonormal rows ( $A_0 A_0^\top = I_r$ ).
Stability Bounds: Furthermore, Proposition 5.3 demonstrates that orthonormal initialization with $B_0=0$ guarantees that the first-step weight update magnitude does not exceed that of full fine-tuning. This bounded update is critical for satisfying the implicit KL-divergence constraints in RLVR, preventing the aggressive policy shifts that lead to training collapse.

Proposed Initialization Strategies

Guided by these theoretical insights, the authors propose two "Geometry-Preserving Orthonormal" initialization methods that retain the spectral information of pretrained weights without the destabilizing singular value scaling found in PiSSA and MiLoRA:

LoRA-RLPO (Principal): Initializes $A_0$ using the top- $r$ right singular vectors ( $V_r^\top$ ) of the pretrained weight matrix $W_0$ , with $B_0 = 0$ . This aligns the adapter with the principal directions of the pretrained model while maintaining orthonormality.
LoRA-RLMO (Minor): Initializes $A_0$ using the bottom- $r$ right singular vectors ( $V_{-r}^\top$ ) of $W_0$ , with $B_0 = 0$ . This targets the minor subspace but, unlike MiLoRA, avoids singular value scaling and non-zero $B_0$ .

Both methods differ from PiSSA and MiLoRA by setting $B_0=0$ and using orthonormal $A_0$ without scaling by singular values ( $\Sigma^{1/2}$ ).

Key Results

Experiments were conducted on mathematical reasoning benchmarks (GSM8K, MATH500, AIME 2022/2023/2024) using the DAPO algorithm on DeepSeek-R1-Distill-Qwen-1.5B and Qwen2.5-7B models.

Performance: LoRA-RLPO achieved the highest average accuracy (65.03%) across benchmarks, outperforming standard LoRA (62.40%), LoRA-RLMO (63.76%), and significantly surpassing PiSSA (4.57%) and MiLoRA (51.71%).
Training Stability: Monitoring KL divergence and gradient norms revealed that PiSSA and MiLoRA suffer from rapid KL growth and gradient explosion, leading to reward collapse. In contrast, LoRA-RLPO and LoRA-RLMO maintained low KL divergence trajectories comparable to or lower than standard LoRA, indicating stable optimization.
Ablation Studies:
- Orthonormality vs. Geometry: Model-agnostic orthonormal baselines (DCT-LoRA, Wavelet-LoRA) outperformed standard LoRA, confirming the benefit of orthonormality alone. However, LoRA-RLPO significantly outperformed these model-agnostic methods, demonstrating that preserving the geometric structure of the pretrained weights (via SVD) provides additional gains.
- Failure Analysis: The paper refutes the hypothesis that MiLoRA fails due to near-zero initialization magnitudes, showing that its initial adapter norm is non-negligible. Instead, the instability is attributed to the combination of subspace geometry and singular value scaling, which amplifies updates beyond the KL budget.
- Generalization: The proposed methods also demonstrated improved convergence and performance in SFT tasks (GLUE, GSM8K), suggesting the benefits extend beyond RLVR.

Significance and Claims

The paper claims to provide the first theoretical explanation for why SVD-based LoRA variants (PiSSA, MiLoRA) fail in RLVR while succeeding in SFT. The authors identify two coupled factors driving this failure:

Subspace Geometry: The choice of singular subspace dictates the optimization trajectory.
Singular Value Scaling: The scaling factor in PiSSA/MiLoRA amplifies update magnitudes, causing rapid violations of the implicit KL-divergence constraint required for stable RLVR training.

The significance of this work lies in establishing that orthonormal initialization, paired with the standard choice of $B_0=0$ , is a principled solution for low-rank adaptation in RLVR. By minimizing the approximation gap to full fine-tuning and bounding update magnitudes, geometry-preserving orthonormal initialization (LoRA-RLPO/RLMO) offers a robust foundation for parameter-efficient RL, resolving the instability observed in prior SVD-based methods. The authors emphasize that their analysis offers a unified framework explaining both the failure of existing variants and the success of their proposed approach.