Technical Summary: CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

1. Problem Statement

Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting Large Language Models (LLMs) to downstream tasks without the prohibitive cost of full fine-tuning. While Low-Rank Adaptation (LoRA) is the dominant method, it learns an additive update $\Delta W = BA$ that bundles geometric degrees of freedom (basis rotation and spectral shifts) into a single low-rank matrix. Recent SVD-based methods (e.g., PiSSA, MiLoRA) utilize the singular value decomposition (SVD) of pretrained weights ($W_0 = U_0 \Sigma_0 V_0^\top$) for initialization or subspace selection but do not explicitly preserve the coupled geometry between the left ($U_0$) and right ($V_0$) singular bases.

Recent theoretical work [8] suggests that stable fine-tuning follows a "coherent SVD rotation" form, where a single orthogonal matrix $Q$ acts on both the left and right singular bases simultaneously, accompanied by a diagonal shift in singular values ($\Delta \Sigma$). Existing methods only partially realize this: additive LoRA-style methods entangle rotation and spectrum, while orthogonal fine-tuning methods enforce rotation but often lack spectral flexibility. Furthermore, the global coherent rotation theorem does not specify how to implement this efficiently within the structured, low-rank constraints of practical adapters.

2. Methodology: CORA

The authors propose CORA (Coherent Orthogonal Rotation Adaptation), a method that extends the global coherent rotation theorem to a per-slice granularity.

2.1 Theoretical Foundation

The paper proves a per-slice analogue of the coherent rotation theorem (Theorem 2). By partitioning a weight matrix $W_0$ into row slices $W_i \in \mathbb{R}^{r \times k}$, the authors show that under mild regularity assumptions, each slice's optimal fine-tuned form is:
$$W_i^* = U_i Q_i (\Sigma_i + \Delta \Sigma_i) Q_i^\top V_i^\top$$
where $Q_i$ is a single orthogonal matrix acting on both the left basis $U_i$ and right basis $V_i$ of the slice. This establishes that the left and right in-basis rotations must be identical ($Q_{U,i} = Q_{V,i} = Q_i$) to minimize the Frobenius perturbation.

2.2 Parameterization and Implementation

To realize this form efficiently without expensive manifold constraints, CORA employs a specific reparameterization:

1. Source Matrix: The method operates on the rank-$r$ SVD truncation of the pretrained weight, $W_{lr} = U_{0,r} \Sigma_{0,r} V_{0,r}^\top$, freezing the residual $W_0 - W_{lr}$.
2. Slice Partitioning: $W_{lr}$ is partitioned into $s = m/r$ row slices.
3. Rotation Learning: Instead of learning $Q_i$ directly on the orthogonal manifold, CORA learns a single unconstrained orthogonal matrix $R_i$ per slice and defines the rotation as $Q_i = U_i^\top R_i U_i$. This similarity transform ensures that the coherent rotation condition ($Q_{U,i} = Q_{V,i}$) is satisfied identically without explicit constraints or penalties during training.
4. Spectral Scaling: An optional per-layer diagonal scale vector $\delta$ is applied to the singular values ($\Sigma_i \to \Sigma_i \text{diag}(1+\delta)$).
5. Cayley Map: To ensure $R_i$ remains orthogonal, it is parameterized via a closed-form Cayley map of a learnable skew-symmetric matrix, avoiding convergence radius constraints.

2.3 Parameter Efficiency

CORA uses $\frac{1}{2}m(r-1)$ trainable parameters per linear layer (plus optional $r$ scalars for $\delta$). This is approximately 4$\times$ fewer parameters than LoRA at the same rank $r$ (which uses $r(m+k)$), and roughly 8$\times$ fewer than LoRA when comparing performance-equivalent configurations.

3. Key Contributions

1. Theoretical Extension: Proved that the coherent rotation theorem applies to individual row slices of a weight matrix, extending the global theory to the granularity of practical adapters.
2. Novel Parameterization: Introduced a reparameterization ($Q_i = U_i^\top R_i U_i$) that enforces the coherent rotation condition ($Q_{U,i} = Q_{V,i}$) as an algebraic identity, eliminating the need for orthogonality constraints during optimization.
3. Efficient Adapter Design: Implemented CORA as a compact adapter using per-slice orthogonal rotations and per-layer diagonal scaling, achieving high performance with significantly fewer parameters than existing SVD-based and general PEFT methods.

4. Experimental Results

CORA was evaluated on LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B across three task families: commonsense reasoning, code generation, and mathematical reasoning.

• Commonsense Reasoning: On LLaMA-2-7B, CORA with rank $r=16$ (6.6M parameters) achieved 82.16% average accuracy. This outperformed PiSSA and MiLoRA (both at $r=32$, ~56M parameters) by 8.4 and 3.0 points respectively, and surpassed DoRA ($r=32$) by 2.5 points.
• Code Generation: On LLaMA-3-8B, CORA ($r=16$, 10.3M parameters) achieved 48.2 HumanEval Pass@1, exceeding DiaBlo ($N=64$, 121M parameters) and LoRI ($r=32$, 45M parameters) by significant margins. On Mistral-7B, CORA maintained strong performance (40.2 Pass@1) with only 10.3M parameters.
• Mathematical Reasoning: CORA performed competitively on GSM8K but lagged on MATH compared to MiLoRA. The authors attribute this to the spectral locality of math knowledge; MiLoRA's focus on minor components (residual directions) appears better suited for tasks where principal components encode less task-relevant structure.
• Ablations:
  • Shared Scale ($\delta$): Enabling the shared spectrum scale was critical, improving performance by 0.8–2.4 points and preventing collapse at low ranks ($r=8$).
  • Two-Sided Rotation: Removing the $V$-side rotation (U-side only) caused significant performance drops (up to 9.8 points), confirming the necessity of the coherent rotation form.
  • Source Matrix: Using the full $W_0$ instead of the truncated $W_{lr}$ improved MATH performance, validating the hypothesis that tail singular directions contain math-specific signals.

5. Significance and Claims

The paper claims that CORA successfully unifies additive SVD-initialized methods and orthogonal rotation families under a single, theoretically grounded form. By explicitly enforcing the coherent rotation condition derived from minimum-perturbation theory, CORA achieves a superior parameter-accuracy trade-off.

The authors state that CORA reaches state-of-the-art results in commonsense reasoning and code generation with approximately 8$\times$ fewer parameters than competing methods like LoRA, DoRA, PiSSA, and MiLoRA. The work demonstrates that explicit geometric constraints (coherent rotation) can be implemented efficiently via algebraic reparameterization, offering a new direction for parameter-efficient adaptation that respects the underlying geometry of pretrained weights.

Limitations noted by the authors:

• Requires offline SVD computation and caching of per-slice factors before training.
• Introduces additional matrix multiplications during the forward pass compared to standard LoRA.
• Evaluated up to 8B parameter models; scalability to 30B+ models is unverified.
• Performance on math tasks is sensitive to the choice of source spectrum (principal vs. residual components).