🤖 machine learning

LoCO: Low-rank Compositional Rotation Fine-tuning

LoCO is a novel parameter-efficient fine-tuning method that constructs orthogonal transformations using low-rank skew-symmetric matrices and compositional rotation chains to preserve the geometric structure of pretrained representations while enabling efficient parallel computation across diverse domains.

Original authors: An Nguyen, Jaesik Choi, Anh Tong

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: An Nguyen, Jaesik Choi, Anh Tong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart robot that has already learned to speak every language, recognize every object, and paint beautiful pictures. This robot is like a "foundation model" in AI. It's so big and powerful that training it from scratch would take a supercomputer years to run.

Now, imagine you want to teach this robot a specific new trick, like writing poetry in the style of Shakespeare or fixing blurry photos. The old way of doing this was to "fine-tune" the whole robot—essentially rewriting its entire brain. This is like trying to teach a new dance to a 1,000-person choir by making every single person learn new steps. It's expensive, slow, and risks making the choir forget how to sing their original songs.

The Problem with Current "Smart" Tricks
To save time, scientists developed "Parameter-Efficient Fine-Tuning" (PEFT). Instead of rewriting the whole brain, they add a tiny, lightweight "adapter" layer. Think of this like giving the choir a set of new sheet music to hold while they sing.

However, most current adapters work by adding new notes to the song. The paper argues that this is a bit clumsy. If you just add notes, you might accidentally change the volume or the rhythm of the original song, distorting the robot's original understanding.

Some newer methods try to rotate the notes instead of adding them. Imagine spinning the choir's formation 90 degrees. This keeps the distance between every singer exactly the same, preserving the perfect harmony of the original song. This is called "Orthogonal Fine-Tuning."

The Bottleneck: The "Spinning" Problem
Here's the catch: Spinning a giant choir (or a high-dimensional AI model) is mathematically heavy. To do a perfect rotation, you usually have to solve a massive, complex puzzle that takes a long time and a lot of memory. It's like trying to spin a 1,000-person choir in a tiny room; you'd need a huge amount of space and time to get everyone to move without bumping into each other.

Existing methods tried to solve this by breaking the choir into small, separate groups (block-diagonal) or using a complex, step-by-step dance routine (butterfly factorization). But these methods either limit how creative the dance can be or are still too slow because the dancers have to wait for the person in front of them to move first.

The Solution: LoCO (Low-rank Compositional Orthogonal Fine-tuning)
The authors of this paper introduce LoCO, a new way to teach the robot that is like a "magic spinning trick."

The Low-Rank Shortcut: Instead of trying to spin the whole giant choir at once, LoCO realizes that you only need to move a few key people to change the whole formation's direction. They use a mathematical trick (called the Sherman-Morrison-Woodbury identity) that lets them calculate the rotation of the entire choir by only doing the math for a tiny handful of people. It's like calculating the spin of a whole galaxy by only tracking a few stars.
The Parallel Dance: Usually, to create a complex rotation, you have to do it step-by-step (Spin left, then spin up, then spin right). LoCO uses a clever approximation that lets them do all these spins at the same time. It's like telling the whole choir to spin in different directions simultaneously, rather than waiting for a line of people to pass a baton. This makes the process incredibly fast and efficient.
The Temperature Knob: One of the coolest features of LoCO is a "temperature" setting. Imagine you've taught the choir a new dance. Sometimes you want them to do the dance exactly as taught; other times, you want them to do it with a little bit of their original style mixed in. LoCO has a simple dial (a temperature parameter) that lets you smoothly slide between the "original robot" and the "newly trained robot" without having to retrain the model. It's like a volume knob for the new skill.

What They Found
The team tested LoCO on three different types of AI robots:

Language Models: Teaching them to understand grammar and solve math problems.
Diffusion Transformers: Teaching them to generate images based on text or sketches.
Vision Transformers: Teaching them to recognize objects in photos.

In every test, LoCO performed as well as or better than the existing methods. It was faster to train, used less computer memory, and didn't distort the robot's original knowledge.

In a Nutshell
LoCO is a new, efficient way to teach giant AI models new skills. Instead of rewriting their brains or clumsily adding new parts, it gently "rotates" their existing knowledge using a fast, parallel mathematical shortcut. It preserves the robot's original intelligence while allowing it to learn new tasks quickly, and it even gives you a dial to control exactly how much of the new skill you want to use.

Technical Summary: LoCO - Low-rank Compositional Rotation Fine-tuning

Problem Statement

Parameter-efficient fine-tuning (PEFT) has become essential for adapting large-scale foundation models (e.g., LLMs, Vision Transformers, Diffusion Transformers) to downstream tasks without the prohibitive computational cost and risk of catastrophic forgetting associated with full fine-tuning. While existing methods like Low-Rank Adaptation (LoRA) achieve efficiency through additive low-rank updates, they do not inherently preserve the geometric structure of pretrained representations.

Orthogonal fine-tuning methods address this by applying multiplicative transformations constrained to the special orthogonal group ($SO(d)$), which preserve pairwise angular relationships and the hyperspherical energy structure of features. However, these methods face a fundamental computational bottleneck: constructing and applying general orthogonal matrices typically requires $O(d^3)$ complexity due to matrix inversion or exponential computations. Prior solutions, such as block-diagonal matrices (OFT) or butterfly factorizations (BOFT), impose structural constraints that limit expressivity or suffer from inefficient memory access patterns on modern hardware. Furthermore, methods based on sequential compositions (e.g., Householder reflections) hinder parallelization, limiting training throughput.

Methodology

The authors propose LoCO (Low-rank Compositional Orthogonal Fine-tuning), a method that constructs orthogonal transformations using low-rank skew-symmetric matrices and compositional rotation chains, optimized for parallel computation.

1. Low-Rank Skew-Symmetric Parameterization

LoCO leverages the mathematical property that the tangent space of the special orthogonal group $SO(d)$ consists of skew-symmetric matrices ( $A^T = -A$ ). Instead of learning a full $d \times d$ skew-symmetric matrix, LoCO parameterizes it using a low-rank outer product form:
$A = UV^T - VU^T$
where $U, V \in \mathbb{R}^{d \times r}$ and $r \ll d$ . This is rewritten using auxiliary matrices $X = [U | -V]$ and $Y = [V | U]$ such that $A = XY^T$ .

2. Efficient Computation via Cayley Transform and Woodbury Identity

To generate the rotation matrix $R \in SO(d)$ from the skew-symmetric matrix $A$ , LoCO employs the Cayley transform:
$R = (I - A)^{-1}(I + A)$
Direct inversion of the $d \times d$ matrix $(I-A)$ is computationally expensive. LoCO utilizes the Sherman-Morrison-Woodbury identity to reduce the inversion complexity. By substituting $A = XY^T$ , the inversion of a $d \times d$ matrix is converted to the inversion of a small $2r \times 2r$ matrix:
$R = I + 2X(I - Y^T X)^{-1}Y^T$
This reduces the computational complexity from $O(d^3)$ to $O(dr^2 + r^3)$ , which simplifies to $O(dr^2)$ given $r \ll d$ .

3. Compositional Rotation Chains and Parallel Approximation

To enhance expressivity, LoCO composes multiple rotation components: $R = R_1 R_2 \dots R_n$ . While exact composition requires sequential matrix multiplication, LoCO introduces a first-order approximation that enables fully parallel computation:
$R \approx I + \sum_{i=1}^n 2X_i(I - Y_i^T X_i)^{-1}Y_i^T$
Theoretical analysis confirms that for fine-tuning scenarios where parameter updates are small (near zero initialization), the higher-order error terms are negligible ( $O(\epsilon^4)$ and above). This approximation allows all rotation components to be computed and summed in parallel, fully utilizing GPU parallelism and avoiding the sequential bottlenecks of methods like BOFT or HRA.

4. Test-Time Temperature Scaling

LoCO introduces a scalar temperature parameter $t$ applied to the skew-symmetric matrix $A$ during inference ( $R = (I - At)^{-1}(I + At)$ ). This allows for smooth interpolation between the pretrained behavior ( $t=0$ ) and the fully adapted behavior ( $t=1$ ), providing controllable adaptation strength without retraining.

Key Contributions

Novel PEFT Architecture: The proposal of LoCO, which constructs orthogonal transformations via low-rank skew-symmetric matrices, enabling efficient adaptation in high-dimensional spaces.
Parallelizable Approximation: The introduction of a first-order approximation scheme for compositional rotation chains. This enables parallel computation of multiple rotation components while preserving orthogonality with bounded approximation error, a feature lacking in prior sequential orthogonal methods.
Controllable Inference: The development of a test-time temperature scaling mechanism that allows flexible control over adaptation strength, offering an alternative to costly regularization schemes used in previous orthogonal methods.
Empirical Validation: Demonstration of state-of-the-art or competitive performance across diverse benchmarks (GLUE, GSM8K, MATH, VTAB-1k, and controllable diffusion generation) while maintaining lower memory footprints and higher training efficiency compared to OFT, BOFT, and HRA.

Experimental Results

The authors evaluated LoCO across three domains:

Large Language Models (NLU & Reasoning): On the GLUE benchmark, LoCO achieved the highest average score (87.75), outperforming HRA (85.64) and OFT (87.63) while using fewer parameters. In mathematical reasoning (LLaMA-2-7B on GSM8K/MATH), LoCO achieved the best GSM8K score (50.19%) and competitive MATH results, surpassing block-diagonal and Householder-based baselines.
Diffusion Transformers: In conditional generation tasks (e.g., edge-to-image, depth-to-image) using FLUX.1, LoCO demonstrated superior alignment fidelity (CLIP Image scores) and competitive image quality metrics (FID, SSIM) compared to LoRA and BOFT. Notably, the temperature scaling mechanism allowed for robust generation quality across varying adaptation strengths, unlike the brittle scaling observed in LoRA.
Vision Transformers (VTAB-1k): LoCO achieved competitive accuracy across Natural, Specialized, and Structured task groups. It notably outperformed other orthogonal methods on structured tasks (63.4% mean accuracy) and showed strong transfer to medical imaging (Patch Camelyon: 88.1%).

Efficiency: LoCO matched the training efficiency of sparse OFT while significantly outperforming dense methods (HRA, BOFT) in both step time and memory usage. HRA and BOFT incurred substantial memory overheads due to dense matrix materialization or sequential dependencies, whereas LoCO's parallel, input-centric design maintained stable throughput.

Significance and Claims

The paper claims that LoCO successfully bridges the gap between the geometric benefits of orthogonal fine-tuning and the computational efficiency required for modern high-dimensional models. By exploiting the low-rank structure of the tangent space and utilizing a parallelizable first-order approximation, LoCO avoids the cubic complexity and sequential bottlenecks of existing orthogonal methods.

The authors emphasize that their method preserves the angular structure of pretrained representations—a critical property for tasks where semantic information is encoded in angles rather than magnitudes—while offering practical advantages such as controllable inference strength and superior scalability. The results suggest that low-rank skew-symmetric parameterization is a more effective strategy for capturing reasoning patterns and semantic alignment than block-diagonal or Householder reflection structures.

The paper concludes modestly, noting that while LoCO addresses scalability for typical scenarios, future work could explore theoretical optimizations for optimal rotation decompositions and extensions to multimodal models.