💻 computer science

Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projection

This paper proposes Ortho-LoRA, a gradient projection method that dynamically resolves task conflicts in Multi-Task LoRA by projecting conflicting gradients onto orthogonal subspaces, thereby significantly mitigating negative transfer and recovering near-single-task performance with negligible computational overhead.

Original authors: Ziyu Yang, Guibin Chen, Yuxin Yang, Aoxiong Zeng, Xiangquan Yang

Published 2026-01-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziyu Yang, Guibin Chen, Yuxin Yang, Aoxiong Zeng, Xiangquan Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: One Brain, Many Jobs

Imagine you have a giant, incredibly smart library (a Large Language Model) that knows almost everything. However, the library is so huge that you can't carry it around in your pocket. To make it portable, you create a small, lightweight "notebook" (called LoRA) that you attach to the library. This notebook allows the library to learn new, specific skills without rewriting the entire encyclopedia.

Usually, you give the library one job at a time (like "write a poem" or "solve a math problem"). But what if you want one single notebook to handle many jobs at once? That is the goal of Multi-Task Learning.

The Problem: The "Traffic Jam" in a Small Room

The paper argues that while sharing one notebook saves space, it causes a traffic jam.

The Scenario: Imagine the notebook is a tiny room with only a few chairs (this is the Low-Rank constraint). You have three friends (Task A, Task B, and Task C) trying to sit down and give you instructions at the same time.
The Conflict: Friend A wants to sit in the corner to write a story. Friend B wants to sit in the same corner to solve a puzzle. Friend C wants to stand in the middle to analyze a sentence.
The Result: Because the room is so small (low capacity), their instructions clash. Friend A pushes Friend B out of the way. Friend C gets bumped. In technical terms, their "gradients" (the directions they want to move the learning) are fighting each other. This is called Negative Transfer. The more tasks you add, the more they fight, and the worse the notebook gets at all of them compared to if you had given each friend their own private notebook.

The Solution: Ortho-LoRA (The "Magic Dance Floor")

The authors propose a new method called Ortho-LoRA. Instead of letting the friends fight over the same spot, they use a special technique to make their instructions "orthogonal" (perpendicular) to each other.

Think of it like a dance floor with invisible lanes:

The Old Way (Joint Training): Everyone tries to dance in the middle of the floor. They bump into each other, step on toes, and the dance is messy.
The Ortho-LoRA Way: The system acts like a dance instructor. It tells Friend A, "You dance North-South." It tells Friend B, "You dance East-West." Even though they are on the same floor, their movements don't interfere because they are moving in different, perpendicular directions.

How it works technically (in simple terms):

The notebook has two main parts: one that reads the input (Matrix A) and one that writes the output (Matrix B).
When the system detects that Friend A and Friend B are trying to push the notebook in opposite directions, it mathematically "projects" Friend A's instruction onto a plane where it doesn't push against Friend B.
Crucially, they do this separately for the "reading" part and the "writing" part. This means the notebook can learn to read a story and a math problem in the same way, but write the answers differently, without the two tasks canceling each other out.

The Results: Getting the Best of Both Worlds

The researchers tested this on a standard set of language tests (GLUE benchmark). Here is what they found:

The Baseline: When they just mashed all tasks together without help (Joint-LoRA), the performance dropped significantly. It was like trying to juggle three balls while blindfolded; you dropped them all.
The Gold Standard: If you train a separate notebook for every single task, performance is perfect, but it takes up 3x more memory (too heavy to carry).
Ortho-LoRA: By using the "perpendicular lane" method, the single shared notebook performed almost as well as the three separate notebooks.
- They recovered 95% of the performance gap between the messy "all-in-one" approach and the perfect "separate" approach.
- It did this with negligible extra computing power. The "dance instructor" (the projection math) is so fast it barely slows things down.

Why This Matters

The paper concludes that Ortho-LoRA is a smart, efficient way to let one small AI module handle many different jobs without them fighting each other. It proves you don't need to build a complex, expensive machine with many separate parts (like some other methods that try to split tasks into different "experts") to get good results. You just need to teach the tasks how to move in different directions on the same small stage.

In short: Ortho-LoRA stops different tasks from stepping on each other's toes, allowing a single, tiny AI adapter to learn multiple skills almost as well as if it had learned them one by one.

Technical Summary: Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projection

1. Problem Statement

The paper addresses the challenge of negative transfer in Multi-Task Learning (MTL) when applied to Large Language Models (LLMs) using Low-Rank Adaptation (LoRA). While LoRA offers a parameter-efficient way to fine-tune LLMs by updating only low-rank matrices ( $A$ and $B$ ), sharing a single adapter across multiple tasks often leads to performance degradation compared to single-task fine-tuning.

The authors argue that this issue is exacerbated in LoRA due to the "bottleneck conflict." Unlike full fine-tuning, which operates in a high-dimensional space with ample degrees of freedom to find a "Pareto stationary" point for conflicting objectives, LoRA constrains optimization to a low-rank manifold (e.g., rank $r=8$ ). This severe capacity reduction forces task-specific gradients to collide more frequently, resulting in destructive interference where improvements in one task come at the expense of others.

2. Methodology: Ortho-LoRA

The proposed solution, Ortho-LoRA, is a gradient projection method tailored specifically to the bipartite algebraic structure of LoRA.

Core Mechanism: Drawing inspiration from PCGrad (Projecting Conflicting Gradients), Ortho-LoRA dynamically projects conflicting task gradients onto the orthogonal complement of each other before parameter updates. If the cosine similarity between two task gradients is negative, the gradient of one task is projected onto the normal plane of the other to remove the conflicting component.
Structure-Aware Decoupling: A key innovation is the application of this projection to the constituent LoRA matrices ( $A$ $A$ and $B$ $B$ ) independently, rather than flattening all parameters into a single vector.
- $A$ projects inputs into a latent subspace (feature extraction).
- $B$ projects latent features back to the output space (state reconstruction).
- By enforcing orthogonality separately on $g^{(A)}$ and $g^{(B)}$ , the method allows the model to share useful input representations even if output requirements differ, preserving transfer information that might be lost in a global projection.
Training Procedure: The algorithm computes gradients for each task separately, detects conflicts, and applies the projection. To ensure unbiased optimization in multi-task settings ( $T > 2$ ), the order of projection is randomized at every training step.
Complexity: While the method requires $T$ backward passes (one per task) instead of one, the computational overhead is negligible. The projection operates solely on the LoRA parameters, which constitute less than 0.1% of the total model size. The primary computational bottleneck remains the forward/backward pass through the frozen LLM backbone.

3. Key Contributions

The paper outlines three primary contributions:

Problem Identification: The authors identify and analyze the "bottleneck conflict" inherent in applying MTL to low-rank adapters, arguing that low-rank constraints amplify negative transfer compared to full fine-tuning.
Algorithm Proposal: They propose Ortho-LoRA, a computationally efficient algorithm that enforces gradient orthogonality specifically within the adapter space (matrices $A$ and $B$ ) to mitigate negative transfer without architectural modifications.
Empirical Validation: Through experiments on the GLUE benchmark, they demonstrate that Ortho-LoRA recovers approximately 95% of the performance gap between multi-task and single-task baselines while using only $1/N$ of the parameters (where $N$ is the number of tasks).

4. Experimental Results

Experiments were conducted on the GLUE benchmark (MNLI, QQP, SST-2) using a RoBERTa-base backbone with LoRA rank $r=8$ .

Performance: Standard Joint-LoRA (shared adapter trained on summed loss) showed a performance drop of ~1.5 points compared to Single-Task LoRA. Ortho-LoRA significantly mitigated this, achieving an average score of 89.6, which is only 0.3 points behind the Single-Task upper bound (89.9).
Recovery Rate: Ortho-LoRA recovered roughly 80% of the performance drop across the board, with near-perfect recovery on the QQP task (87.9 vs. 88.1 for single-task).
Ablation Studies:
- Rank Sensitivity: The method provided consistent gains across ranks $r \in \{4, 8, 16, 32\}$ . The improvement was most pronounced at lower ranks ( $r=4$ ), confirming the method's value in highly parameter-efficient scenarios where the bottleneck is most severe.
- Convergence: Although Ortho-LoRA took 1.4x longer in wall-clock time due to multiple backward passes, it converged in fewer epochs than Joint-LoRA. It stabilized the optimization trajectory, avoiding the oscillating loss curves caused by conflicting updates.

5. Significance and Claims

The paper claims that Ortho-LoRA offers a robust "middle-ground" solution between the efficiency of joint training and the performance of single-task fine-tuning.

Efficiency vs. Performance: It demonstrates that explicit gradient orthogonality can achieve task disentanglement comparable to complex architectural solutions (like Mixture-of-Experts or FlyLoRA) without introducing additional parameters (routers) or complex routing logic.
Scalability: The method maintains the structural simplicity of a single shared LoRA module, making it suitable for deploying a single system to handle multiple downstream tasks with minimal storage overhead.
Future Directions: The authors modestly suggest that future work could explore applying this projection strategy to other PEFT methods (e.g., Prefix-Tuning) and combining it with dynamic task weighting.

In summary, Ortho-LoRA provides a lightweight, optimization-level intervention that effectively resolves the specific gradient conflicts arising from the low-rank constraints of LoRA, enabling high-performance multi-task adaptation with negligible computational cost.