🤖 AI

Shared LoRA Subspaces for almost Strict Continual Learning

The paper proposes "Share," a novel parameter-efficient continual learning method that dynamically updates a single shared low-rank subspace to integrate knowledge from diverse tasks and modalities without data replay or multiple adapters, achieving significant memory and parameter reductions while preventing catastrophic forgetting.

Original authors: Prakhar Kaushik, Ankit Vaidya, Shravan Chaudhari, Rama Chellappa, Alan Yuille

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Prakhar Kaushik, Ankit Vaidya, Shravan Chaudhari, Rama Chellappa, Alan Yuille

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly smart library (a large AI model) that knows a little bit about everything. Now, imagine you need to teach this library new skills every day: first how to write poetry, then how to diagnose car problems, then how to paint in the style of Van Gogh.

The Old Problem: The "One Book Per Skill" Mess
Traditionally, when you teach the library a new skill, you have to write a whole new book (a new set of instructions) for it. If you teach it 100 skills, you end up with 100 heavy books.

The Cost: Storing 100 books takes up a huge amount of shelf space (memory) and is expensive to manage.
The Forgetfulness: If you try to rewrite the same book to add new skills, the library often forgets the old skills. This is called "catastrophic forgetting." It's like trying to learn a new language by erasing your native one; you end up knowing neither well.

The "LoRA" Fix (and its limits)
Researchers came up with a clever trick called LoRA. Instead of writing a whole new book, you just add a small, thin "sticky note" to the existing library for each new skill. This saves space.

The Catch: If you have 100 skills, you still need 100 different sticky notes. You still have to manage a pile of 100 separate notes, and the library can't easily mix the knowledge between them.

The New Solution: "Share"
The paper introduces a method called Share. Think of Share not as a pile of sticky notes, but as a single, magical, expandable sketchbook.

Here is how it works, using a simple analogy:

1. The Shared Subspace (The "Universal Sketchbook")

Imagine that all the skills you want to learn (poetry, car repair, painting) actually share some fundamental "building blocks" or "directions" in how they work.

The Discovery: The authors found that if you look at the math behind these different skills, they all seem to converge toward the same few "directions" or "axes" in a hidden space.
The Analogy: Think of these directions as the primary colors (Red, Blue, Yellow). You don't need a separate paint tube for every single shade of green or purple you ever want to make. You just need the three primary colors and a way to mix them.

2. How "Share" Learns (The Process)

Instead of creating a new adapter for every task, Share does this:

Step 1: The Foundation. It starts by creating a small "foundational sketchbook" (a shared low-rank subspace). It looks at the first few tasks and figures out the "primary colors" (the main directions) needed to describe them.
Step 2: Learning New Skills. When a new task arrives (e.g., "Learn to bake bread"), the system doesn't write a new book or a new sticky note. Instead, it asks: "How can I mix the existing primary colors in my sketchbook to make 'bread'?"
- It only learns a tiny set of mixing coefficients (numbers that tell it how much Red, Blue, and Yellow to use).
- It does not change the primary colors themselves (the foundation) drastically; it just tweaks how they are combined.
Step 3: Merging. If the new skill requires a slightly different "color" that the current sketchbook doesn't have, the system gently expands the sketchbook to include this new direction, then re-calculates how to mix all the old skills with this new one.

3. Why It's a Game Changer

One Model, Infinite Skills: Instead of 100 separate sticky notes, you have one single set of instructions (the sketchbook) that can handle all 100 skills.
No Forgetting: Because the system constantly re-mixes the "colors" to include new information, it doesn't erase the old skills. In fact, the paper claims it sometimes helps the old skills get better (backward transfer) because the new mixing instructions are more efficient.
Massive Savings:
- Space: The paper claims 100 times fewer parameters (the "ink" used to write the instructions) and 281 times less memory than traditional methods.
- Real-world impact: A single "Share" model can replace hundreds of task-specific adapters. It's like replacing a library of 500 books with a single, smart tablet that can instantly generate any of those 500 stories.

4. What They Tested

The authors proved this works on a wide variety of "languages" the AI speaks:

Reading Comprehension: Understanding different types of text questions.
Image Classification: Recognizing different types of flowers, food, and objects.
3D Pose Estimation: Figuring out how 3D objects are rotated, even when they are partially hidden (occluded).
Art Generation: Learning different art styles (like "Soviet Propaganda" or "Bob Ross") for text-to-image generation.
Scaling: They even tested it with 50 different "LoRA" adapters arriving one by one, showing the system could merge them all into one efficient model.

The Bottom Line

Share is like teaching a master chef to cook 1,000 different dishes not by giving them 1,000 different recipe books, but by teaching them the fundamental chemistry of cooking and how to mix a few core ingredients in new ways. It saves space, prevents the chef from forgetting how to make the first dish when they learn the thousandth, and allows the chef to keep learning forever without needing a bigger kitchen.

Technical Summary: Shared LoRA Subspaces for Almost Strict Continual Learning

Problem Statement

Adapting large pretrained models (e.g., LLMs, VLMs, Diffusion models) to new tasks continually faces two primary challenges: catastrophic forgetting and the high computational/memory cost of retraining. While parameter-efficient tuning methods like Low-Rank Adaptation (LoRA) reduce the number of trainable parameters, standard LoRA implementations typically require separate adapters for each task. This approach hinders cross-task knowledge sharing, fails to support Strict Continual Learning (learning without data replay, additional models, or model size growth), and leads to inefficient resource usage as the number of tasks scales. Existing methods often rely on orthogonal subspaces or mixture-of-experts systems that still violate strict continual learning constraints by requiring multiple model instances or growing parameter sets.

Methodology: Share

The authors propose Share, a novel framework for Parameter-Efficient Continual Finetuning (PaCT) that learns and dynamically updates a single, shared low-rank subspace to accommodate multiple tasks and modalities.

Core Hypothesis

Share is based on the hypothesis that LoRA adapters for similar tasks and modalities share a common low-rank subspace. By identifying the principal basis vectors of this subspace, new adapters can be expressed as linear combinations of these bases, eliminating the need for separate, task-specific adapters.

Algorithmic Workflow

The method operates in three phases for each layer of the pretrained model:

Initialization:
- The foundational subspace (principal basis vectors) is initialized using $t \ge 1$ existing LoRA adapters.
- LoRA matrices ( $A$ and $B$ ) from previous tasks are stacked, centered, and subjected to Singular Value Decomposition (SVD).
- The top $k$ principal basis vectors ( $\alpha, \beta$ ) are selected to span the reusable shared subspace.
- Only randomly initialized task-specific coefficients ( $\epsilon$ ) are trained, while the basis vectors remain frozen. This initialization is data- and gradient-free.
Continual Adaptation:
- Upon receiving new data or a new LoRA adapter, the system temporarily expands the subspace by introducing $\phi$ temporary basis vectors and their coefficients.
- These temporary components are optimized to capture new information while preserving the foundational subspace.
Merging & Fine-tuning:
- The temporary basis vectors are merged with the existing foundational subspace.
- New shared basis vectors are computed via SVD of the stacked reconstructed adapters.
- Task-specific coefficients are analytically recalculated using matrix projection and the Moore-Penrose pseudoinverse to minimize reconstruction error.
- This process allows for the compression of hundreds of adapters into a single set of factors without requiring data replay.

Theoretical Analysis

The paper provides theoretical bounds on the reconstruction error and empirical risk minimization. It demonstrates that if a new task lies largely within the shared principal subspace, the error bound for the Share approximation is tighter than that of a full-rank adaptation. The method approximates the "Universal Weight Subspace" dynamically, rather than extracting it beforehand.

Key Contributions

Replay-Free Continual Learning: Introduces a method for large pretrained models that learns continually without data replay, additional models, or parameter growth, approaching the conditions of "Strict Continual Learning."
Hybrid Stream Integration: Enables continual learning from hybrid streams of both raw data and pre-trained LoRA adapters, seamlessly merging them into a single model.
Extreme Efficiency: Achieves up to 100× reduction in trainable parameters and 281× memory savings compared to traditional LoRA methods.
Scalable Deployment: A single set of Share principal factors can replace hundreds of task-specific LoRA adapters, facilitating asynchronous and scalable model serving.
Broad Applicability: Validated across diverse modalities including image classification, 3D object pose estimation, natural language understanding, and text-to-image generation.

Experimental Results

The authors evaluated Share across multiple benchmarks, comparing it against non-continual LoRA (separate adapters), Joint LoRA (upper bound), and other continual learning baselines.

Natural Language Understanding (GLUE Benchmark):
- Share achieved an average performance of 83.44% with only 0.012M parameters (0.29 MB).
- This matches the performance of non-continual LoRA (83.43%) which required 7.2M parameters (81.6 MB) across six adapters.
- The method demonstrated backward knowledge transfer, where learning subsequent tasks improved performance on earlier tasks (e.g., CoLA scores improved from 56.00 to 59.81).
Image Classification (CIFAR-100, Food-101, etc.):
- On CIFAR-100, Share achieved 94.20% accuracy with 0.10M parameters, matching the upper bound and outperforming the next best method (DAP) which required nearly twice the parameters.
- Share consistently demonstrated the lowest forgetting rates among all tested methods.
3D Object Pose Estimation:
- Using Pascal3D+, Share operated with only 1M parameters (vs. 25M for baselines) under strictly replay-free conditions.
- It outperformed replay-based methods (e.g., iNeMO) across all occlusion levels, achieving 81.80% accuracy on the P3D task.
Text-to-Image Generation:
- Applied to the Flux model, Share managed four continual art-style tasks using a single set of factors.
- Qualitative results showed performance comparable to individual LoRAs, with a 20× reduction in model size for the first four tasks.
Large-Scale Asynchronous Learning:
- In an experiment involving 500 LoRAs (with 50 arriving incrementally), Share maintained 90-99% of original performance on in-distribution tasks.
- On out-of-distribution (OOD) tasks, Share achieved an average Rouge-L score of 55.89, significantly outperforming non-continual model merging methods (TIES) and approaching individual LoRA performance (73.75).

Significance and Claims

The paper claims that Share is among the earliest works to present a viable solution for Parameter-Efficient Continual Finetuning applicable to diverse and complex models. Its significance lies in:

Democratizing Access: By drastically reducing memory and compute requirements, it allows ordinary researchers and users with limited resources to continually fine-tune large-scale AI systems.
Environmental Impact: The reduction in resource use contributes to a lower environmental footprint for model deployment.
Practical Scalability: It offers a practical solution for lifelong learning in large-scale systems, enabling the reuse of community resources (LoRA adapters) and supporting scalable, asynchronous learning paradigms without the computational burden of traditional model merging.

The authors note that while the current work focuses on fine-tuning, future work aims to extend Share to support learning from scratch.