🤖 machine learning

GLAM: Efficient Continual Learning at Scale via Grouped LoRA Adapter Merging

GLAM is an efficient continual learning framework that trains lightweight LoRA adapters with importance scalars for each task, then prunes, rescales, and merges them into grouped modules to achieve state-of-the-art accuracy on long task sequences while significantly reducing trainable parameters and training time.

Original authors: Irene Testa, Luigi Quarantiello, Eric Nuertey Coleman, Samrat Mukherjee, Julio Hurtado, Vincenzo Lomonaco

Published 2026-08-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Irene Testa, Luigi Quarantiello, Eric Nuertey Coleman, Samrat Mukherjee, Julio Hurtado, Vincenzo Lomonaco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant, super-smart robot to recognize everything in the world, from your pet cat to a rare bird, and then to a specific type of car. The robot starts with a massive, pre-loaded brain that already knows a lot. But here's the tricky part: every time you teach it something new, it tends to get a bit "forgetful," accidentally wiping out the memories of what it learned yesterday. This is a famous problem in computer science called "catastrophic forgetting." Usually, to fix this, you'd have to retrain the robot's entire brain from scratch every time, which takes forever and costs a fortune in electricity. Scientists have been trying to find a way to teach these robots new tricks without making them forget the old ones, hoping to build machines that can learn continuously, just like humans do, without needing a massive memory upgrade every single day.

Enter GLAM, a new method that acts like a clever, organized librarian for this robot's brain. Instead of stuffing every new lesson into a separate, bulky book that takes up endless shelf space, GLAM teaches the robot to write tiny, lightweight notes (called "adapters") for each new task. But here's the magic trick: instead of keeping thousands of these notes, GLAM groups similar notes together, prunes away the messy, unimportant scribbles, and merges them into a single, neat summary. It's like taking a stack of 50 different travel journals, reading through them, and condensing them into one perfect travel guide that holds all the best tips without being heavy.

The researchers found that this approach is incredibly efficient. When they tested GLAM on a sequence of up to 50 different tasks (like recognizing 50 different sets of images), it didn't just learn well; it learned fast. In fact, it was about three times faster to train than the best competing methods and used only about 20% of the "trainable brain power" (parameters) that others needed. While other methods struggled to keep up as the number of tasks grew, GLAM stayed sharp, achieving the highest accuracy on all the tests they ran. It proved that you can keep a robot learning new things over a long time without it forgetting the past or needing a supercomputer to do the math.

The Problem: The Robot's Short Memory

Think of a modern AI model as a giant, pre-trained brain that has already read the entire internet. It's amazing, but it has a flaw: if you try to teach it something new, like how to recognize a specific breed of dog, it often gets confused and forgets how to recognize the cat it knew before. This is the "forgetting" problem.

To stop this, scientists usually try to freeze the brain's main parts and only let a small, special section learn the new stuff. This is called LoRA (Low-Rank Adaptation). Imagine the brain is a giant library, and LoRA is like adding a small, sticky note to a specific shelf instead of rewriting the whole book. It's efficient, but if you have 50 different tasks, you end up with 50 different sticky notes. Managing 50 notes is messy, and if you try to read them all at once, it slows the robot down.

The GLAM Solution: The Grouping Game

GLAM (Grouped LoRA Adapter Merging) changes the game by organizing these sticky notes into teams. Here is how it works, step-by-step:

The New Lesson: When the robot learns a new task, GLAM creates a tiny, lightweight note (an adapter) just for that job. It also gives this note a "importance score" to say how much it matters.
The Grouping: Instead of keeping every note separate, GLAM randomly assigns the new note to one of a few "groups" (like putting a new book on one of five specific shelves).
The Cleanup (Pruning): Before the note is added to the group, GLAM does a quick cleanup. It looks at the note and throws away the weak, unimportant parts (the "noise"), keeping only the strong, clear information. This stops the notes from getting messy and interfering with each other.
The Merge: The new, cleaned-up note is then gently blended into the group's existing summary. The group summary gets updated to include the new info, but it stays as just one note.
The Final Result: At the end, all the group summaries are combined into a single, super-efficient module. The robot doesn't need to look at 50 notes; it just looks at one compact summary that holds everything it learned.

Why It's a Big Deal

The researchers tested this on three different image datasets: CIFAR-100 (100 types of objects), CUB-200 (200 types of birds), and ImageNet-R (200 types of artistic images). They made the robot learn in sequences of up to 50 tasks, which is much longer than the usual 10-task tests most scientists use.

The results were impressive:

Speed: GLAM trained about 3 times faster than the best competing method (CL-LoRA).
Size: It used only 16–19% of the trainable parameters that the best baseline used.
Accuracy: It got the highest scores on all three datasets. For example, on the CUB-200 bird dataset, it reached 70.96% accuracy, beating the next best method by a significant margin.
Forgetting: It forgot less than the others. While some methods forgot about 33% of what they learned, GLAM only forgot about 20% on that same dataset.

The paper suggests that this "grouping and merging" strategy is the key to making AI that can learn continuously without getting bogged down. By keeping the number of active notes low and cleaning them up as they go, GLAM allows the robot to learn a long sequence of tasks efficiently, without needing to remember every single detail separately. It's a step toward machines that can truly learn and grow over time, just like we do.

=== SUMMARY ===

Technical Summary: GLAM – Efficient Continual Learning at Scale via Grouped LoRA Adapter Merging

Problem Statement

Continual Learning (CL) aims to enable models to learn incrementally from sequences of tasks while retaining previously acquired knowledge, addressing the persistent issue of catastrophic forgetting. While large pre-trained Foundation Models offer rich representations that partially mitigate forgetting, they struggle to adapt efficiently to evolving data distributions without incurring prohibitive computational costs.

Existing Parameter-Efficient Fine-Tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), are typically designed for static, single-task scenarios. Applying LoRA to CL presents three key challenges:

Catastrophic Forgetting: Preventing the loss of knowledge from previous tasks.
Knowledge Transfer: Enabling effective transfer between related tasks.
Scalability: Efficiently allocating parameters as the number of tasks grows.

Current LoRA-based CL approaches often store separate adapters for each task (requiring task identifiers at inference) or employ complex isolation strategies that limit knowledge transfer. Alternatively, post-hoc merging methods consolidate adapters only after training, limiting transfer during the learning phase and often relying on manually tuned, suboptimal coefficients. Furthermore, many existing methods treat adapters independently, failing to support shared learning or adaptive reuse, and incur high training costs as task sequences lengthen.

Methodology: GLAM

The authors propose GLAM (Grouped LoRA Adapter Merging), a framework designed for class-incremental continual learning that dynamically groups and merges adapters as tasks unfold, without requiring access to previous task samples. The method operates through a three-stage pipeline:

1. Task-Specific LoRA Training

For each new task $T_i$ , GLAM trains a lightweight, task-specific LoRA adapter $\Delta W_i = B_i A_i$ alongside an associated importance scalar $\alpha_i$ .

Optimization: The adapter $\Delta W_i$ and its scalar $\alpha_i$ are optimized on the current dataset $D_i$ .
Freezing: The base pre-trained weights $W_0$ and previously learned group adapters $\{\Delta W_{G_j}\}$ are kept frozen.
Dynamic Scaling: The scaling factors of existing group adapters $\{\alpha_{G_j}\}$ are updated during the training of the new task to maintain balanced relative importance.
Classifier Masking: Logits for classes outside the current task are masked to optimize only the relevant classifier parameters.

2. Adapter Grouping

To prevent unbounded parameter growth, GLAM employs a dynamic grouping strategy rather than retaining a separate module for every task.

Group Association: Upon completing training for task $T_i$ , the new adapter is assigned to a group $G_j$ via a random sampling strategy from a set of available group slots (up to a maximum $G_{max}$ ). If the slot is empty, a new group is initialized; otherwise, the adapter joins an existing group.
Importance Aggregation: Each group maintains a single scaling factor $\alpha_{G_j}$ , updated as a running average of the importance factors of all adapters assigned to it.
Magnitude-Based Pruning: Before merging, the incoming adapter undergoes magnitude-based pruning. Only the top- $k\%$ weights (by absolute value) in matrices $B_i$ and $A_i$ are retained. This step reduces interference from weak or redundant components without the high computational cost of Fisher-based criteria.
Intra-Group Merging: The pruned adapter is linearly merged with the current group adapter $\Delta W_{G_j}$ using a weighted average based on the group size. This consolidates multiple task-specific updates into a single group-level representation.

3. Classifier Alignment and Final Concatenation

Classifier Alignment: To mitigate bias toward recently learned classes, a classifier-alignment stage is performed. Synthetic features are sampled from stored Gaussian statistics (mean and variance) of class features from all seen tasks. The final linear classifier is retrained on these synthetic features while the feature extractor remains frozen.
Final Concatenation: At inference, all group adapters are combined into a single unified module. This is achieved by scaling each group adapter by its importance factor $\alpha_{G_j}$ and concatenating them along the rank dimension. The resulting model has a single compact adapter $\Delta W_{final}$ , ensuring constant computational cost regardless of the number of tasks.

Key Contributions

Scalable CL Methodology: A novel approach for foundation models that leverages PEFT techniques, combining task-specific adaptation with a mechanism to jointly learn LoRA modules and their importance coefficients.
Progressive Merging Pipeline: A strategy that consolidates task-specific adapters into a bounded set of group adapters. This controls parameter growth and mitigates merging interference over long task sequences through sequential grouping and magnitude-based pruning.
Empirical Validation: Extensive experiments on diverse benchmarks (CIFAR-100, CUB-200, ImageNet-R) demonstrating state-of-the-art performance in dynamic CL scenarios with long task sequences (up to 50 tasks), while significantly reducing training time and trainable parameters compared to strong baselines.

Experimental Results

The authors evaluated GLAM on standard benchmarks using sequences of up to 50 tasks, a setting significantly more demanding than the typical 10-task protocols found in recent literature.

Accuracy and Forgetting: GLAM achieved the highest average accuracy across all three benchmarks. Compared to the strongest-performing baseline (which varies by dataset and metric, e.g., CL-LoRA on CIFAR-100, SEMA on ImageNet-R), GLAM achieved an average increase of 6% in accuracy and an 8% reduction in forgetting. For instance, on CIFAR-100, GLAM (84.73%) significantly outperformed CL-LoRA (77.56%), while on CUB-200, GLAM (70.96%) surpassed CL-LoRA (61.49%).
Efficiency: GLAM demonstrated superior computational efficiency. It achieved approximately a 3× training speedup over the strongest-performing baseline while utilizing only 16–19% of the trainable parameters used by that baseline (e.g., ~3.6M parameters for GLAM vs. ~19.2M for CL-LoRA on CIFAR-100).
Scalability: As the number of tasks increased from 10 to 50, GLAM maintained robust performance with minimal degradation, whereas competing methods exhibited noticeable accuracy drops.
Ablation Studies: Analyses confirmed the effectiveness of the sequential, two-phase merging strategy. Random grouping was found to be as effective as more complex clustering strategies, and magnitude-based pruning proved effective in reducing interference.

Significance and Claims

The paper positions GLAM as a significant step toward lifelong learning in foundation models. By addressing the scalability bottleneck of existing PEFT-based CL methods, GLAM enables the continuous addition of new knowledge without the linear explosion of parameters or computational cost.

The authors claim that GLAM offers a practical solution for real-world scenarios where models must adapt to long sequences of tasks under constrained time and resource budgets. The method successfully balances the stability-plasticity trade-off, preserving knowledge from prior tasks while efficiently integrating new information, all while maintaining a constant inference cost through the final adapter concatenation. The work suggests that dynamic grouping and merging, rather than static isolation or post-hoc consolidation, is a viable path for efficient, large-scale continual learning.