🤖 machine learning

Memory-Efficient Continual Learning with CLIP Models

This paper proposes a memory-efficient, distributionally robust continual learning method that dynamically reweights losses per class to enable CLIP models to adapt to new tasks while minimizing catastrophic forgetting, even with minimal memory buffers.

Original authors: Ryan King, Gang Li, Bobak Mortazavi, Tianbao Yang

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ryan King, Gang Li, Bobak Mortazavi, Tianbao Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart student named CLIP. This student is amazing at connecting pictures to words. If you show them a photo of a cat and the word "cat," they instantly understand the link. They've studied a massive library of books and photos, so they know a lot about the world.

However, there's a problem. If you try to teach this student something new—like how to recognize a specific type of rare bird—they might accidentally forget everything they knew about regular cats. This is called "catastrophic forgetting." It's like trying to write a new chapter in a book, but the ink is so wet it smears out the previous pages.

The paper by Ryan King and his team at Texas A&M University asks: How do we teach this student new things without erasing the old stuff, especially when we don't have a lot of space to keep old notes?

Here is how they solved it, using two main ideas:

1. The Problem: The "Tiny Backpack"

Usually, to stop a student from forgetting, teachers give them a "memory buffer." This is like a backpack where the student keeps a few examples of everything they've learned before (old photos of cats, dogs, cars, etc.). When learning something new, the teacher pulls out the old photos to remind the student: "Hey, don't forget this!"

But in the real world, we often can't carry a huge backpack. Maybe we only have room for a few photos. When the backpack is small, the student gets confused. They focus too much on the new thing and drop the old stuff. The paper found that standard methods fail when this "backpack" is too small.

2. The Solution: Two New Teaching Strategies

The authors proposed two ways to fix this, both based on how the student learns.

Strategy A: The "Global Group Hug" (Global Contrastive Loss)

Imagine the student is learning to sort photos. Instead of just looking at one photo at a time, this method asks the student to look at the whole group of photos they have (the new ones and the few old ones in the backpack) all at once.

How it works: The student learns to pull similar things together (like all cats) and push different things apart (cats vs. dogs).
The Trick: Because the student is looking at the whole group, they don't get confused by just one new photo. They remember the "shape" of the old knowledge.
Best for: When you have a medium-to-large backpack. It works great when you have enough old examples to keep the group hug balanced.

Strategy B: The "Tough Coach" (Distributionally Robust Optimization)

This is the paper's big innovation for when the backpack is very small.

Imagine the student is taking a test. Some questions are easy (they know the answer), and some are hard (they keep getting them wrong). A normal teacher might just move on after the student gets a few right. But this "Tough Coach" method says: "Stop! We need to focus on the hard questions!"

How it works: The system automatically detects which classes (like specific types of birds) are giving the student the most trouble or are under-represented in the tiny backpack. It then gives those difficult examples extra weight or importance during training.
The Metaphor: It's like a coach who notices a player is struggling with left-handed throws. Instead of practicing everything equally, the coach spends 80% of the time on left-handed throws until the player gets better.
Best for: When the backpack is tiny. It ensures the student doesn't ignore the old, forgotten details just because there aren't many examples left.

3. The Results: What Happened?

The team tested these methods on two types of challenges:

Learning new categories: Like going from knowing 100 types of animals to learning 100 more.
Learning new environments: Like learning to recognize the same animals, but now they are in different styles of photos (cartoons, sketches, real life).

The findings were:

Standard methods (like the ones used before) crashed when the memory backpack was small. They forgot the old stuff quickly.
The "Global Group Hug" worked very well when there was enough space to keep a decent number of old examples.
The "Tough Coach" (DRO) was the superstar for small backpacks. Even with very few old examples, it kept the student's memory sharp and prevented them from forgetting the past while learning the future.

The Bottom Line

The paper shows that by changing how the model pays attention to its old memories—either by looking at the whole group or by focusing extra hard on the difficult parts—we can teach AI to learn continuously without losing its past. It's like teaching a student to be a lifelong learner who never forgets their first lessons, even when they are busy studying for their final exams.

Technical Summary: Memory-Efficient Continual Learning with CLIP Models

Problem Statement
Contrastive Language-Image Pretraining (CLIP) models demonstrate superior capabilities in understanding image-text relationships but face significant challenges when adapting to new data streams without forgetting prior knowledge (catastrophic forgetting). While continual learning (CL) is essential for dynamic environments, applying it to CLIP is under-explored. Existing approaches often rely on memory buffers containing past task data to rehearse old information. However, CLIP's contrastive loss function suffers when these memory buffers are small, leading to performance degradation on previous tasks. The core challenge addressed is how to efficiently leverage limited memory buffers in CLIP-based continual learning to balance performance on new tasks while retaining knowledge of old tasks.

Methodology
The authors propose two distinct approaches to extend CLIP for class-incremental and domain-incremental learning settings, utilizing a bimodal contrastive learning objective that jointly encodes images and text labels.

Global Contrastive Loss (GCL):
- Mechanism: This approach treats old and new data equally during fine-tuning. It defines a contrastive objective over the union of the current task data ( $D_t$ ) and a memory bank ( $M_t$ ) of past samples.
- Optimization: To handle the computational cost of summing over the entire dataset for contrastive terms, the method employs moving average estimators ( $u^I$ and $u^T$ ) for the gradient components. This allows the model to maintain a "global" view of the data distribution across tasks, enabling information from prior tasks to inform the current gradient updates without requiring full re-computation of the dataset.
- Goal: To enhance resistance to catastrophic forgetting by jointly embedding labels and images, leveraging CLIP's pre-trained capabilities.
Group Distributionally Robust Optimization (GDRO):
- Motivation: In scenarios with fixed memory sizes, the number of examples per class decreases as new tasks are added, creating an imbalance between previous and current task distributions. Standard GCL struggles with this imbalance.
- Mechanism: This method introduces a group DRO objective that dynamically reweights class losses. It assigns greater weight to classes with higher losses (harder classes) during training.
- Optimization: The objective is formulated as a minimax problem ( $\min_w \max_{p \in \Delta} \sum p_k h_k$ ), which is equivalently expressed using a log-sum-exp formulation. To address the challenge of nested functions in gradient estimation, the authors apply Stochastic Compositional Optimization. They maintain moving average estimators for both the loss terms and the contrastive components, allowing for efficient gradient computation using mini-batches.

Key Contributions

Bimodal Contrastive Framework: The paper extends CLIP to continual learning by proposing a bimodal contrastive objective that jointly embeds image and text labels, improving adaptability and retention compared to standard linear classifiers.
Memory-Efficient Algorithms: The authors introduce GCL and GDRO, specifically designed to function effectively even with minimal memory usage.
Dynamic Reweighting: The GDRO method specifically addresses the class imbalance inherent in fixed-size memory buffers by dynamically reweighting losses, ensuring robust performance when memory is constrained.

Experimental Results
The methods were evaluated on class-incremental datasets (CIFAR-100, ImageNet1K) and a domain-incremental dataset (DomainNet), using a pretrained CLIP ViT-B/16 encoder.

ImageNet1K (Class Incremental): The proposed methods significantly outperformed baselines (including EWC, DER, iCaRL, Co2L, and FOSTER) across all memory sizes. Notably, unimodal contrastive approaches like Co2L experienced sharp performance drops as memory size decreased, whereas the proposed methods maintained robustness.
CIFAR-100 (Class Incremental): As memory size decreased, the GDRO method maintained performance comparable to zero-shot evaluation, indicating solid retention despite some forgetting. While the contrastive method (GCL) performed better with larger memory, its performance degraded significantly without memory. In contrast, GDRO remained stable under strict memory constraints.
DomainNet (Domain Incremental): Both methods outperformed baseline zero-shot results. While contrastive methods like CaSSLe struggled to retain knowledge without a memory bank, the GDRO objective demonstrated superior retention and adaptability compared to GCL after completing all tasks.

Significance and Claims
The paper claims that embedding both input data and labels within a bimodal contrastive framework reduces catastrophic forgetting more effectively than traditional linear classifier approaches. The significance of the work lies in demonstrating that:

Multimodal Learning: Leveraging the joint embedding of images and text inherent to CLIP is beneficial for continual learning in dynamic environments.
Adaptive Weighting: Dynamic reweighting of classes (via GDRO) is particularly effective for retaining knowledge when memory resources are limited, addressing a key bottleneck in memory-based continual learning.
Efficiency: The proposed methods allow CLIP models to adapt quickly to new tasks while minimizing catastrophic forgetting, even with minimal memory usage, making them suitable for resource-constrained real-world data streams.