🤖 machine learning

Continual Distillation Learning for Rehearsal-Free Class-Incremental Learning via Decoupled Prompting

This paper introduces Decoupled Continual Distillation Learning (D-CDL), a novel framework for rehearsal-free class-incremental learning that improves knowledge transfer from larger to smaller Vision Transformers by explicitly decoupling task-specific adaptation from distillation through persistent, global Knowledge-Distillation prompts.

Original authors: Qifan Zhang, Yunhui Guo, Yu Xiang

Published 2026-08-14

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Qifan Zhang, Yunhui Guo, Yu Xiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize animals. You start with cats, then dogs, then birds. The tricky part is that as the robot learns about birds, it tends to forget what a cat looks like. This is a famous headache in the world of artificial intelligence called "catastrophic forgetting." To fix this, scientists have developed a clever trick called "prompt-based learning." Instead of retraining the robot's entire brain (which is huge and expensive), they give it a tiny, sticky note—a "prompt"—that tells it how to think about the current task. This way, the robot keeps its original brain frozen and safe, only changing the sticky notes to adapt to new animals.

However, there's a catch: bigger brains are generally better at recognizing things. A giant robot brain (a large model) is much smarter than a small, efficient one. But giant brains are slow and use too much energy for everyday gadgets. The dream is to take the knowledge from the giant brain and squeeze it into the small, fast brain so the small one becomes just as smart without the heavy weight. This process is called "knowledge distillation." Usually, you can do this easily if you have a big pile of old photos to study from. But in the real world of continual learning, you don't get to look at old photos; you only get the new animal pictures as they arrive, one by one. This creates a unique puzzle: How do you teach a small robot to learn from a big one, step-by-step, without the small robot forgetting the big one's lessons every time a new animal shows up?

This is the exact puzzle tackled in a new paper by researchers at the University of Texas at Dallas. They discovered that the standard ways of trying to squeeze knowledge from a big brain to a small one actually fail in this specific "step-by-step" scenario. They found that the small robot keeps forgetting the teacher's lessons because the "sticky notes" it uses to learn are constantly being swapped out for new ones. To solve this, they invented a new method called Knowledge Distillation based on Prompts (KDP). Instead of relying on the swapping sticky notes, they added a special, permanent "master note" that stays with the robot forever, acting as a dedicated bridge to the teacher's knowledge. Their experiments show that this new method helps the small robot learn much better and forget much less than previous attempts, effectively letting it run as smart as the giant brain but with the speed of a small one.

The Problem: The "Swapping Sticky Note" Glitch

To understand why this is hard, let's look at how these "prompt-based" robots work. Imagine the robot has a library of sticky notes (prompts). When it sees a cat, it picks a "cat note." When it sees a dog, it picks a "dog note." This is great for learning new things without forgetting old ones because the robot just swaps the note.

But here is where the "distillation" (teaching from a big brain) breaks down. The researchers found that when they tried to teach the small robot using the big one, the small robot would pick a "cat note" to learn from the big teacher. Then, when a new task (like birds) arrived, it would throw away the "cat note" and pick a "bird note." The problem? The lessons the big teacher taught about cats were stuck inside that "cat note." Once the note was swapped out, the small robot forgot everything the teacher had taught it about cats. The researchers call this "distillation information forgetting." It's like trying to learn a language from a teacher, but every time you switch to a new chapter, you throw away the notebook where the teacher wrote down your homework.

The researchers tried the usual tricks, like looking at the teacher's final answers (logits) or the teacher's intermediate thoughts (features), but these didn't work well because the "note-swapping" mechanism kept wiping the slate clean. They even tested unfreezing part of the robot's brain to let it learn permanently, but that caused the robot to forget its old tasks even faster, defeating the whole purpose.

The Solution: The "Forever Bridge"

To fix this, the team introduced a new kind of sticky note called a KD Prompt (Knowledge Distillation Prompt). Unlike the regular notes that get swapped out for every new animal, these KD Prompts are special. They are globally accessible, meaning they stay with the robot no matter what task it is doing.

Think of it this way:

Regular Prompts: These are like temporary flashcards. You use a "Cat Flashcard" for cats, then toss it and grab a "Dog Flashcard" for dogs.
KD Prompts: These are like a permanent, unerasable whiteboard attached to the robot's wall. No matter what animal you are looking at, the robot can always write the teacher's lessons on this whiteboard.

By inserting these permanent KD Prompts into the robot's brain, the small model has a dedicated space to store the big teacher's wisdom that never gets thrown away. They also added a special "KD Classifier" (a second brain for grading) that specifically checks if the robot is copying the teacher correctly, separate from the robot's main job of recognizing animals.

What They Found

The researchers tested this idea on two famous sets of image data: CIFAR-100 (100 types of small images) and ImageNet-R (100 types of artistic renditions of objects). They set up a scenario where the robot had to learn 10 different tasks, one after another, with 10 classes in each.

They compared their new KDP method against older methods like standard knowledge distillation and feature distillation. The results were clear:

Old Methods: When using standard distillation, the small robot's accuracy only went up a tiny bit, and it actually forgot previous tasks more than it did without any help. For example, on the ImageNet-R dataset, a standard method called "KD" got an average accuracy of 69.91% with a forgetting rate of 7.64%.
The New Method (KDP): With their new "Forever Bridge" approach, the small robot became significantly smarter and forgot much less. Using the same dataset, the KDP method boosted the average accuracy to 71.92% and dropped the forgetting rate to just 5.61%.

When they tested this with an even bigger teacher (a "ViT-Large" model) teaching a medium-sized student ("ViT-Base"), the gap widened even more. The KDP method achieved an average accuracy of 78.62% with a forgetting rate of 3.46%, outperforming every other method they tried.

Why It Matters

The paper suggests that this isn't just a small tweak; it's a fundamental fix for a specific type of learning problem. By separating the "learning new tasks" notes from the "learning from the teacher" notes, they solved the forgetting issue that plagued previous attempts.

The researchers note that this does come with a cost: you have to train both the big teacher and the small student at the same time, which takes more time and computer memory. However, once the training is done, the small student model is ready to go. It can run on devices that need to be fast and efficient, yet it carries the smarts of a much larger model.

In short, the paper shows that if you want a small, fast AI to learn continuously without forgetting, you can't just copy-paste the teacher's brain. You have to give the student a permanent, dedicated space to keep the teacher's lessons safe, no matter what new things it encounters next.

Technical Summary: Continual Distillation Learning (CDL)

1. Problem Definition: Continual Distillation Learning (CDL)

The paper introduces a new research problem termed Continual Distillation Learning (CDL). This problem addresses the challenge of improving the inference efficiency of prompt-based Continual Learning (CL) models without sacrificing performance.

Context: In prompt-based CL (e.g., L2P, DualPrompt, CODA-Prompt), a pre-trained Vision Transformer (ViT) backbone is kept frozen, and learning is shifted to optimizing a pool of "prompts." Empirical observations show that larger ViT backbones (e.g., ViT-Large) yield better performance than smaller ones (e.g., ViT-Small) in these CL settings.
The Challenge: While large models perform better, they incur high computational costs during inference. The goal of CDL is to distill knowledge from a large "Teacher" ViT to a smaller "Student" ViT within a rehearsal-free, task-incremental learning setting.
Distinction from Traditional KD: Unlike traditional Knowledge Distillation (KD) which uses an offline dataset to train a teacher and then distill to a student, CDL operates sequentially. The teacher model is updated task-by-task, and the student must learn from the teacher using only the data of the current task, without access to past data.

2. Key Observation: Distillation Information Forgetting

The authors empirically found that existing KD methods (Logit Distillation and Feature Distillation) fail to effectively improve student models in the CDL setup. They identify the root cause as Distillation Information Forgetting:

Mechanism: In prompt-based CL, different tasks select different subsets of prompts from a prompt pool via a query-key mechanism.
The Failure: Knowledge distilled from the teacher is embedded into the specific prompts selected for a given task. When a new task arrives, the student selects a different set of prompts. Since the backbone is frozen, the knowledge transferred in the previous task's prompts is effectively discarded for the new task. The student must re-learn the teacher's knowledge from scratch for each new task, leading to poor cross-task knowledge transfer.
Failed Alternatives: The paper notes that unfreezing parts of the ViT backbone to allow updates is not a viable solution, as it leads to catastrophic forgetting of previous tasks, violating the core principle of prompt-based CL.

3. Methodology: Knowledge Distillation based on Prompts (KDP)

To address Distillation Information Forgetting, the authors propose Knowledge Distillation based on Prompts (KDP).

Core Components

KD Prompts:
- A new type of prompt is introduced specifically for knowledge distillation.
- Unlike standard CL prompts which are task-specific and selected via a query-key mechanism, KD Prompts are globally accessible and shared across all tasks.
- They do not rely on the selection mechanism; they are inserted into the frozen ViT backbone of the student model for every task.
- They are inserted using Prefix-Tuning, splitting the prompt into Key ( $P_K$ ) and Value ( $P_V$ ) components and appending them to the MSA layers of the ViT.
KD Token and Classifier:
- A dedicated Distillation Token (KD Token) is inserted at the end of the first layer of the ViT backbone.
- This token propagates through the network and connects to a separate KD Classifier.
- The KD Classifier is trained to minimize the divergence between the student's output and the teacher's output (soft labels) using a standard logit distillation loss (KL divergence with temperature $\tau$ ).
Training Objective:
The student model's total loss function ( $L_S$ ) combines three terms:
- Student Classification Loss: Standard cross-entropy loss for the current task using the student's main classifier.
- Knowledge Distillation Loss: The loss between the student's KD classifier and the teacher's classifier.
- Prompt Pool Loss: The loss associated with selecting/optimizing the original CL prompts (specific to the underlying CL method, e.g., L2P or CODA-Prompt).

4. Experimental Results

The authors evaluated KDP on CIFAR-100 and ImageNet-R datasets using a class-incremental setting (10 tasks, 10 classes each). They tested three prompt-based CL backbones: L2P, DualPrompt, and CODA-Prompt.

Teacher-Student Pairs: Experiments were conducted with ViT-Large $\to$ ViT-Base and ViT-Base $\to$ ViT-Small.
Performance:
- Existing KD methods (KD, DKD, FitNets, ReviewKD) showed negligible improvements or even increased forgetting rates compared to the baseline student.
- KDP consistently outperformed all baselines. For example, on ImageNet-R with CODA-Prompt (ViT-Base $\to$ ViT-Small), KDP achieved 71.92% average accuracy with 5.61% forgetting, compared to 67.44% and 8.52% for the baseline without distillation.
- The combination of CODA-Prompt and KDP achieved State-of-the-Art (SOTA) performance, approaching the accuracy of larger models while maintaining the efficiency of smaller ones.
Ablation Studies:
- KD Prompts vs. KD Classifier: Both components are necessary for optimal performance.
- Unfreezing Backbone: Unfreezing the last layer of the ViT resulted in severe forgetting (accuracy dropped to ~30%), confirming the necessity of keeping the backbone frozen.
- Prompt Position: Simply adding more trainable prompts to the standard CL prompt pool (without the global KD prompt mechanism) did not yield the same benefits, proving that the global, cross-task nature of KD prompts is the key factor.

5. Significance and Contributions

The paper claims the following contributions:

Problem Formulation: It introduces the CDL problem, defining the specific challenge of distilling knowledge from large to small models within the constraints of rehearsal-free, prompt-based continual learning.
Discovery of a Limitation: It is the first to identify and analyze Distillation Information Forgetting, explaining why standard KD methods fail in prompt-based CL due to the task-specific nature of prompt selection.
Novel Method (KDP): It proposes KDP, a method that utilizes globally shared prompts and a dedicated distillation token to facilitate continuous cross-task knowledge transfer without unfreezing the backbone.
Empirical Validation: The method demonstrates significant improvements in accuracy and reductions in forgetting across multiple prompt-based CL frameworks and datasets, establishing a new baseline for efficient continual learning.

The authors conclude that KDP effectively bridges the gap between the high performance of large models and the efficiency requirements of small models in continual learning scenarios.