💻 computer science

LDEPrompt: Layer-importance guided Dual Expandable Prompt Pool for Pre-trained Model-based Class-Incremental Learning

LDEPrompt introduces a layer-importance guided dual expandable prompt pool that overcomes limitations of existing methods by enabling adaptive layer selection and dynamic prompt pool expansion, achieving state-of-the-art performance in class-incremental learning.

Original authors: Linjie Li, Zhenyu Wu, Huiyu Xiao, Yang Ji

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Linjie Li, Zhenyu Wu, Huiyu Xiao, Yang Ji

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef who has spent years perfecting a massive, world-famous cookbook (this is your Pre-trained Model). You know how to cook everything from French pastries to Japanese sushi. Now, you want to learn new recipes for dishes from countries you've never visited, but you have a strict rule: you cannot rewrite the old pages of your cookbook, and you don't have space to keep every single old recipe book on your shelf.

This is the challenge of Class-Incremental Learning: learning new things without forgetting the old things, and without needing to store all the old data.

Here is how the paper's solution, LDEPrompt, solves this problem using a clever kitchen analogy.

1. The Problem with Old Methods

Previous attempts to solve this were like giving the chef a small, fixed-size sticky note pad.

The Issue: The chef had to write new recipes on these sticky notes. But the pad only had 5 spots. When a new recipe came in, the chef had to erase an old one to make room.
The Result: The chef forgot the old recipes (Catastrophic Forgetting). Also, the chef had to guess where to stick the note (e.g., near the "Appetizers" section or the "Desserts" section) without really knowing which spot was best.

2. The LDEPrompt Solution: A Smart, Growing Kitchen

The authors propose a new system called LDEPrompt. Think of it as a magical kitchen with two special features: Smart Placement and Dual Storage.

Feature A: The "Layer-Importance" Radar (Smart Placement)

Before the chef starts cooking a new dish, they don't just guess where to put the notes. They run a quick "scout" through their existing cookbook.

How it works: The system checks every chapter of the cookbook to see which one is most excited or "informative" about the new dish.
The Analogy: If the new dish is a spicy curry, the system realizes the "Spices" chapter is the most important place to add a note, rather than the "Baking" chapter.
The Benefit: Instead of cluttering the whole book with notes, the chef only places the new instructions in the specific chapters that actually need them. This saves space and keeps the instructions clear.

Feature B: The "Dual Pool" System (Smart Storage)

This is the magic trick that stops forgetting. The chef uses two different types of storage:

The Global Vault (The Frozen Library):
- This is a secure, glass-encased library where all the old, perfect recipes are stored.
- Rule: Once a recipe is put here, it is frozen. You can read it, but you cannot change it. This ensures the chef never forgets the old skills.
- Expansion: When the chef learns a new recipe, they don't overwrite the old ones. They add the new recipe to this library, making the library grow bigger over time.
The Training Counter (The Active Workspace):
- This is a small, temporary counter where the chef practices the current new dish.
- The Trick: Before practicing, the chef looks into the Global Vault and grabs the most similar old recipes (using a "similarity match").
- The Process: The chef mixes these old recipes with some new, blank notes on the counter to learn the new dish. Once the lesson is done, the new notes are moved to the Global Vault, and the counter is cleared for the next day.

3. Why This is a Game-Changer

No Overwriting: Because the old recipes are in the "Frozen Library," the new learning on the "Counter" never erases the past.
Dynamic Growth: The library isn't a fixed size. It grows as you learn more, so you never run out of space.
Smart Selection: By only putting notes in the "important chapters" (layers), the system is more efficient and doesn't get confused by too many notes in the wrong places.

The Result

The paper tested this "Smart Kitchen" on three famous cooking challenges (datasets: CIFAR100, CUB, and VTAB).

The Outcome: The chef using LDEPrompt cooked better than any other chef using the old "sticky note" methods.
The Score: They improved accuracy by significant margins (up to 3% better in some cases), proving that you can learn new things forever without forgetting the old ones.

In short: LDEPrompt is like giving a student a magical, expanding notebook where they only write in the right chapters, and they have a separate, unchangeable vault for their past knowledge, ensuring they never lose what they've already learned.

1. Problem Statement

The paper addresses Class-Incremental Learning (CIL) within the paradigm of Pre-trained Models (PTMs). The core challenge is to enable a model to learn new classes sequentially without access to old data (exemplar-free) while avoiding catastrophic forgetting (the loss of previously learned knowledge).

Existing prompt-based CIL methods suffer from three critical limitations:

Prompt Overwriting: Prompts selected repeatedly across different tasks are prone to being overwritten during training, leading to forgetting.
Fixed Capacity: Existing prompt pools have a fixed size, limiting the model's ability to accommodate new tasks as the number of classes grows.
Backbone Dependence: Prompt selection relies solely on embeddings from a frozen pre-trained backbone, which lacks task-specific information and creates a strong dependency on the backbone's initial representations.

2. Methodology: LDEPrompt

The authors propose LDEPrompt, a framework featuring two main innovations: Layer-importance guided selection and a Dual Expandable Prompt Pool.

A. Layer-Importance Evaluation

Before training a new task, the model performs a forward pass on the current task data through the frozen backbone to evaluate the Information Gain (IG) of each layer.

Calculation: $IG(l) = I(x; h_l) - I(x; h_{l-1})$ , where $I$ is mutual information between input $x$ and layer representations $h$ .
Selection: Layers are normalized using a softmax function. Only layers with importance scores above the mean are selected to host prompt pools. This avoids parameter redundancy and aligns prompt insertion with task-specific characteristics.

B. Dual Expandable Prompt Pool Architecture

To solve the issues of overwriting and fixed capacity, LDEPrompt introduces a dual-pool system:

Global Prompt Pool ( $P_g$ ):
- Stores prompts from all past tasks.
- Remains frozen during new task training to preserve historical knowledge.
- Acts as a reservoir for knowledge retention.
Training Prompt Pool ( $P_c$ ):
- Has a fixed capacity ( $S$ ) for the current task.
- Contains task-specific prompts that are trainable.
- Construction:
  - Retrieval: For the current task, the model retrieves the top- $s$ most similar prompts from $P_g$ using cosine similarity between the input [CLS] token and stored keys.
  - Initialization: The remaining $S-s$ slots are initialized randomly.
- Expansion: After training, the newly learned prompts in $P_c$ are frozen and merged into $P_g$ , dynamically expanding the global pool's capacity for future tasks.

C. Optimization

The model uses standard cross-entropy loss to optimize the trainable prompts in $P_c$ and the classifier head. The backbone remains frozen throughout the process.

3. Key Contributions

Adaptive Layer Selection: Unlike methods that insert prompts into all layers or fixed layers, LDEPrompt dynamically selects the most informative layers based on information gain, improving efficiency and performance.
Dual-Pool Mechanism: The separation of a frozen global pool and a trainable task-specific pool prevents catastrophic overwriting of old knowledge while allowing the prompt space to grow indefinitely.
Dynamic Expansion: The framework supports an unbounded number of tasks by expanding the global pool, addressing the limitation of fixed-size prompt pools in previous works.
Reduced Backbone Dependence: By integrating task-specific information into the prompt selection and training process, the method reduces reliance on the static representations of the frozen backbone.

4. Experimental Results

The authors evaluated LDEPrompt on three widely used benchmarks: CIFAR100, CUB, and VTAB, under various incremental settings (e.g., B0-Inc10, B0-Inc20).

Performance: LDEPrompt achieved State-of-the-Art (SOTA) results across all datasets.
- CIFAR100: Outperformed the second-best method by 0.40% (Avg) and 0.50% (Last).
- CUB: Surpassed the runner-up by 2.11% (Avg) and 2.91% (Last).
- VTAB: Improved by 3.23% (Avg) and 1.37% (Last).
Ablation Studies:
- Removing the dual-pool design or the freezing mechanism significantly degraded performance, confirming the necessity of both components.
- Inserting prompts into all layers (without layer importance selection) resulted in lower performance due to parameter redundancy, validating the efficiency of the layer-selection strategy.

5. Significance

LDEPrompt represents a significant advancement in Continual Learning for pre-trained vision models.

Scalability: It provides a viable solution for lifelong learning scenarios where the number of classes is unbounded, overcoming the capacity constraints of previous prompt-based methods.
Efficiency: By selecting only the most informative layers, it optimizes the trade-off between model capacity and performance.
Knowledge Preservation: The dual-pool architecture offers a robust mechanism to balance the plasticity (learning new tasks) and stability (retaining old tasks) required for effective incremental learning.

The code is publicly available, facilitating further research in prompt-based continual learning.