Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning

This paper proposes Semantic-Guided Dynamic Sparsification (SGDS), a novel class-incremental learning method that mitigates catastrophic forgetting and enhances plasticity by sculpting class-specific sparse activation subspaces to facilitate knowledge transfer among similar classes while preventing interference between dissimilar ones, thereby outperforming existing parameter-constrained approaches.

The Gist

Imagine you are a master chef who has spent years perfecting a classic French recipe book (this is your Pre-Trained Model). Now, you want to learn to cook dishes from entirely new cuisines—say, Thai and Mexican—without forgetting how to make that perfect French soufflé.

The problem? If you try to learn the new recipes by rewriting your old book, you might accidentally erase the French instructions. If you just keep the old book closed and write a tiny, separate note for the new recipes, you might run out of space or get confused about which note to use.

This is the challenge of Class-Incremental Learning (CIL): teaching an AI to learn new things without forgetting the old, all while being efficient.

The Old Way: The "Rigid Filing Cabinet"

Most current methods try to solve this by building a Rigid Filing Cabinet.

• They take your existing knowledge (the French book) and freeze it.
• For every new cuisine, they add a small, separate drawer (an Adapter).
• To make sure you don't mix up the Thai spices with the French herbs, they force these new drawers to be completely orthogonal (at a 90-degree angle) to everything else.

The Flaw: Imagine trying to fit a giant, flexible yoga mat into a tiny, rigid box. By forcing the new drawers to be perfectly rigid and separate, you limit how much the AI can actually learn and adapt. You are sacrificing plasticity (the ability to bend and learn) for stability (not forgetting).

The New Way: SGDS (The "Smart Traffic Controller")

The paper proposes a new method called SGDS (Semantic-Guided Dynamic Sparsification). Instead of forcing the drawers to be rigid, SGDS acts like a Smart Traffic Controller for the AI's thoughts (activations).

Here is how it works, using a simple analogy:

1. The "Semantic Strategy" (The GPS)

Before the AI learns a new dish, SGDS asks: "Is this new dish similar to something we already know?"

• Scenario A (Similar): If you are learning to make "Pad Thai" and you already know "Pad See Ew," SGDS says, "Great! Let's use the same kitchen counter space for both." It encourages the AI to share its mental space for similar things.
• Scenario B (Different): If you are learning to make "Sushi" (Japanese) and you already know "Tacos" (Mexican), SGDS says, "No overlap! These are too different. Let's build a completely new, empty room for Sushi."

2. "Semantic Exploration" (Drawing the Map)

SGDS guides the AI's thoughts into specific "lanes" or "rooms."

• If the tasks are similar, it aligns their paths so they walk together.
• If the tasks are different, it forces them into orthogonal subspaces. Think of this like building a new room that has no doors connecting to the old rooms. This ensures that when you think about Sushi, you don't accidentally knock over the Tacos.

3. "Activation Compaction" (The Vacuum Cleaner)

This is the secret sauce. Even if you have a new room for Sushi, you don't want it to be a giant, empty warehouse where thoughts get lost.

• SGDS uses a Vacuum Cleaner (Targeted Sparsification) to suck out all the unnecessary clutter.
• It forces the AI to use only the most important neurons (the core ingredients) for that specific task.
• By making the "Sushi room" very compact and sparse, it leaves a huge amount of empty space (a "Null Space") around it. This empty space is a Sanctuary where future tasks (like learning Italian) can be built without ever bumping into the Sushi or Tacos.

Why is this better?

• The Old Way (Rigid Constraints): Like trying to park cars in a garage where every car is bolted to the floor. It's stable, but you can't move anything, and you can't fit new cars easily.
• The New Way (SGDS): Like a dynamic parking lot with smart sensors. It groups similar cars together, but for unique cars, it clears out a specific, compact spot and leaves the rest of the lot wide open for future arrivals.

The Result

The researchers tested this on difficult image datasets (like recognizing animals or objects).

• Performance: SGDS beat the current best methods (State-of-the-Art) by a significant margin.
• Privacy: Because it doesn't need to store old images (exemplars) to remember them, it's perfect for privacy-sensitive situations like healthcare.
• Efficiency: It learns new things faster and forgets less, all while using fewer computing resources.

In a nutshell: Instead of locking the AI's brain into rigid boxes to prevent confusion, SGDS gently guides its thoughts into organized, compact, and separate "mental rooms," leaving plenty of room for the future.

Technical Summary

1. Problem Statement

Class-Incremental Learning (CIL) aims to enable models to learn new classes sequentially without forgetting previously learned ones (catastrophic forgetting).

• Current Paradigm: The state-of-the-art approach involves freezing a large Pre-trained Model (PTM) and using lightweight Parameter-Efficient Fine-Tuning (PEFT) modules, such as adapters.
• The Limitation: To prevent interference between tasks, existing methods often enforce rigid constraints on the adapter parameters (e.g., orthogonality).
• The Core Conflict: While parameter constraints reduce interference, they severely limit model plasticity. Since adapters are already low-rank, further constraining their parameters hinders the model's ability to learn new tasks effectively. The paper argues that interference stems not just from parameters, but from the interaction between parameters and neural activations.

2. Methodology: Semantic-Guided Dynamic Sparsification (SGDS)

SGDS proposes a paradigm shift: instead of constraining the parameter space, it proactively guides the activation space. It constructs sparse, orthogonal subspaces for different tasks by governing the orientation and rank of activation subspaces through a two-phase process.

A. Semantic Strategy Formulation

Before training, SGDS analyzes the semantic relationship between new classes and existing classes using prototypes (averaged embeddings from the frozen PTM).

• Similarity Metric: It computes cosine similarity between class prototypes.
• Decision Logic:
  • Knowledge Reuse: If a new class is highly similar to existing classes, it is assigned to reuse the existing activation subspace.
  • New Subspace Allocation: If a new class is dissimilar, it is assigned a new, non-overlapping activation subspace to prevent interference.

B. Two-Phase Training Process

SGDS operates on the input activations of the adapters within the PTM (specifically Vision Transformers).

1. Phase 1: Semantic Exploration (Orientation Control)

   • Goal: Control the orientation of the activation subspace.
   • Mechanism: Uses a probabilistic sampling mechanism to determine which neurons activate.
     • For Knowledge Reuse classes: Encourages overlap with existing subspaces to foster transfer.
     • For New Subspace classes: Enforces activation into the null space of previous tasks (orthogonal directions) to prevent interference.
   • Implementation: Uses historical counters to track activation usage and adjusts probabilities to steer activations away from previously occupied directions.

2. Phase 2: Activation Compaction (Rank Control)

   • Goal: Control the rank (size) of the activation subspace.
   • Mechanism: Inspired by Heterogeneous Dropout, it applies targeted sparsification to compress the activation representation.
   • Effect: By forcing activations to be sparse, the effective rank of the subspace is reduced. This creates a higher-dimensional null space (a "sanctuary") where future tasks can be learned without interfering with prior knowledge.
   • Outcome: This ensures that even if subspaces are oriented correctly, they remain compact, maximizing the capacity for future tasks.

3. Key Contributions

1. Novel Paradigm: Introduced SGDS, the first method to mitigate CIL interference by guiding activation subspaces (via orientation and rank control) rather than constraining adapter parameters.
2. Stability-Plasticity Balance: Demonstrated that leaving parameters unconstrained while structuring the activation space allows for superior plasticity (learning new tasks) without sacrificing stability (retaining old knowledge).
3. State-of-the-Art Performance: Achieved new records on major benchmarks, outperforming leading parameter-constrained methods.
4. Exemplar-Free Efficiency: The method operates without storing past data (exemplars), ensuring privacy and low storage costs.

4. Experimental Results

The authors evaluated SGDS on four benchmarks: CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet, using a ViT-B/16 backbone.

• Performance Gains:
  • On ImageNet-R, SGDS achieved an average accuracy of 85.41%, outperforming the previous SOTA (TUNA) by 1.19%.
  • On CIFAR-100, it reached 94.98% average accuracy.
  • It consistently outperformed both parameter-constrained methods (e.g., L2P, CODA-Prompt, TUNA) and strong rehearsal-based methods (e.g., iCaRL, FOSTER), despite being exemplar-free.
• Ablation Studies:
  • Both Semantic Exploration (orientation) and Activation Compaction (rank) were proven essential. Removing either led to significant performance drops.
  • Guiding activations in later layers of the network (where task-specific information is concentrated) was found to be more effective than global parameter constraints.
• Visualization: t-SNE visualizations confirmed that SGDS organizes activations into distinct, compact clusters based on semantic similarity, whereas baseline models showed intermingled, unstructured activations.

5. Significance and Impact

• Theoretical Shift: The paper challenges the prevailing dogma that interference must be solved by constraining parameters. It proves that activation-space guidance is a more flexible and potent mechanism for managing the stability-plasticity dilemma.
• Practical Applicability:
  • Privacy: As an exemplar-free method, it is ideal for privacy-sensitive domains (e.g., healthcare) where storing past user data is prohibited.
  • Efficiency: It reduces computational overhead by avoiding the storage and replay of large exemplar buffers.
  • Scalability: The method scales well as the number of tasks increases, maintaining performance where other methods degrade.

In summary, SGDS represents a significant advancement in Continual Learning by leveraging the structure of neural activations to dynamically allocate resources, offering a robust solution to catastrophic forgetting without compromising the model's ability to learn new concepts.