From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning

Imagine you are a master chef who has just opened a new restaurant. Every week, you add a new menu item to your repertoire.

The Problem: The "Unorganized Fridge"
In the world of AI, this is called Class-Incremental Learning. The AI needs to learn new things (like recognizing a "poodle" or a "scooter") without forgetting how to recognize the old things (like a "cat" or a "bicycle").

Most current methods work like a chef who keeps every single recipe in a giant, messy pile on the counter.

When a customer orders a "poodle," the chef has to scan through every single recipe in the pile to find the right one.
If the chef gets too many recipes, the pile becomes huge, and it takes forever to find the right one.
Worse, the chef might accidentally mix up the "poodle" recipe with the "cat" recipe because they are just sitting next to each other in the mess. This is called "catastrophic forgetting" or just general confusion.

The Solution: The "Smart Library" (SAEF)
The authors of this paper propose a new system called SAEF (Semantic-guided Adaptive Expert Forest). Instead of a messy pile, they build a Smart Library with a specific organizational structure.

Here is how it works, step-by-step:

1. Grouping by "Concept" (The Bookshelves)

First, the system looks at all the new things it has learned and groups them by their "vibe" or concept.

It puts all the animals (dogs, cats, birds) on Shelf A.
It puts all the vehicles (cars, bikes, scooters) on Shelf B.
It puts all the furniture (chairs, tables) on Shelf C.

This is called Conceptual Clustering. It stops the system from trying to compare a "dog" directly with a "scooter" unless absolutely necessary.

2. Building the "Tree of Experts" (The Branches)

Inside each shelf (like the Animal shelf), the system builds a family tree of experts.

It takes the "Dog" recipe and the "Cat" recipe. Since they are similar, it creates a new "Pet" expert that knows the basics of both.
It takes the "Pet" expert and the "Bird" expert and creates a "Creature" expert.
This creates a hierarchy: At the top is a general "Animal" expert. Below that are specific "Pet" and "Bird" experts. At the bottom are the specific "Dog" and "Cat" experts.

This is the Expert Forest. It's not just a flat list; it's a structured tree where knowledge is shared logically.

3. The "Smart Search" (Finding the Recipe)

Now, a customer walks in and orders a picture of a Golden Retriever.

The Old Way: The chef scans the entire messy pile of 1,000 recipes. Slow and confusing.
The SAEF Way:
1. The system asks the "Global Expert" (the head librarian): "Is this an animal, a vehicle, or furniture?" The librarian says, "Animal."
2. The system goes to the Animal Shelf.
3. It asks the "Pet vs. Bird" expert: "Is this a pet or a bird?" The expert says, "Pet."
4. It goes to the "Pet" branch and asks the "Dog vs. Cat" expert: "Is this a dog?" The expert says, "Yes, very confident!"
5. Result: The system found the answer by checking only 3 or 4 specific experts instead of 1,000.

4. The "Confidence Vote" (The Final Decision)

Sometimes, the answer isn't 100% clear. Maybe the picture is a bit blurry.

The system doesn't just pick one expert. It gathers opinions from the "Animal" expert, the "Pet" expert, and the "Dog" expert.
However, it listens more to the experts who are most confident. If the "Dog" expert is 99% sure, its opinion counts more than the "Animal" expert who is only 60% sure.
This creates a final, highly accurate prediction.

Why is this a big deal?

Speed: Because it doesn't have to check every single recipe, it is much faster (up to 6 times faster in their tests).
Accuracy: Because it groups similar things together, it doesn't get confused. It remembers old things better because it understands how they relate to new things.
No Memory Loss: It doesn't need to keep old photos in a drawer to remember them. The "tree structure" itself holds the memory efficiently.

In a Nutshell:
Instead of throwing all your knowledge into a giant, chaotic bucket, SAEF organizes it into a well-labeled, multi-level library. When you need an answer, it doesn't search the whole library; it walks down the right aisle, up the right branch, and finds the exact expert you need in a flash.

1. Problem Statement

Class-Incremental Learning (CIL) aims to train models to learn new classes sequentially without forgetting previously learned ones. While Pre-trained Model (PTM) based CIL using Parameter-Efficient Fine-Tuning (PEFT) (specifically training lightweight adapters for each task) has become a state-of-the-art approach to prevent catastrophic forgetting, current methods suffer from a critical limitation:

Unstructured Knowledge: Existing methods treat task-specific adapters as a flat, unstructured pool. During inference, they often rely on ensembling all adapters or retrieving a single best match.
Inefficiency and Noise: This "flat" approach fails to leverage semantic relationships between tasks (e.g., the similarity between "dog" and "cat"). It forces the model to consider all prior knowledge equally, leading to noisy predictions and inefficient inference (requiring $O(N)$ complexity where $N$ is the number of tasks).
Suboptimal Transfer: Global knowledge sharing across all tasks is often counterproductive; true knowledge transfer should be selective, hierarchical, and context-specific.

2. Methodology: Semantic-guided Adaptive Expert Forest (SAEF)

The authors propose SAEF, a framework that transforms isolated task adapters into a structured, hierarchical "forest" of experts. The method operates in three distinct stages:

Stage 1: Conceptual Clustering

Instead of treating all tasks as a single group, SAEF first groups tasks into semantic clusters.

Prototypes: For each task $t$ $t$ , two prototypes are defined:
- Semantic Prototype ( $p^s_t$ ): Derived from text embeddings of class names using a frozen CLIP text encoder.
- Visual Prototype ( $p^f_t$ ): The mean feature vector of training samples extracted by the PTM backbone and the first task's adapter (to bridge domain gaps).
Clustering: The K-Means algorithm partitions the $T$ tasks into $K$ conceptual clusters (e.g., "Animals," "Vehicles") based on semantic prototypes. The optimal $K$ is determined automatically using the silhouette score.

Stage 2: Hierarchical Construction

Within each conceptual cluster, SAEF builds a balanced binary tree of experts via a bottom-up merging process.

Leaf Nodes: Each initial task adapter serves as a leaf node.
Merging: The algorithm iteratively finds the two most visually similar nodes (based on cosine similarity of their visual prototypes) and merges them into a new parent node.
Parameter Merging: The parent's adapter parameters ( $\theta_p$ ) are created by merging child parameters ( $\theta_i, \theta_j$ ) using a vectorized operation that preserves dominant signals:
$\theta_p \leftarrow \text{sign}(\theta_i + \theta_j) \odot \max(|\theta_i|, | \theta_j|)$
Global Expert: Finally, the roots of all $K$ trees are merged into a single Global Expert to serve as the entry point for inference.

Stage 3: Adaptive Inference

SAEF employs a dynamic, entropy-guided search strategy to classify new inputs.

Parallel Search: For a given input, the model performs a parallel search down each of the $K$ trees.
Entropy-Guided Path Selection: Starting from the root, the search recursively descends to the child node that yields the lowest predictive entropy (highest confidence). This creates an optimal path from root to leaf for each tree.
Fusion: The final prediction is a weighted average of the outputs from the Global Expert and all experts along the identified paths. The weights are determined by the negative entropy (confidence) of each expert, normalized by a temperature parameter $\tau$ .

3. Key Contributions

Structured Knowledge Organization: SAEF is the first method to automatically organize isolated task adapters into a semantic hierarchy (a forest of trees), enabling targeted knowledge transfer between related tasks while isolating unrelated ones.
Adaptive Inference Strategy: It introduces a dynamic search mechanism that activates only a subset of relevant experts based on input entropy, significantly reducing inference latency compared to full ensembles.
State-of-the-Art Performance: The method achieves superior accuracy on multiple benchmarks, outperforming existing PTM-based CIL methods without requiring exemplar storage (exemplar-free setting).

4. Experimental Results

The authors evaluated SAEF on four challenging benchmarks: CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet, using a ViT-B/16 backbone.

Accuracy: SAEF achieved SOTA performance across all datasets.
- On ImageNet-R, it achieved 84.54% average accuracy, outperforming the previous best (SSIAT) by 1.34%.
- On CIFAR-100, it reached 94.53% average accuracy.
Comparison with Rehearsal Methods: Despite being an exemplar-free method (storing no past data), SAEF outperformed strong rehearsal-based methods (like FOSTER and MEMO) that store 20 exemplars per class. This highlights the scalability of structured knowledge retrieval over instance replay.
Efficiency:
- Inference Speed: By using the adaptive search, SAEF reduces inference time by ~5.9x compared to full-ensemble methods (like EASE) while maintaining high accuracy.
- Complexity: The hierarchical structure reduces inference complexity from $O(N)$ to $O(\log N)$ within trees.
Ablation Studies: Removing any component (Clustering, Hierarchical Construction, or Adaptive Inference) resulted in performance drops, confirming that the synergy between semantic grouping and entropy-guided search is critical.

5. Significance

This paper addresses a fundamental bottleneck in continual learning: the trade-off between forgetting prevention and inference efficiency.

Paradigm Shift: It moves the field from treating adapters as a "bag of tricks" to viewing them as a structured, hierarchical knowledge base.
Practicality: The method is highly suitable for resource-constrained environments (e.g., edge devices, robotics) because it eliminates the need for storing old data (exemplars) while drastically speeding up inference compared to current ensemble methods.
Generalizability: The approach of organizing parameters based on semantic relationships offers a new direction for improving knowledge reuse in large-scale pre-trained models.