🤖 machine learning

Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts

The paper proposes CaRE, a scalable continual learning framework utilizing a bi-level routing Mixture-of-Experts mechanism to effectively handle extremely long task sequences of over 300 tasks, accompanied by the introduction of the challenging OmniBenchmark-1K dataset for evaluation.

Original authors: Meng Lou, Yunxiang Fu, Yizhou Yu

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Meng Lou, Yunxiang Fu, Yizhou Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize thousands of different objects, one new category at a time. The problem is that as you teach it new things (like "corgis"), it tends to forget the old things it learned (like "hamburgers"). This is called "catastrophic forgetting."

The paper introduces a new system called CaRE (Continual Learner with efficient Bi-Level Routing Mixture-of-Experts) designed to solve this. Here is how it works, explained simply:

1. The Problem: The "One-Size-Fits-All" Trap

Most current AI systems try to learn new tasks by tweaking the same brain they already have. If you teach them about dogs, they might accidentally overwrite the rules for recognizing cats.

The Paper's Insight: Instead of forcing the AI to use the same "brain" for everything, CaRE gives the AI a massive library of specialized tools. When a new task arrives, the AI doesn't just learn; it learns how to pick the right tools from its library.

2. The Solution: The "Smart Librarian" System (BR-MoE)

CaRE uses a mechanism called Bi-Level Routing Mixture-of-Experts (BR-MoE). Think of the AI's brain as a giant library with many specialized "experts" (mini-brains) inside it.

Level 1: The Head Librarian (Router Selection)
When a new image comes in (say, a picture of a Corgi), the system doesn't just guess. It asks a series of "Head Librarians" (called Class Perceptrons) to look at the image and say, "How confident are you that this belongs to your specific section?"
- If the "Dog Librarian" is very confident (low uncertainty), it gets picked.
- If the "Car Librarian" is confused (high uncertainty), it is ignored.
- The Trick: The system picks the top few librarians who seem most relevant, not just one. This ensures the AI considers related concepts (like other animals) while focusing on the main one.
Level 2: The Specialized Experts (Dynamic Expert Routing)
Once the top librarians are chosen, they each call upon their own team of Specialized Experts (small, efficient adapters).
- Each expert is a mini-brain trained specifically on a past task (e.g., one expert knows all about dogs, another about birds).
- The librarians activate the top few experts that are most helpful for the current image.
- The "Shared Expert": There is also one "Super-Expert" that knows a little bit about everything learned so far. This ensures the AI never loses its general knowledge.

3. The "Every Layer" Advantage

Usually, AI systems make decisions only at the very end. CaRE is different: it has this "Librarian + Expert" system built into every single layer of its brain.

Analogy: Imagine a factory assembly line. In a normal factory, the final manager decides what to do with the product. In CaRE, every worker on the line (every layer) has their own mini-librarian who decides which specific tools to use right then and there. This allows the AI to build a very detailed and accurate picture of the object step-by-step.

4. The New Challenge: The "OmniBenchmark-1K"

To prove this works, the authors realized existing tests were too easy. Most tests only had 20 tasks (like learning 20 types of animals).

They created a new, super-hard test called OmniBenchmark-1K.
This dataset has 1,000 different classes (categories) and nearly 200,000 images.
They tested CaRE by teaching it in sequences of 100, 200, and even 301 tasks in a row.
The Result: While other AI systems started to fail and forget things as the number of tasks grew, CaRE kept getting better. It outperformed all other methods by a large margin, even when the task list was incredibly long.

Summary

CaRE is like a student who doesn't just memorize facts but learns how to recall the right memory for the right situation.

It checks its confidence to find the most relevant "memory rooms" (Level 1 Routing).
It pulls out the specific "tools" from those rooms to solve the problem (Level 2 Routing).
It does this at every step of its thinking process, not just at the end.

This allows it to learn hundreds of new things without forgetting the old ones, a feat no other system has successfully demonstrated at this scale.

Technical Summary: Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts (CaRE)

1. Problem Statement

Continual learning, particularly Class-Incremental Learning (CIL), aims to enable models to learn new tasks sequentially without forgetting previous knowledge. While recent approaches leveraging Pre-trained Models (PTMs) and Parameter-Efficient Fine-Tuning (PEFT) have shown promise, they face significant challenges when scaling to very long task sequences (e.g., hundreds of tasks).

The core difficulties identified are:

Balancing Discriminative and Comprehensive Representations: Existing methods often struggle to generate representations that are both highly discriminative for specific classes and comprehensive enough to leverage complementary knowledge from semantically related historical tasks. Coarse-grained strategies (e.g., global prompts) fail to exploit fine-grained complementary cues effectively.
Multi-level Local Decisions: Vision models possess varying levels of abstraction across different network depths. A static routing strategy fails to make optimal local decisions at each intermediate layer to selectively incorporate historical knowledge.
Lack of Scalable Benchmarks: Most existing CIL benchmarks (e.g., CIFAR-100, standard ImageNet subsets) are limited to a small number of classes (often <100), making them unsuitable for evaluating performance on long sequences (100–300+ tasks). This leaves the scalability of current methods to real-world, unbounded task streams largely unexplored.

2. Methodology: CaRE and BR-MoE

The authors propose CaRE (Continual Learner with efficient Bi-Level Routing Mixture-of-Experts), built upon a frozen Vision Transformer (ViT) backbone. The core innovation is the Bi-Level Routing Mixture-of-Experts (BR-MoE) module, which is integrated into every ViT building block.

2.1 Architecture Overview

For each incremental task $t$ , CaRE learns a triplet of parameter-efficient components within every BR-MoE module:

Class Perceptron ( $C_t$ ): Extracts class-level discriminative information.
Router Network ( $R_t$ ): Generates gating coefficients for expert selection.
Expert/Adapter ( $E_t$ ): A task-specific adapter for feature transformation.
Additionally, a Shared Expert ( $\bar{E}$ ) is maintained, updated via Exponential Moving Average (EMA) across all tasks to encode consolidated cross-task knowledge.

2.2 The Bi-Level Routing Mechanism

The routing process occurs in two dynamic stages for every input feature $z_a$ at each layer:

Stage 1: Dynamic Router Selection

The input feature's [CLS] token is fed into all learned class perceptrons $\{C_t\}$ .
Each perceptron produces a probability distribution over its task's classes.
The entropy of these distributions is calculated. Lower entropy indicates higher confidence that the input belongs to that task.
The system selects the Top-M router networks associated with the lowest entropy values (most likely tasks). This ensures the model retrieves knowledge from the most semantically relevant historical tasks.

Stage 2: Dynamic Expert Routing

Each of the selected $M$ routers generates gating scores for its associated set of task-specific experts (adapters).
For each router, the Top-K experts with the highest gating scores are activated and aggregated.
The Shared Expert ( $\bar{E}$ ) is always activated to enrich the representation with general cross-task knowledge.
The final output is the sum of the aggregated task-specific features and the shared expert's output.

2.3 Training Objectives

Classification Loss: Standard angular penalty loss is applied at the final layer.
Class Perceptron Supervision: To ensure robust semantic guidance at intermediate layers, the class perceptron at each layer $\ell$ is supervised not only by its own classification loss but also by a KL-divergence loss against the final-layer softmax probabilities. This forces intermediate layers to mimic high-level semantic abstractions, enabling effective local decision-making.

3. Key Contributions

3.1 The CaRE Framework

CaRE is the first continual learner designed to scale effectively to very long task sequences (100 to 300+ non-overlapping tasks). By employing a bi-level routing mechanism, it dynamically injects both discriminative and comprehensive representations into every intermediate network layer, allowing for customized knowledge retrieval at different levels of abstraction.

3.2 OmniBenchmark-1K

To address the lack of suitable evaluation protocols for long sequences, the authors introduce OmniBenchmark-1K. Curated from OmniBenchmark-V2, this dataset contains 1,000 classes across 21 visual realms with ~190,000 images. It is specifically designed to facilitate scalable CIL evaluations with hundreds of tasks, overcoming the limitations of existing benchmarks like CIFAR-100.

3.3 Empirical Validation

Extensive experiments demonstrate that CaRE outperforms state-of-the-art PTM-based CIL methods (e.g., L2P, DualPrompt, TUNA, MOS, MIN) across diverse settings:

Long Sequences: On OmniBenchmark-1K, CaRE achieves significant gains in last accuracy ( $A_B$ ) over baselines across 100, 151, 200, and 301 task settings. For instance, it outperforms TUNA by 8.23% and MIN by 8.68% in $A_B$ on 100 and 151 tasks, respectively.
Short Sequences: CaRE also maintains superior performance on classical benchmarks (CIFAR-100, ImageNet-R, ImageNet-A, VTAB) with 5–20 tasks, demonstrating robustness across sequence lengths.
Efficiency: CaRE achieves a favorable trade-off between performance and computational cost, using significantly fewer trainable parameters and lower inference latency compared to methods like MOS and TUNA while delivering higher accuracy.

4. Significance and Claims

The paper claims that CaRE represents a significant step forward in making continual learning practical for real-world scenarios where data streams are non-stationary and task sequences are potentially unbounded.

Scalability: The work is presented as the first to successfully scale CIL to over 300 non-overlapping tasks while maintaining clear performance superiority.
Mechanism Validity: The authors argue that the bi-level routing mechanism effectively addresses the "stability-plasticity" dilemma by allowing the model to dynamically retrieve relevant historical knowledge (via router selection) and aggregate complementary features (via expert routing) at every layer.
Community Impact: By releasing both the CaRE code and the OmniBenchmark-1K dataset, the authors aim to inspire further research into extremely long task sequences, a domain previously hindered by a lack of appropriate benchmarks and scalable methods.

The paper concludes with a modest acknowledgment that while CaRE scales to 300+ tasks, the model complexity still grows linearly with the number of tasks (due to appending new modules), and future work may focus on further architectural simplification and extending these protocols to vision-language models.