One Adapter for All: Towards Unified Representation in Step-Imbalanced Class-Incremental Learning

This paper introduces One-A, a unified framework for step-imbalanced class-incremental learning that employs asymmetric subspace alignment and directional gating to merge task updates into a single adapter, thereby balancing stability and plasticity while maintaining constant inference costs.

The Gist

Imagine you are a chef running a restaurant. Your goal is to learn new recipes over time without forgetting the old ones. This is exactly what Class-Incremental Learning (CIL) tries to do for AI: teach it new categories of things (like "cats," "dogs," "cars") while remembering everything it learned before.

However, most AI research assumes you learn in a very neat, balanced way: maybe you learn 10 new recipes every Tuesday.

The Real-World Problem: The "Step Imbalance"
In the real world, learning isn't neat.

• Monday: You get a massive delivery of 50 new exotic fruits to learn. (A "Big Step")
• Tuesday: You get just 2 new types of berries. (A "Small Step")
• Wednesday: You get 30 new vegetables.

This is what the paper calls Step-Imbalanced Class-Incremental Learning (SI-CIL).

Why is this a disaster for current AI?
Current AI methods treat every day the same. They try to learn the 2 berries just as intensely as the 50 fruits.

• The Result: The AI gets confused. The tiny, noisy updates from the "2 berries" day mess up the solid knowledge it built on the "50 fruits" day. It's like trying to rearrange a massive library because someone added two new books; you might accidentally knock over the whole shelf.
• The "Adapter" Problem: Usually, to learn new things, AI creates a new "notebook" (called an adapter) for every day. If you have 100 days, you have 100 notebooks. To make a prediction, the AI has to flip through all 100 notebooks, which is slow and expensive.

The Solution: "One Adapter for All" (One-A)
The authors propose a clever new system called One-A. Instead of keeping 100 separate notebooks, they merge everything into one single, super-smart notebook.

Here is how they do it, using three simple metaphors:

1. The "Big Boss" vs. The "Intern" (Asymmetric Alignment)

Imagine the "Big Step" (50 fruits) is the Big Boss, and the "Small Step" (2 berries) is an Intern.

• Old Way: The Boss and the Intern sit at the same table and try to agree on everything. The Intern's tiny, shaky opinions might accidentally convince the Boss to change their mind about how to slice a mango.
• One-A Way: The Boss sets the table (the "Dominant Subspace"). The Intern is only allowed to write notes within the Boss's existing framework. The Intern can add details, but they can't move the table or change the Boss's core rules. This keeps the AI stable.

2. The "Volume Knob" (Information-Adaptive Weighting)

Not all days are equally important.

• If you learned 50 fruits, that's a lot of information. You should listen to that day more.
• If you learned 2 berries, that's less information.
• One-A automatically turns up the volume on the "Big Days" and turns down the volume on the "Small Days." It doesn't treat a 2-berry day the same as a 50-fruit day. It weighs the importance of the data before merging it.

3. The "Traffic Light" (Directional Gating)

This is the most clever part. Imagine the AI's knowledge is a highway with many lanes (directions).

• The Fast Lanes (High Energy): These are the main roads where the AI learned the most important things (like "what a fruit is"). We want to keep these lanes closed to new traffic so we don't crash.
• The Side Streets (Low Energy): These are the quiet roads where the AI hasn't learned much yet. We can open these lanes to let new, small updates in.
• One-A puts a traffic light on every single lane.
  • If a lane is a "Main Road" (learned from a big task), the light turns Red (Stop! Don't change this).
  • If a lane is a "Side Street" (less important), the light turns Green (Go! Add the new berry info here).

The Final Result
By using these three tricks, One-A manages to:

1. Remember everything: It doesn't forget the big lessons.
2. Learn new things: It can still absorb small, new details without getting confused.
3. Be Fast: Instead of carrying 100 notebooks, it only carries one. When you ask it a question, it doesn't have to search through a pile of papers; it just looks at its single, perfectly organized notebook.

In Summary
The paper solves the problem of "uneven learning days" by teaching the AI to respect the big lessons while carefully fitting in the small ones, all while keeping its memory system lightweight and fast. It's like upgrading from a chaotic filing cabinet to a single, self-organizing, super-efficient brain.

Technical Summary

1. Problem Definition: Step-Imbalanced Class-Incremental Learning (SI-CIL)

Context:
Class-Incremental Learning (CIL) aims to learn new classes over time while retaining knowledge of previous ones. Most existing research assumes a balanced setting where each incremental task introduces the same number of classes.

The Gap (SI-CIL):
In real-world applications (e.g., e-commerce seasonal updates vs. daily arrivals), the number of classes per task varies significantly. The authors define this as Step-Imbalanced CIL (SI-CIL), where "steps" (incremental updates) contain highly unequal numbers of classes.

• Large Tasks: Contain many classes, providing abundant supervision and stable gradients.
• Small Tasks: Contain few classes, generating noisy, unstable updates.

Challenges:

1. Optimization Bias: Existing methods treat all tasks equally. Large tasks dominate the shared representation space, while small tasks inject unstable updates that degrade performance.
2. Subspace Drift: Naive merging (averaging) or symmetric alignment causes the dominant subspaces learned from large tasks to be distorted by noisy updates from small tasks.
3. Inference Overhead: Many state-of-the-art methods maintain separate adapters for each task, leading to linearly increasing inference costs and parameter overhead as tasks accumulate.
4. Catastrophic Forgetting: Small tasks often overfit or shift the shared representation, causing the model to forget knowledge from large, earlier tasks.

2. Methodology: One-A Framework

The authors propose One-A, a unified framework that maintains a single adapter throughout the learning process. It incrementally merges task-specific updates using an asymmetric, direction-aware strategy.

Core Components:

A. Asymmetric Subspace Alignment (ASA)

• Concept: Unlike symmetric methods (e.g., KnOTS) that align all tasks equally, One-A identifies the "Base" adapter (from the larger task or accumulated history) and the "Align" adapter (from the smaller current task).
• Mechanism:
  1. Perform Singular Value Decomposition (SVD) on the Base adapter ($\Delta W_b = U_b \Sigma_b V_b^T$) to extract its dominant subspace.
  2. Freeze the dominant subspace ($U_b \Sigma_b$).
  3. Project the Align adapter into this fixed subspace: $V_{a \to b} = (\Delta W_a^T U_b \Sigma_b^{-1})^T$.
• Goal: Prevent small tasks from rotating or redefining the critical subspaces learned from large tasks, thereby stabilizing the core representation.

B. Information-Adaptive Global Weighting (GW)

• Concept: Not all tasks contribute equally. The fusion weights are determined by the "information content" of the task.
• Mechanism:
  • The authors use the number of classes in a task as a proxy for information content ($\phi(\text{Info}_t) = \#\text{classes}$).
  • Global weights ($w_b, w_a$) are computed to balance the contribution of the Base and Align adapters during the initial fusion of singular vectors ($V_{fused} = w_b V_b + w_a V_{a \to b}$).
• Goal: Ensure that tasks with more diverse data (more classes) have a stronger influence on the global representation.

C. Directional Gating (DG)

• Concept: A single global weight cannot simultaneously preserve stability (for high-energy directions) and allow plasticity (for low-energy directions).
• Mechanism:
  • Compute a gate $g_i \in [0, 1]$ for each singular direction $i$ based on the normalized singular values of the Base adapter.
  • High-energy directions (Head): $g_i \approx 0$. The gate preserves the Base direction, preventing drift.
  • Low-energy directions (Tail): $g_i \approx 1$. The gate allows the Align adapter's update to be injected.
  • Formula: $V_{final} = V_b + g \odot (V_{fused} - V_b)$.
• Goal: Achieve a fine-grained balance between stability (protecting old knowledge) and plasticity (learning new knowledge) without manual thresholds.

D. Optimization Objective

• To aid small tasks with limited supervision, a contrastive loss is added as an auxiliary objective.
• The weight of this loss is adaptive: higher for small tasks (fewer classes) to enforce structural constraints, and lower for large tasks.
• A dynamic epoch schedule allocates more training epochs to larger tasks.

3. Key Contributions

1. Problem Formulation: Systematically introduces and analyzes Step-Imbalanced CIL, highlighting the unique challenges of task-size disparity compared to balanced or sample-imbalanced settings.
2. Unified Framework: Proposes One-A, the first method to merge task updates into a single adapter while explicitly handling step imbalance, eliminating the inference overhead of multi-adapter systems.
3. Asymmetric Fusion Strategy: Introduces a novel direction-aware asymmetric merging mechanism (ASA + GW + DG) that preserves dominant subspaces from large tasks while flexibly integrating information from small tasks.
4. State-of-the-Art Performance: Demonstrates that a single, asymmetrically fused adapter can outperform complex multi-adapter baselines while maintaining constant inference cost.

4. Experimental Results

Benchmarks:
Evaluated on CIFAR100, CUB200, ImageNet-A, and ImageNet-R under various step-imbalance factors ($\gamma$) and task counts ($T$).

Key Findings:

• Accuracy: One-A achieves the highest accuracy across all datasets and imbalance settings.
  • On ImageNet-A, it improved $A_T$ by 7.8% over the second-best method.
  • On ImageNet-R, it improved $A_T$ by 9.4%.
• Efficiency:
  • Inference Cost: One-A uses 1 adapter regardless of the number of tasks. In contrast, methods like MOS require $\sim 40\times$ more FLOPs due to multi-adapter retrieval.
  • Parameters: Maintains a constant parameter count, unlike methods that accumulate adapters.
• Ablation Studies:
  • Removing Asymmetric Alignment leads to significant performance drops due to subspace drift.
  • Removing Directional Gating results in a trade-off where the model either forgets too much (instability) or fails to learn new tasks (rigidity).
• Robustness: The method performs well even under extreme imbalance ($\gamma = 0.001$) and in descending task sequences (large-to-small).

5. Significance

• Real-World Applicability: SI-CIL better reflects real-world deployment scenarios (e.g., dynamic product catalogs, evolving sensor data) where data arrival is unpredictable and uneven. One-A provides a practical solution for these conditions.
• Efficiency vs. Performance: It breaks the traditional trade-off where high accuracy in CIL often requires expensive multi-adapter inference. One-A proves that a single, well-fused adapter can achieve superior performance with minimal computational overhead.
• Theoretical Insight: The work highlights that symmetric fusion strategies are fundamentally flawed for imbalanced data. The proposed asymmetric approach, which prioritizes the stability of dominant subspaces while allowing controlled plasticity in minor directions, offers a new paradigm for model merging in continual learning.

In summary, One-A addresses the critical gap between idealized balanced CIL research and messy real-world data streams, offering a scalable, efficient, and highly accurate solution for step-imbalanced continual learning.