💻 computer science

Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning

The paper proposes Miles, a class-incremental learning method that leverages pre-trained models by decoupling learnable modules and utilizing an expandable subspace with metric learning to effectively balance performance and computational cost while mitigating catastrophic forgetting.

Original authors: Kai Jiang, Zisong Lin, Hongyuan Zhang, Xueru Bai, Xuelong Li

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kai Jiang, Zisong Lin, Hongyuan Zhang, Xueru Bai, Xuelong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot to recognize new things, like a new type of bird or a rare flower, without it forgetting everything it already knows about dogs and cats. This is the world of Class Incremental Learning (CIL). In the past, if you wanted to teach a robot a new concept, you often had to retrain its entire brain from scratch, which was slow and prone to "catastrophic forgetting"—where the robot would suddenly forget how to recognize a dog just because it learned about a new flower.

To solve this, scientists started using Pre-Trained Models (PTMs). Think of these as robots that have already spent years reading every book in the library and watching every nature documentary. They already know the basics of the world. The challenge now is how to teach these "expert" robots new, specific skills without messing up their existing knowledge or making them too heavy and slow to run. The big question is: How do we add new skills to an expert's brain efficiently, keeping the old skills sharp while learning the new ones, all without needing a massive library of old photos to review?

Enter MILES (Metric Learning with Expandable Subspace), a new method proposed by researchers Kai Jiang and their team. They noticed that even though these pre-trained robots are smart, they sometimes get confused when trying to learn new categories because the "mental space" where they store ideas gets cluttered. The old ideas and new ideas start overlapping, like trying to park a new car in a garage that's already packed with old ones.

To fix this, MILES uses a clever two-part strategy. First, it creates a Subspace Prototype Projection (SPP). Imagine the robot has a giant, messy garage (the feature space). MILES builds a series of tiny, specialized "mini-garages" or subspaces for each new task. When a new category arrives, the robot projects its features into this new mini-garage, where it can neatly organize the new items right next to a "prototype" (a perfect example) of that item, keeping them far away from the old, unrelated items. This is like having a dedicated shelf for your new collection of stamps so they don't get mixed up with your old coin collection.

However, just building a shelf isn't enough if the robot's eyes (the pre-trained model) aren't seeing the details clearly enough for the new task. So, MILES adds a Subspace Representation Extension (SRE). This is like giving the robot a pair of specialized, lightweight glasses for each new task. These glasses don't replace the robot's main vision; they just add a little extra zoom and clarity to the intermediate steps of seeing, allowing the robot to learn the new task much better without needing to rebuild its entire brain.

Finally, the researchers noticed a problem: sometimes the robot gets too good at the new task and starts ignoring the old ones, or the "distance" between items gets weirdly inconsistent. To stop this, they introduced Distance Regularization (DR). Think of this as a strict librarian who ensures that the distance between any two books on the shelf is always fair and consistent, no matter which section of the library you are in. This prevents the robot from getting too obsessed with the new stuff and forgetting the balance of the whole library.

The team tested MILES on six different datasets, ranging from simple images like CIFAR-100 to complex real-world photos like ImageNet-R. The results were impressive: MILES achieved the highest accuracy in most settings, beating other top methods. For example, on the CIFAR-100 dataset with 10 tasks, it reached an accuracy of 93.93%, which was nearly 1% better than the second-best method, while forgetting very little of what it learned before. The method also proved to be efficient, using very few extra computer resources and memory compared to other approaches.

In short, MILES suggests that by combining the power of pre-trained knowledge with smart, lightweight additions that organize new information into dedicated spaces and keep everything balanced, we can teach AI to learn continuously without forgetting. It's a step toward creating AI that can grow and adapt like a human, learning new things every day without losing its past.

Technical Summary: MILES (Metric Learning with Expandable Subspace)

Problem Statement
Class-Incremental Learning (CIL) aims to learn new concepts sequentially from a data stream without forgetting previously learned categories. While traditional CIL methods often train models from scratch—leading to long training times and poor generalization—Pre-trained Model (PTM)-based approaches offer a promising alternative by leveraging prior knowledge. However, existing PTM-based CIL methods face a critical trade-off between performance and computational efficiency. Methods that share the same parameter space often suffer from catastrophic forgetting, while those that expand a new branch for each task incur significant computational costs due to multiple forward propagations or linear increases in inference time with the number of tasks. Furthermore, directly fine-tuning PTMs across multiple downstream tasks still encounters severe forgetting, and existing methods often struggle to balance model plasticity (learning new tasks) and stability (retaining old knowledge) without modifying the frozen backbone or overfitting to new tasks.

Methodology
The proposed method, MILES (Metric Learning with Expandable Subspace), addresses these challenges by decoupling learnable modules from the pre-trained backbone and orchestrating an efficient expansion of the parameter space through guided optimization. The architecture consists of a frozen pre-trained backbone (e.g., Vision Transformer) and three key components:

Subspace Prototype Projection (SPP):
- Motivation: Observations indicate that unseen categories in PTM feature spaces exhibit clustering rather than dispersion. MILES leverages this by computing class prototypes for new categories without training.
- Mechanism: For each new task $t$ , a lightweight, learnable adapter ( $A_t$ ) is cascaded after the backbone. This adapter projects the original features into a task-specific subspace.
- Structure: The adapter utilizes a residual design comprising a full-dimensional MLP branch and a progressive down/up projection branch (similar to a 1D U-Net operating on channel dimensions). This structure compresses features into a low-dimensional bottleneck and restores them, enabling task-specific feature re-parameterization while maintaining compatibility with the frozen backbone.
- Optimization: A center loss minimizes the distance between intra-task features and their corresponding prototypes. To prevent overfitting to new tasks, a stability loss is introduced, which maximizes the distance between pseudo-features of old categories (modeled via Gaussian distributions) and the new task prototypes.
Subspace Representation Extension (SRE):
- Motivation: Relying solely on the frozen backbone limits the model's ability to learn high-dimensional semantics from new data, as the backbone cannot adapt to new task distributions.
- Mechanism: A low-rank sub-network ( $S_t$ ) is extended for each task to learn intermediate features from the pre-trained model.
- Structure: The SRE sub-network mimics the architecture of the backbone's transformer blocks using two low-rank MLP modules per block. One approximates the attention module, and the other approximates the Feed-Forward Network (FFN).
- Integration: The output of the SRE sub-network is concatenated with the original backbone features before being fed into the SPP adapter. This enhances the model's representational capacity without modifying the pre-trained weights or requiring multiple forward passes during inference (only one forward pass through the backbone is needed, followed by parallel processing of SRE branches).
Distance Regularization (DR):
- Motivation: Without unified constraints, the model may overfit to specific task subspaces, resulting in significantly smaller distances for new tasks compared to previous ones, leading to bias.
- Mechanism: Two regularization terms are added to the loss function:
  - Task Average Distance Constraint: Ensures the average distance from feature vectors to irrelevant task prototypes remains consistent across tasks.
  - Class Average Distance Constraint: Maintains a consistent average distance between intra-task features and their prototypes across all task subspaces.
- Goal: These terms balance the metric equilibrium across tasks, mitigating overfitting and ensuring a consistent classification criterion.

Key Contributions

Integration of Metric Learning and PTMs: The paper integrates metric learning with pre-trained models in a CIL setting, proposing SPP to optimize the distance between current task features and corresponding prototypes within a new task subspace.
Subspace Representation Extension (SRE): A novel low-rank sub-network is introduced to learn intermediate features from the PTM. This strengthens the model's representational capacity with minimal modification to the pre-trained network and avoids the linear increase in inference time associated with multi-branch architectures.
Distance Regularization (DR): A regularization strategy is proposed to maintain consistent average distances for each task, effectively solving the overfitting issue often observed in new task subspaces.

Experimental Results
Extensive experiments were conducted on six benchmark datasets: CIFAR-100, CUB-200, Omnibenchmark, ImageNet-A, ImageNet-R, and FOOD-101. The evaluation covered both short ( $T=10$ ) and long ( $T=50$ ) task sequences.

Performance: MILES achieved state-of-the-art (SOTA) performance across various CIL settings. For instance, on CIFAR-100 ( $T=10$ ), it achieved 93.93% average accuracy with a forgetting rate of 3.12%, outperforming the runner-up (DGR) by 0.88% in accuracy and 1.73% in forgetting rate. Similar superior results were observed on CUB, Omnibenchmark, and ImageNet variants.
Efficiency: The method maintains a low computational overhead. It requires only a single forward pass through the frozen backbone during inference. The number of trainable parameters is comparable to the smallest competing methods (e.g., FeCAM), and the Multiply-Accumulate Operations (MACs) are minimal compared to methods that require multiple forward passes (like EASE) or extensive parameter updates.
Ablation Studies: Experiments confirmed the effectiveness of each component. Adding SPP improved performance over a baseline; adding SRE further enhanced representational capacity; and adding DR consistently reduced the forgetting rate and stabilized performance across different hyperparameter settings.
Robustness: The method demonstrated robustness to different learning orders and hyperparameter combinations.

Significance and Claims
The paper claims that MILES successfully orchestrates an efficient expansion of the parameter space by harnessing prior information within pre-trained knowledge. By decoupling learnable modules and employing guided optimization via metric learning, MILES achieves a superior trade-off between performance and computational expenditure compared to existing PTM-based CIL methods. The work validates that leveraging the clustering nature of features in PTM spaces, combined with lightweight subspace extensions and distance regularization, allows for effective continual learning without catastrophic forgetting or excessive computational cost.

Limitations and Future Work
The authors acknowledge that the current study focuses on standard class-incremental settings where the pre-trained backbone provides transferable representations and downstream data remains within the target semantic scope. The method's robustness under Out-of-Distribution (OOD) shifts (in style, acquisition conditions, or semantic content) has not been systematically investigated. Future work aims to extend MILES toward OOD-aware CIL by incorporating uncertainty-aware modeling and test-time shift correction.

Technical Summary: MILES (Metric Learning with Expandable Subspace)

More like this