🤖 AI

Beyond Point-wise Neural Collapse: A Topology-Aware Hierarchical Classifier for Class-Incremental Learning

This paper proposes a topology-aware hierarchical classifier, HC-SOINN, combined with a Structure-Topology Alignment via Residuals (STAR) method, to overcome the limitations of point-wise Neural Collapse assumptions in Class-Incremental Learning by effectively modeling complex class manifolds and adapting to non-linear feature drift.

Original authors: Huiyu Yi, Zhiming Xu, Dunwei Tu, Zhicheng Wang, Baile Xu, Furao Shen

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Huiyu Yi, Zhiming Xu, Dunwei Tu, Zhicheng Wang, Baile Xu, Furao Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize different types of animals. You start with dogs and cats. The robot learns well. Then, you introduce birds, then fish, then reptiles. The problem is that as the robot learns the new animals, it starts to "forget" the old ones or gets confused about what a dog actually looks like. This is called Catastrophic Forgetting.

For a long time, the best way to fix this was to treat every animal category as a single, perfect "average" point in the robot's mind. If you showed the robot a picture of a dog, it would just check: "Is this closer to the 'Average Dog' point or the 'Average Cat' point?"

The Problem: The "Average" is a Lie
The authors of this paper argue that this "single average point" idea is broken. In the real world, animals (and data) aren't perfect, single dots. They are complex, wiggly shapes.

The Analogy: Imagine trying to describe a "Dog" by picking just one spot in a park. But dogs run everywhere! Some are small, some are big, some are running left, some are running right. If you only remember one spot, you'll miss the dogs running in the other areas.
The Reality: As the robot learns new things, the "shape" of the old data (the dogs) shifts and twists in complicated, non-linear ways. A single point can't follow these twists. It's like trying to track a snake with a single dot; the dot will always be left behind.

The Solution: A Flexible Map (HC-SOINN)
Instead of one single point, the authors propose a new classifier called HC-SOINN. Think of this not as a single dot, but as a flexible, stretchy net or a connect-the-dots map.

Local Centers: Instead of one "Average Dog," the robot creates many small "local centers" (dots) that connect to each other. These dots form a web that covers the entire shape of where the dogs actually are.
Hierarchical: It builds this map in two steps. First, it makes a rough sketch (like drawing the outline of a dog). Then, it refines the details (adding the ears, tail, and paws) to make sure the map fits the data perfectly.
Dual-View Decision: When the robot sees a new picture, it checks two things:
- The Global View: "Is this generally close to the center of the dog family?"
- The Local View: "Is this specifically close to one of the detailed dots in our net?"
  It combines these two answers to make a smarter decision.

The Twist: The Map Moves (STAR)
There's a second problem. Even with a great map, the "ground" keeps moving. As the robot learns new tasks, the features of the old dogs shift and drift. A static map will eventually be in the wrong place.

The authors introduce a second tool called STAR.

The Analogy: Imagine your "Dog Net" is floating on a river. As the river flows (new data comes in), the net needs to move with the current, or it will get stuck on a rock.
How it works: STAR acts like a GPS tracker. It doesn't just watch the whole net move; it tracks every single dot in the net individually. It sees exactly how much each dot needs to shift to stay on top of the drifting data. It then gently pulls the whole net to follow the new shape of the data. This turns "resisting the drift" (fighting the river) into "adapting to the drift" (surfing the river).

The Results
The authors tested this new "Stretchy Net + GPS Tracker" system on three different sets of image data (like a mix of everyday objects, fine-grained bird species, and artistic drawings).

They took seven of the best existing AI methods and swapped their old "single-point" classifiers for their new "stretchy net" system.
The Outcome: In almost every case, the robot got much better at remembering old things while learning new ones. It reduced forgetting significantly, especially in difficult, long sequences of learning.
Efficiency: Surprisingly, making the map more complex didn't slow the robot down. Because the "heavy lifting" is done by the main brain (the feature extractor), adding this smart net to the end barely changed the time it took to make a decision.

In Summary
The paper says: "Stop trying to force complex, shifting data into a single, rigid point. Instead, build a flexible, multi-point map that can stretch and move along with the data." By doing this, the AI remembers the past much better while learning the future.

Technical Summary: Beyond Point-wise Neural Collapse

1. Problem Statement

Class-Incremental Learning (CIL) aims to enable models to learn new categories sequentially without catastrophically forgetting previously acquired knowledge. While the Nearest Class Mean (NCM) classifier has become the de facto standard for CIL due to its robustness against forgetting, its optimality relies heavily on the Neural Collapse (NC) theory. NC theory posits that during the terminal phase of training, intra-class features collapse into single points (class means), making the NCM classifier equivalent to an optimal linear classifier.

However, the authors argue that in practical CIL settings, this ideal state is frequently violated due to three factors:

Insufficient Convergence: Models are often not trained to the point of collapse to avoid overfitting.
Constrained Training: Incremental steps often use regularization or distillation that prevents full convergence on new classes.
Feature Drift: As the backbone updates for new tasks, the representations of old classes shift.

The paper demonstrates via Procrustes distance analysis that this feature drift is not a simple rigid transformation but involves significant non-linear deformations (with normalized distances reaching 0.3–0.4). Consequently, class features reside on complex, high-dimensional manifolds (e.g., "dumbbell" or "crescent" structures) rather than collapsed points. This renders the single-prototype NCM suboptimal, as it fails to represent these complex geometries, leading to performance degradation.

2. Methodology

The proposed framework consists of two main components: a topology-aware classifier (HC-SOINN) and a drift adaptation mechanism (STAR).

2.1. HC-SOINN (Hierarchical-Cluster SOINN)

To capture the intrinsic structure of class manifolds, the authors propose HC-SOINN, which replaces the single-prototype assumption with a "local-to-global" topological representation.

Hierarchical Initialization: Instead of applying incremental clustering directly to raw streams (which is sensitive to noise and order), the method first accumulates features in a buffer and performs Agglomerative Hierarchical Clustering (using UPGMA and cosine distance) to initialize a stable manifold skeleton.
Spherical SOINN Refinement: The initial nodes are refined using a simplified Self-Organizing Incremental Neural Network (SOINN) adapted for the unit hypersphere.
- Topology Learning: Edges are created between the winner and runner-up nodes based on Competitive Hebbian Learning (CHL).
- Spherical Update: Node positions are updated using Spherical Linear Interpolation (SLERP) to ensure sub-prototypes evolve along the manifold surface rather than drifting into the hypersphere's interior.
Dual-View Inference: The final decision score combines a Global View (cosine similarity to the class mean $\mu_c$ ) and a Local View (maximum cosine similarity to any local sub-prototype $v \in V_c$ ). The score is a weighted sum:
$S(x, c) = \alpha \cdot \cos\langle \tilde{f}_\theta(x), \mu_c \rangle + (1-\alpha) \cdot \max_{v \in V_c} \cos\langle \tilde{f}_\theta(x), v \rangle$

2.2. STAR (Structure-Topology Alignment via Residuals)

To address the non-linear feature drift that static topological methods cannot fully withstand, STAR introduces a "drift adaptation" paradigm.

Pointwise Trajectory Tracking: STAR establishes a one-to-one mapping between topological nodes and anchor samples. It tracks the drift of each node individually by comparing current features against stored reference features.
Residual Alignment: An Exponential Moving Average (EMA) smooths the drift vectors ( $\delta_i$ ). These residuals are applied to transport the sub-prototypes in the current feature space, allowing the manifold structure to actively deform and align with the evolving feature distribution without retraining the backbone.
Global Synchronization: The global class mean is updated synchronously using the average drift of all nodes.

3. Key Contributions

Revisiting NCM Optimality: The authors use Procrustes analysis to empirically demonstrate that feature drift in CIL causes significant non-linear structural deformations, exposing the limitations of single-prototype methods in practical scenarios.
HC-SOINN Classifier: A novel topology-aware classifier that represents class manifolds using a hierarchical structure of local centers and a global center, offering a more accurate geometric representation than single prototypes.
STAR Mechanism: A shift from "drift resistance" to "drift adaptation." By leveraging pointwise trajectory tracking, STAR enables the learned topology to actively deform to match complex non-linear feature evolution.
Extensive Validation: The approach is integrated into seven state-of-the-art CIL methods (including prompt-based, adapter-based, and representation-based paradigms) across three benchmarks, demonstrating consistent improvements.

4. Experimental Results

The method was evaluated on Split CIFAR-100, Split CUB-200, and Split ImageNet-R.

Performance Gains: Integrating HC-SOINN (with or without STAR) into seven baselines (e.g., SimpleCIL, DualPrompt, CODA-Prompt, SEMA, CL-LoRA) resulted in consistent improvements in both Average Accuracy (AAvg) and Last-Task Accuracy (ALast).
- Notable gains include SEMA + HC-SOINN achieving a +21.00% improvement on Split CUB-200 (ALast) and CODA-Prompt + HC-SOINN surpassing other SOTA methods on Split CUB-200.
Classifier Analysis: When isolating the classifier contribution (fixing the backbone), the proposed HC-SOINN + STAR consistently outperformed parametric FC layers, standard NCM, and the non-linear Kolmogorov-Arnold Classifier (KAC), particularly in long-sequence and out-of-distribution tasks.
Efficiency:
- Training: The full method adds only ~3.96% overhead compared to the baseline, achieved via class-wise parallelization.
- Inference: The method introduces virtually zero latency overhead (+0.01%) because the backbone (99.7% of compute time) dominates the inference cost, and the topological distance calculations are vectorized on GPU.
Storage: The adaptive topology requires storing an average of ~18–21 nodes per class, which is comparable to or lower than the 20-exemplar buffers used in traditional rehearsal-based methods, yet without the need for gradient-based retraining.

5. Significance and Claims

The paper claims that the reliance on single-prototype classifiers in CIL is theoretically unsound when Neural Collapse is incomplete. By modeling classes as complex manifolds rather than points, the proposed framework bridges the gap between theoretical optimality and practical CIL constraints.

Paradigm Shift: The work advocates for shifting from "drift resistance" (trying to prevent feature shifts) to "drift adaptation" (actively deforming the decision boundary to match the shift).
Robustness: The method is presented as "tuning-free," maintaining high performance across diverse datasets and backbones with a single unified set of hyperparameters.
Limitations: The authors acknowledge that the performance gains come with moderate computational overhead for topological maintenance and additional storage for anchor buffers. Furthermore, the current scoring mechanism relies solely on node distances, leaving the rich edge connectivity of the learned graph unexploited during classification.

The authors conclude that their topological framework provides a more reliable solution for complex incremental shifts than existing linear or fixed non-linear boundaries, establishing new state-of-the-art performance across evaluated datasets.