Multi-Level Knowledge Distillation and Dynamic Self-Supervised Learning for Continual Learning

1. Problem Statement

The paper addresses Class-Incremental Learning with Repetition (CIR), a more realistic and challenging scenario than traditional Class-Incremental Learning (CIL).

• The Challenge: In CIR, the model encounters a mixture of new classes and previously seen classes across sequential tasks. Unlike standard CIL, where classes are strictly new, CIR assumes the model must handle "repeated" exposure to old classes while learning new ones.
• Constraints: The setting is rehearsal-free regarding labeled data. The model cannot store large amounts of past labeled data due to privacy and storage constraints. Instead, it is provided with abundant unlabeled external data (e.g., from the internet) alongside small batches of new labeled data.
• Goal: Maximize final classification accuracy on the test set while maintaining stability (preventing catastrophic forgetting of old classes) and plasticity (rapidly learning new concepts) without relying on stored exemplars of past tasks.

2. Methodology

The authors propose a framework built upon a baseline of local cross-entropy and feature replay, enhanced by two core components designed to leverage unlabeled data effectively.

A. Multi-Level Knowledge Distillation (MLKD)

To ensure stability and mitigate catastrophic forgetting, the authors introduce a sophisticated distillation mechanism that extracts knowledge from multiple previous models across different perspectives.

• Multi-Model Approach: Instead of distilling from a single previous model, the system maintains multiple previous models. It selects features and logits based on output confidence to create a robust ensemble of knowledge.
• Exponential Moving Average (EMA): All previous models are gradually updated via EMA with the current model to reflect new knowledge while preserving historical stability.
• Multi-Level Distillation:
  • Feature-Level KD: Uses L2 loss to distill features from previous models. The weight of this loss increases linearly over tasks, as early-stage features may lack sufficient information for later tasks.
  • Logit-Level KD (Correlation-Based): Standard instance-wise logit distillation fails due to distribution shifts between training data and unlabeled external data. To solve this, the authors use Gram matrices to distill correlations:
    • Instance Correlation: Captures relationships between samples within a batch ($G = l \cdot l^T$).
    • Class Correlation: Captures relationships between classes ($M = l^T \cdot l$).
      This approach ensures the model learns the structural relationships of the data rather than just specific instance predictions, compensating for data distribution shifts.

B. Dynamic Self-Supervised Learning (Dynamic SSL)

To enhance plasticity and accelerate the learning of new classes, the authors utilize the unlabeled dataset via self-supervised learning.

• Mechanism: A self-supervised loss (specifically Rotation Prediction) is applied to the unlabeled data to encourage the extraction of general, task-agnostic features.
• Dynamic Weighting: A naive application of SSL can degrade performance on the primary classification task. To balance this, the authors introduce a dynamic weighting factor $\alpha = c \cdot \omega^t$ (where $t$ is the task number and $\omega < 1$).
  • As the number of tasks increases, the weight of the SSL loss decreases.
  • This ensures the model focuses heavily on the primary classification task in later stages while leveraging SSL for generalization in early stages.

C. Final Loss Function

The total loss combines several components:

1. Asynchronous Cross-Entropy ($L_{ACE}$): Computed only on classes present in the current batch.
2. Dynamic SSL Loss ($L_{SSL}$): Applied to unlabeled data with dynamic weighting.
3. Logit Constraint Loss ($L_{LC}$): From XDER, to help discriminate classes from different tasks.
4. Feature Replay ($L_{der}$): Uses a small memory buffer for feature replay.
5. Feature KD ($L_{featureKD}$) & Logit KD ($L_{logitKD}$): The proposed multi-level distillation losses.

3. Key Contributions

1. MLKD Framework: A novel distillation strategy that combines multiple previous models, EMA updates, and correlation-based logit distillation (using Gram matrices) to handle distribution shifts in unlabeled data.
2. Dynamic SSL Strategy: A method to integrate self-supervised learning into continual learning with a dynamic weighting scheme that balances plasticity and stability over time.
3. Rehearsal-Free CIR Solution: Demonstrates that high performance in CIR can be achieved without storing labeled exemplars, relying instead on external unlabeled data and efficient knowledge transfer.
4. Challenge Performance: Achieved 2nd place in the CVPR 2024 5th CLVISION Challenge.

4. Experimental Results

The experiments were conducted on a subset of ImageNet-1K (130 classes: birds, sea animals, bugs) with a stream of 50 tasks. The model was ResNet-18 with strict memory (200 exemplars) and time constraints.

• Performance Comparison:
  • Fine-tuning: 5.82% (Severe catastrophic forgetting).
  • Baseline: 19.54%.
  • Baseline + Dynamic SSL: 23.82%.
  • Baseline + MLKD: 39.82%.
  • Full Method (Ours): 42.00%.
• Ablation Studies:
  • MLKD Components: Adding EMA, Correlation-based Logit KD (CLKD), and Multiple Previous Models (MPM) progressively improved accuracy from 39.40% (Feature KD only) to 42.00%.
  • Number of Models: Using 3 previous models yielded the optimal balance (42.00%); using 4 models slightly decreased performance due to redundancy.
  • Dynamic Weighting: Applying dynamic weighting to SSL improved accuracy from 40.76% (static weight) to 42.00%.
  • SSL Choice: Simple Rotation Prediction outperformed complex contrastive methods (SimCLR, SimSiam, VICReg) likely due to the challenge's constraints on batch size and training time.

5. Significance

This work is significant for the field of Continual Learning because:

• Real-World Applicability: It addresses the CIR scenario, which better reflects real-world data streams where old and new classes mix, compared to the idealized disjoint class assumption of traditional CIL.
• Privacy and Storage Efficiency: By proving that unlabeled external data can replace stored labeled exemplars for knowledge retention, the method offers a viable solution for privacy-sensitive environments where storing past user data is prohibited.
• Robustness to Distribution Shift: The use of Gram matrices for logit distillation provides a robust mechanism to transfer knowledge even when the unlabeled data distribution differs significantly from the training distribution.
• State-of-the-Art Performance: The proposed method sets a new benchmark for rehearsal-free continual learning, demonstrating that careful integration of self-supervision and multi-level distillation can significantly outperform standard baselines.