Technical Summary: Prototype Latent World Model Replay for Class-Incremental Learning

1. Problem Statement

Class-Incremental Learning (CIL) requires a model to learn new categories over time without access to raw data from previous tasks. The primary challenge is catastrophic forgetting, where learning new classes overwrites the decision regions and feature representations of old classes. Existing solutions face specific limitations:

• Regularization-based methods constrain parameter changes but do not explicitly preserve old-class feature distributions.
• Rehearsal-based methods store raw exemplars, which is often constrained by storage limits, privacy, or deployment scenarios where old data cannot be retained.
• Generative replay methods attempt to synthesize old images but can be computationally costly and unstable, often producing artifacts that harm classifier training.

The paper posits a central question: Can a model remember old classes without storing or regenerating old images? The authors argue that while raw images are unavailable, old classes can still be represented by their semantic states within a stable latent space.

2. Methodology: Prototype Latent World Model Replay

The proposed framework, Prototype Latent World Model Replay, decouples the learning system into two distinct spaces to preserve old knowledge while adapting to new tasks.

2.1 Core Architecture

1. Stable Latent State Space ($\mathcal{H}$):
   • A frozen, ImageNet-pretrained encoder ($E$) maps input images to a latent state space $\mathcal{H}$.
   • The encoder is frozen during incremental learning to ensure the coordinate system remains fixed. This prevents the semantic meaning of stored prototypes from drifting as new tasks are learned.
2. Adaptive Classification Space ($\mathcal{Z}$):
   • A lightweight, trainable adapter ($A$) maps the stable latent states $h \in \mathcal{H}$ to a task-adaptive feature space $z \in \mathcal{Z}$.
   • A classifier ($G$) operates on $z$ to produce logits for all observed classes.
   • Only the adapter and classifier are updated during incremental training, allowing plasticity for new classes while keeping the underlying "world memory" stable.

2.2 Prototype Latent World Memory

Instead of storing raw images, the method stores old classes as distributions over latent states.

• Multi-Prototype Representation: For each class $c$, the latent states $\{h_i\}$ are clustered into $K$ groups to capture intra-class variations (e.g., different poses or backgrounds).
• Distribution Modeling: Each cluster $k$ is summarized by a mean $\mu_{c,k}$ and a diagonal variance $\sigma^2_{c,k}$.
• Memory Structure: The memory for a class is a mixture of these prototype-centered distributions:
  $$p(h | c) = \frac{1}{K} \sum_{k=1}^{K} \mathcal{N}(h; \mu_{c,k}, \text{diag}(\sigma^2_{c,k}))$$
  This constitutes a "latent world model" that describes possible semantic states for a class without reconstructing pixels.

2.3 Training Procedure

When a new task $t$ arrives:

1. Latent Replay: For each old class, the model samples synthetic latent states $\tilde{h}_{old}$ from the stored prototype distributions. These states are fed directly into the adapter and classifier (bypassing the frozen encoder).
2. New Class Learning: Real images from the current task are processed through the frozen encoder and adapter.
3. Objective Function: The total loss combines three terms:
   • $L_{new}$: Standard cross-entropy for new classes.
   • $L_{replay}$: Cross-entropy for the sampled old latent states, ensuring the classifier retains old-class signals.
   • $L_{con}$: A supervised contrastive loss applied in the adapter space $\mathcal{Z}$. This term pulls same-class features (both real new and replayed old) together and pushes different-class features apart, explicitly enforcing geometric separation between old and new classes.

3. Key Contributions

1. Latent World Model Framework: A memory-free CIL framework that preserves old classes as distributions over stable pretrained latent states rather than raw images.
2. Prototype-Based Class Memory: A mechanism representing each class with multiple latent prototypes and variances, capturing multi-modal intra-class variation, combined with latent replay and supervised contrastive separation.
3. Empirical Validation: Comprehensive evaluation on Split CIFAR-100 demonstrating that stable latent-state replay is the primary driver of performance gains, with contrastive separation refining the decision geometry.

4. Experimental Results

The method was evaluated on Split CIFAR-100 under three incremental protocols: Inc5 (5 new classes per stage), Inc10, and Inc20.

• Performance vs. Fine-Tuning: The full model (Ours-LWM+Con) significantly outperforms standard fine-tuning without storing raw exemplars.
  • Inc5: Last Accuracy improved from 4.55% (Fine-tuning) to 31.64%.
  • Inc10: Last Accuracy improved from 9.06% to 37.06%.
  • Inc20: Last Accuracy improved from 16.96% to 43.10%.
• Comparison with Baselines:
  • The method outperforms memory-free baselines (e.g., LwF, FeTrIL, SimpleCIL) and is competitive with strong exemplar-based methods (e.g., DER, iCaRL) despite storing zero raw images.
  • It achieves Average Accuracies (AvgAcc) of 45.86%, 52.19%, and 56.18% for Inc5, Inc10, and Inc20, respectively.
• Ablation Studies:
  • Stable H-Space: The most significant gain comes from the stable latent state space. Removing it (reverting to standard fine-tuning) causes Last Accuracy to drop to single digits.
  • Contrastive Loss: Adding the supervised contrastive term provides a marginal but consistent improvement in separating old and new decision boundaries (e.g., +0.12% in Inc10 LastAcc), refining the geometry rather than driving the primary gain.

5. Significance and Claims

The paper claims that catastrophic forgetting can be mitigated by preserving class distributions in a stable latent space rather than storing raw data or relying solely on mutable weights.

• Shift in Memory Paradigm: The work suggests a shift from "exemplar-based" continual learning (storing a subset of past observations) to "state-distribution-based" continual learning (storing a compact model of where a class lives in semantic space).
• Behavioral Evidence: Interpretability analyses (t-SNE visualizations and task-retention heatmaps) show that while fine-tuning collapses old task accuracy to near zero, the proposed method maintains non-zero accuracy on old tasks. This indicates that the stored latent distributions remain queryable and meaningful even after many new tasks are learned.
• Modest Scope: The authors acknowledge that the method relies on the quality of the frozen pretrained encoder. If the encoder fails to separate semantically similar classes, the latent distributions may overlap, limiting performance. The current implementation uses simple Gaussian sampling, leaving room for future exploration with more expressive density estimators or self-supervised objectives.

In conclusion, the paper demonstrates that a Prototype Latent World Model can serve as a compact, reusable memory for continual recognition, effectively bridging pretrained representation learning, latent replay, and class-incremental recognition without the need for raw exemplar storage.