Technical Summary: Remembering by Reconstructing

Problem Statement

The paper addresses Domain Incremental Learning (DIL), a specific challenge within Continual Learning (CL) where models must adapt to a sequence of evolving, non-stationary data domains without access to the domain identity at test time. While standard DIL approaches strive to prevent catastrophic forgetting (the loss of performance on previous domains when learning new ones), this work posits that in real-world streaming scenarios (e.g., video), data is highly correlated temporally, and domain shifts are often gradual.

The authors challenge the conventional view that forgetting must be strictly avoided. Instead, they argue that for streaming data, a model that specializes heavily on the current domain (inducing temporary forgetting of others) can be beneficial, provided it can rapidly recover ("remember") previous domains when they reappear. The core problem is how to leverage Test-Time Training (TTT) on streaming video to dynamically select the appropriate domain-specific knowledge without explicit domain labels.

Methodology

The proposed method, "Remembering by Reconstructing," utilizes a dual-headed architecture combined with Low-Rank Adaptation (LoRA) and Test-Time Training.

1. Model Architecture

The model consists of a shared backbone encoder $E$ and two parallel decoders:

• Main Task Head ($M$): Performs the downstream task (e.g., action recognition or semantic segmentation).
• Self-Supervised MAE Head ($D$): A Masked Autoencoder (MAE) branch trained to reconstruct masked input patches. This branch serves as a probe for domain adaptation.

2. Training Phase (Offline)

• Initialization: The model is initialized with a pre-trained MAE checkpoint. The encoder $E$ and MAE decoder $D$ are fine-tuned alongside the main task head $M$ on an initial task $T_0$.
• Domain-Specific LoRAs: For each subsequent domain $T_i$, the base parameters $\theta_0$ are frozen. New, trainable LoRA adapters ($\Delta_i$) are initialized for all linear layers in the model.
• Specialization: These LoRAs are trained exclusively on the new domain data using a combined loss of the main task and the MAE reconstruction task. This process naturally induces forgetting on other domains as the LoRA specializes, affecting both the main head and the MAE head.

3. Inference Phase (Online Test-Time Training)

At test time, the model encounters a continuous stream of data. The goal is to determine which LoRA (or combination of LoRAs) best matches the current input.

• Weighted Combination: Instead of selecting a single LoRA or naively merging them, the model maintains a weighted sum of all $N$ trained LoRAs:
  $$\theta_{TT} = \theta_0 + \sum_{k=1}^{N} c_k \Delta_k$$
  where $c = \{c_1, ..., c_N\}$ are learnable weighting coefficients.
• Reconstruction-Driven Adaptation: The coefficients $c$ are optimized online using the incoming data stream. The optimization objective is the MAE reconstruction loss on the current input (or a small batch of recent frames):
  $$c = \arg \min_c \mathcal{L}_{\text{MAE}}(F_{\text{MAE}}(x_{\text{mask}}), x)$$
  Because the MAE head also suffered from forgetting during the incremental training phase, minimizing the reconstruction loss on the current domain forces the model to "reverse" that forgetting, effectively steering the weights toward the LoRA specialized for the current domain.
• Softmax Selection: The optimized coefficients are passed through a softmax (with temperature $\tau$) before being applied to the main task head to ensure confident selection of the most relevant LoRA.
• Efficiency: Only the small set of coefficients $c$ is updated, not the full model. Updates occur periodically (e.g., every $n$ frames) using a sliding window buffer of recent frames, making the approach computationally efficient for streaming.

Key Contributions

1. Paradigm Shift on Forgetting: The paper proposes embracing forgetting as a mechanism for specialization, relying on TTT to recover lost knowledge dynamically, rather than attempting to prevent it entirely.
2. Dual-Head TTT for DIL: It introduces a novel application of TTT in a DIL setting, using the MAE reconstruction loss as a proxy to identify the current domain and adjust LoRA weights accordingly.
3. Streaming Adaptation: The method is specifically designed for video streams where consecutive frames are correlated, allowing for rapid adaptation to gradual domain shifts without needing to process every frame for updates.
4. New Benchmark: The authors introduce Domains-Campus, a new semantic segmentation dataset featuring egocentric video streams with gradual domain transitions and unseen intermediate environments, addressing the lack of such benchmarks in existing literature.

Experimental Results

The method was evaluated on two tasks:

• Video Action Recognition: Using the NTU RGB+D120 dataset with a domain-incremental setup involving different camera viewpoints and locations.
• Semantic Segmentation: Using the new Domains-Campus dataset and Places365 for pre-training.

Key Findings:

• Performance: The proposed method achieved the highest average accuracy (or Dice score) among all baselines, including Replay, DRIFT, and LoRA merging.
• Forgetting: It demonstrated significantly lower forgetting ($F$) compared to naive fine-tuning and competitive performance with state-of-the-art continual learning methods.
• Surpassing Joint Training: On the Domains-Campus dataset, the method outperformed a "Joint Fine-Tuning" baseline (where all tasks are trained simultaneously). The authors attribute this to the "expert" nature of individual LoRAs, which can outperform a single generalist model in their specific domains.
• Ablation Studies: The method proved robust to hyperparameter variations. It worked effectively with small batch sizes and update periods (e.g., updating every 16 frames), and surprisingly, even with a window size of 1 (updating on the current frame only).

Significance and Claims

The paper claims that its approach offers a more realistic perspective on continual learning for real-world applications. By acknowledging that data arrives as a continuous stream and that temporary forgetting can be beneficial for focusing on the present, the method leverages temporal continuity and locality to achieve rapid adaptation.

The authors emphasize that while the method is highly effective for streaming data, it introduces computational overhead during inference due to the TTT step. They suggest that future work could focus on reducing the number of LoRAs (via merging or pruning) or developing more efficient triggers for when updates are necessary, rather than performing periodic updates.

The work concludes that combining domain-specific parameter-efficient tuning (LoRA) with online self-supervised adaptation (TTT) provides a powerful framework for handling evolving, non-stationary data streams in video-based applications.