Technical Summary: Cross-Sample Relational Fusion (CORF)

1. Problem Formulation: Cross-Domain Class-Incremental Learning (CDCIL)

The paper addresses a critical gap in current continual learning research: the simultaneous challenge of Catastrophic Forgetting and Domain Shift. While standard Class-Incremental Learning (CIL) focuses on learning new classes without forgetting old ones, and Domain Generalization (DG) focuses on robustness across domains, neither fully addresses the scenario where a model must learn new classes sequentially while encountering data from both seen and unseen domains.

The authors formulate this as Cross-Domain Class-Incremental Learning (CDCIL). In this setting:

• Training: The model receives a stream of tasks, where each task contains samples from multiple "seen" domains.
• Inference: The model is evaluated on all previously learned classes across both the seen domains and a completely "unseen" domain that was excluded from training.
• Challenge: Existing CIL methods fail under domain shifts due to a lack of generalization mechanisms, while DG methods fail in incremental settings because they lack mechanisms to resist forgetting when the label space expands.

2. Methodology: Cross-Sample Relational Fusion (CORF)

The authors propose CORF, a unified framework designed to learn domain-agnostic, incrementally stable representations. CORF integrates two synergistic components: Dual-Sensitive Refinement (DSR) for data-level augmentation and Hierarchical Kernel-Based Distillation (HKD) for structural knowledge preservation.

A. Dual-Sensitive Refinement (DSR)

DSR aims to construct a domain-agnostic representation space by strategically synthesizing an auxiliary dataset. It leverages spatial contribution maps (via Grad-CAM) and predictive confidence to guide sample blending:

1. High-Confidence Blending:

   • Samples with high predictive confidence are identified as reliable semantic anchors.
   • For a high-confidence sample $x_H$, a counterpart $\tilde{x}_H$ from a different domain but the same class is retrieved.
   • A spatial contribution map identifies regions critical for classification. The model blends the low-contribution regions (often containing domain-specific noise) of $x_H$ and $\tilde{x}_H$, while preserving the high-contribution regions of $x_H$.
   • Goal: Force the model to rely on semantically consistent features across domains while ignoring domain-specific artifacts.

2. Low-Confidence Blending:

   • Samples with low confidence (ambiguous or noisy) are identified.
   • These are paired with a sample from a different class within the same batch.
   • The blending occurs in high-contribution regions to encourage feature disentanglement.
   • Goal: These synthesized samples are assigned to temporary auxiliary classes (disjoint from the original label space) to prevent corrupting the original classification boundaries while encouraging the model to learn robust decision boundaries.

B. Hierarchical Kernel-Based Distillation (HKD)

To mitigate catastrophic forgetting without relying on logit-level distillation (which assumes distributional consistency), CORF employs HKD. This method preserves the relational topology of samples across multiple feature hierarchies.

• Relational Kernel Maps: Instead of aligning global logits, HKD constructs kernel maps at multiple network layers that capture the conditional similarity between pairs of samples within a batch.
• Hybrid Kernel Design: The method uses a combination of Cosine kernels (to preserve angular/semantic direction) and Student-t kernels (to capture local neighborhood structures and handle outliers).
• Symmetric Divergence: The current model is trained to align its relational kernel distributions with those of the old model (from the previous task) using a symmetric divergence loss.
• Goal: This ensures that the fine-grained, domain-sensitive structural relationships between samples are preserved, even as the model adapts to new tasks and domains.

C. Unified Objective

The final training objective combines the standard classification loss on original and synthesized samples with the hierarchical distillation loss:
$$\min \sum_{(x,y) \in S_n \cup A_n} L_{cls}(h_\Psi(x), y) + L_{reg} + \beta \cdot L_{kernel}$$
Where $S_n$ is the original training set, $A_n$ is the auxiliary set generated by DSR, and $L_{kernel}$ is the aggregated distillation loss.

3. Key Contributions

1. Problem Formulation: The paper formally defines CDCIL, highlighting the limitations of existing CIL and DG methods when applied to scenarios involving both sequential class learning and domain shifts.
2. Dual-Sensitive Refinement (DSR): A novel data augmentation strategy that uses confidence and spatial contribution maps to synthesize domain-invariant samples, effectively decoupling semantic cues from domain biases.
3. Hierarchical Kernel-Based Distillation (HKD): A forgetting mitigation strategy that aligns relational structures across feature hierarchies using hybrid kernels, offering a more robust alternative to logit-based distillation under domain shift.
4. Unified Framework: Demonstrates that DSR and HKF can be seamlessly integrated into existing CIL algorithms (e.g., iCaRL, DER, Replay) to enhance their generalizability.

4. Experimental Results

The authors evaluated CORF on three benchmark datasets: OfficeHome, DomainNet, and PACS, using ResNet-34 as the backbone.

• Performance Gains: CORF consistently outperformed baseline methods (including FineTune, Replay, iCaRL, DER, MEMO, FOSTER) across all datasets.
  • On OfficeHome (5 tasks), integrating CORF with DER improved the Harmonic Mean (HM) of Seen and Unseen domains from 42.41% to 46.17%.
  • On PACS, the integration with FOSTER achieved an HM of 60.97%, surpassing the baseline by a significant margin.
• Ablation Studies:
  • Removing DSR led to a drop in generalization (Unseen Domain performance).
  • Removing HKD led to increased forgetting (Seen Domain performance degradation).
  • The combination of both yielded the best results, confirming their complementary nature.
• Hyperparameter Sensitivity: The method showed robustness to variations in the refinement weight ($\alpha$) and distillation coefficient ($\beta$), with optimal performance at $\alpha=0.3$ and $\beta=0.01$.
• Feature Analysis: Visualizations of pairwise similarity matrices showed that CORF produced more compact intra-class clusters and better-separated inter-class boundaries compared to baselines.
• Compatibility: The framework was successfully adapted to Transformer-based backbones (L2P and DualPrompt with ViT), though gains were more modest due to frozen backbone parameters, confirming its plug-and-play nature.

5. Significance and Claims

The paper claims that CORF provides a unified solution to the dual challenges of forgetting and domain shift, which are often treated separately in literature. By explicitly modeling the relational dependencies between samples across domains and tasks, CORF enables the construction of representations that are both domain-agnostic and incrementally stable.

The authors emphasize that their approach does not require storing large amounts of past data (beyond standard exemplar budgets) or complex generative models. Instead, it relies on selective refinement and structural alignment to enhance robustness. The results suggest that addressing domain shift and catastrophic forgetting simultaneously is not only feasible but necessary for real-world applications like autonomous driving, where environments and object categories evolve continuously.

The paper concludes that while CORF is currently most effective with trainable convolutional backbones, it offers a promising direction for future research in more diverse incremental learning scenarios and complex domain shifts.