HyCal: A Training-Free Prototype Calibration Method for Cross-Discipline Few-Shot Class-Incremental Learning

1. Problem Definition

The paper addresses a critical gap in Few-Shot Class-Incremental Learning (FSCIL) when applied to real-world, heterogeneous scenarios.

• The Limitation of Existing Methods: Standard FSCIL and cross-domain FSCIL methods typically assume homogeneous visual domains and balanced data distributions. They often fail when data comes from diverse disciplines (e.g., medical imaging vs. textures) with varying visual complexity and sample availability.
• The Core Phenomenon: Domain Gravity: The authors identify a structural bias termed Domain Gravity. This is a representational asymmetry where overrepresented or low-entropy domains (data-rich or visually coherent) disproportionately influence the shared embedding space of Pretrained Vision-Language Models (VLMs) like CLIP.
  • Consequence: This causes "prototype drift," where the model's decision boundaries shift toward dominant domains, leading to catastrophic forgetting and degraded performance on underrepresented or high-entropy domains (e.g., medical images or rare textures).
• The Proposed Setting: XD-VSCIL: To rigorously evaluate this, the authors introduce Cross-Discipline Variable Few-Shot Class-Incremental Learning (XD-VSCIL). Unlike standard settings, XD-VSCIL involves sequential tasks from distinct academic disciplines with:
  • Variable class counts per task.
  • Variable sample counts per class (highly imbalanced).
  • Significant distribution shifts between tasks.

2. Methodology: HYCAL

The authors propose HYCAL (Hybrid Prototype Calibration), a training-free method that operates on frozen CLIP embeddings. It requires no backbone fine-tuning, no additional parameters, and no model expansion.

Core Mechanism

HYCAL calibrates class prototypes by fusing two complementary geometric metrics:

1. Cosine Similarity: Captures directional alignment (global orientation). It is robust to magnitude shifts and provides stable global structure.
2. Mahalanobis Distance: Captures covariance-aware magnitude. It accounts for the specific variance and correlation structure of the data within a class, correcting for domain-specific entropy and scale differences.

Theoretical Basis

• Complementarity: The paper proves (Theorem 1 & 2) that under isotropic conditions, Cosine Similarity depends on the direction of the difference vector, while Mahalanobis distance depends on the magnitude (norm). Since these are statistically independent, combining them yields strictly more mutual information regarding the class label than either metric alone.
• Anisotropy in Real VLMs: In real-world VLMs, embeddings are anisotropic. The authors argue that this anisotropy actually strengthens the complementarity, as domain shifts distort direction and magnitude differently, making the fusion of both cues essential for robustness.

Algorithmic Workflow

1. Prototype Construction: For each new class $c$ in a few-shot setting, HYCAL computes a fused embedding (sum of image and text embeddings from CLIP).
   • It calculates the mean embedding ($\mu_c$).
   • It estimates the empirical covariance ($\Sigma_c$) and applies regularization ($\Sigma^{reg}_c = (1-\lambda)\Sigma_c + \lambda \gamma I$) to ensure stability in few-shot regimes.
2. Hybrid Distance Calibration: During inference, for a test sample $z_{test}$ and a class prototype $P_c = \{\mu_c, \Sigma_c^{-1}\}$, HYCAL computes:
   • $d_{maha}$: Mahalanobis distance (normalized).
   • $s_{cos}$: Cosine similarity.
3. Dynamic Weighted Summation: The final prediction is determined by adaptively weighting these two metrics based on the number of samples ($K_c$) available for that class:
   $$c_{pred} = \arg \max_c \left[ w_c \cdot d_{maha} + (1-w_c) \cdot s_{cos} \right]$$
   Where $w_c$ is a sigmoid function of $K_c$. This allows the model to rely more on covariance (Mahalanobis) when sufficient data exists to estimate it reliably, and shift toward directional alignment (Cosine) in extremely sparse regimes.

3. Key Contributions

1. Identification of Domain Gravity: The formal definition of a new structural bias in VLM-based continual learning where domain imbalance causes representational drift, explaining why existing methods fail in heterogeneous settings.
2. XD-VSCIL Benchmark: The introduction of a new benchmark setting that reflects real-world heterogeneity (8 diverse datasets: Aircraft, ArtBench, DTD, EuroSAT, Galaxy, MNIST, OrganMNIST, Flowers) with variable shot counts and severe class/domain imbalance.
3. HYCAL Method: A training-free, parameter-free calibration method that fuses Cosine and Mahalanobis distances. It achieves robustness without modifying the frozen backbone.
4. New Metric (SCDE): The proposal of Cross-Discipline Efficiency (CDE) score, which holistically evaluates zero-shot generalization, plasticity, forgetting mitigation, and data efficiency, weighted by data scarcity.

4. Experimental Results

The authors evaluated HYCAL against state-of-the-art training-free methods (FeCAM, RanPAC, KLDA) and a trainable baseline (Primal-RAIL) across three XD-VSCIL settings: Balanced-in-Class, Cross-Scale Imbalance, and High-Scale Domain Imbalance.

• Performance in High-Scale Imbalance: HYCAL significantly outperformed all baselines.
  • It achieved an Average Accuracy of 54.48% (vs. 53.49% for Primal-RAIL and 49.98% for RanPAC).
  • It achieved a Last Accuracy of 63.50%, showing superior retention of knowledge across diverse domains.
  • It obtained the highest SCDE score (53.70), indicating the best balance between adapting to new domains and retaining old knowledge.
• Robustness to Domain Order: Experiments with random domain sequences showed HYCAL maintains consistent performance, proving it is not sensitive to the order in which domains are introduced.
• Ablation Studies:
  • Metric Fusion: The dynamic combination of Cosine and Mahalanobis consistently outperformed using either metric alone, especially in imbalanced settings.
  • Fusion Strategy: Sum-based fusion of image/text embeddings outperformed concatenation.
  • Efficiency: HYCAL requires only 5.4 GFLOPs to add 10 new classes, compared to 12 GFLOPs for Primal-RAIL, and scales linearly with the number of classes.

5. Significance

• Real-World Applicability: By addressing "Domain Gravity," HYCAL makes VLM-based continual learning viable for practical applications where data is inherently imbalanced and comes from disparate fields (e.g., a medical AI system learning new diseases while retaining knowledge of general textures).
• Efficiency: As a training-free method, it eliminates the computational cost and risk of catastrophic forgetting associated with fine-tuning large foundation models. It is highly scalable for edge devices or scenarios with limited compute.
• Theoretical Insight: The paper provides a rigorous theoretical justification for why combining directional (Cosine) and distributional (Mahalanobis) cues is necessary for robust classification in anisotropic embedding spaces, moving beyond the assumption of isotropic Gaussian distributions often made in prior work.

In summary, HYCAL offers a simple yet powerful solution to the "Domain Gravity" problem, enabling stable, efficient, and robust few-shot learning across highly heterogeneous and imbalanced data distributions without requiring model retraining.