Technical Summary: BP-TTA: Balanced and Prototype-Guided Test-Time Adaptation in Dynamic Scenarios

1. Problem Definition

The paper addresses the limitations of existing Test-Time Adaptation (TTA) methods when deployed in real-world dynamic scenarios. While traditional TTA assumes test data is independently and identically distributed (i.i.d.) from a single target domain, practical applications involve Practical Test-Time Adaptation (PTTA) settings characterized by two simultaneous challenges:

1. Continual Domain Shifts: The test distribution evolves continuously over time due to changing environmental conditions (e.g., lighting, weather, sensor noise).
2. Class Imbalance in Streaming Data: Due to temporal correlations in real-world streams (e.g., a vehicle on a highway seeing mostly cars, then entering a residential area with pedestrians), short-time windows often exhibit severe class imbalance.

Existing methods often fail in this setting because:

• Class Imbalance: Gradient updates become dominated by majority classes, causing the model to overfit short-term patterns and neglect minority classes.
• Continuous Shifts: Reliance on pseudo-labels or entropy minimization becomes unstable as the distribution drifts, leading to noisy supervision and catastrophic forgetting.

2. Methodology: BP-TTA

The authors propose BP-TTA (Balanced and Prototype-Guided Test-Time Adaptation), a framework that integrates Batch-Balanced Sampling (BBS) and Category Prototype-Guided Adaptation (CPGA) within a Mean Teacher architecture (student and teacher networks).

A. Batch-Balanced Sampling (BBS)

To mitigate class imbalance in streaming data, BBS dynamically reconstructs balanced training batches.

• Memory Bank: Maintains a class-aware memory bank $\mathcal{M}$ storing samples with attributes: pseudo-label ($\hat{y}_i$), age ($a_i$), confidence ($\gamma_i$), and uncertainty ($u_i$).
• Selection Mechanism: Uses a selection score $H_i = \exp(a_i/K) + u_i$ to manage the memory bank, prioritizing recent and uncertain samples while removing redundant ones.
• Batch Reconstruction: For each incoming batch, the method:
  • Downsamples majority classes by selecting the top-$N$ highest-confidence samples.
  • Supplements minority classes by retrieving high-confidence samples from the memory bank.
  • Replicates samples if necessary to ensure a fixed number of samples per class ($N_{per-class}$).
• Differentiation: Unlike prior work (e.g., RoTTA) that relies solely on memory bank samples, BBS operates directly on the incoming batch, ensuring the balanced batch reflects the current target distribution while correcting for imbalance.

B. Category Prototype-Guided Adaptation (CPGA)

To stabilize adaptation under continuous domain shifts, CPGA leverages class prototypes to guide feature learning.

• Prototype Initialization & Update: Class prototypes $\{p_c\}$ are initialized from the source model's classifier weights. They are updated via an Exponential Moving Average (EMA) mechanism using only high-confidence samples ($\gamma_i \geq \tau$):
  $$p_c^{(t+1)} = \alpha p_c^{(t)} + (1-\alpha)z_i \quad \text{if } \hat{y}_i = c \text{ and } \gamma_i \geq \tau$$
• Prototype Loss: A supervision loss is introduced to minimize the distance between sample features and their corresponding class prototypes, encouraging intra-class clustering and inter-class separation:
  $$\mathcal{L}_{proto} = \frac{1}{|S|} \sum_{i \in S} (1 - \text{sim}(z_i, p_{\hat{y}_i}))$$
  where $S$ is the set of high-confidence samples.

C. Overall Optimization

The total loss function combines standard consistency loss (between student and teacher under augmentation) and the prototype loss:
$$\mathcal{L}_{total} = \mathcal{L}_{sup} + \lambda \mathcal{L}_{proto}$$
The student model is updated via gradient descent, while the teacher model is updated via EMA.

3. Key Contributions

1. Novel Framework: Proposes BP-TTA, the first method to jointly address class imbalance and continual domain shifts in PTTA settings by combining batch-balanced sampling with prototype-guided adaptation.
2. Batch-Balanced Sampling (BBS): Introduces a mechanism to dynamically reconstruct balanced batches from imbalanced streams using a class-aware memory bank, ensuring reliable pseudo-labels for all classes.
3. Category Prototype-Guided Adaptation (CPGA): Develops a method to maintain evolving class prototypes during inference, using prototype similarity as a constraint to improve pseudo-label reliability and feature space stability.
4. Comprehensive Evaluation: Demonstrates superior performance over state-of-the-art TTA methods (including TENT, CoTTA, EATA, SAR, and RoTTA) across three datasets (CIFAR10-C, CIFAR100-C, ImageNet-C) and the DomainNet-126 dataset under PTTA conditions.

4. Experimental Results

The paper reports extensive experiments validating the effectiveness of BP-TTA:

• Performance: BP-TTA consistently outperforms all baselines. On CIFAR10-C, CIFAR100-C, and ImageNet-C, it improved average accuracy by margins of 3.08%, 1.84%, and 1.59% respectively over the second-best methods.
• Ablation Studies:
  • Component Efficacy: Using BBS alone significantly reduces error rates compared to the source model. CPGA alone struggles under severe imbalance due to noisy prototype updates. The combination (BBS+CPGA) yields the best results.
  • Hyperparameters: The method is robust to variations in confidence threshold $\tau$ and smoothing factor $\alpha$. A weighting factor $\lambda=10$ for the prototype loss was found optimal.
  • Batch Size: Moderate batch sizes yield the best balance between stability and response speed.
• Robustness:
  • Domain Shift Frequency: BP-TTA degrades more slowly than RoTTA as the frequency of domain changes increases.
  • Imbalance Severity: The method maintains stable performance even as the Dirichlet concentration parameter $\delta$ decreases (indicating higher imbalance).
  • Sequence Order: Unlike DA-TTA, which shows high variance across different domain shift sequences, BP-TTA maintains consistent performance regardless of the order of shifts.
  • Model Architecture: The method is effective across various backbone networks (ResNet, WideResNet, ResNeXt) without significant performance drops.
• Visualization: t-SNE visualizations confirm that CPGA produces more compact intra-class clusters and better inter-class separation compared to baseline prototype methods like T3A.

5. Significance and Claims

The paper claims that BP-TTA provides a robust solution for Practical Test-Time Adaptation, a setting that more accurately reflects real-world deployment challenges than previous static or purely continual adaptation paradigms.

• Addressing Real-World Constraints: By explicitly handling the interaction between class imbalance and domain shifts, the method prevents the "catastrophic failures" often seen when models are exposed to streaming data with temporal correlations.
• Stability: The integration of prototype guidance stabilizes the learning process, preventing the model from overfitting to dominant classes or being misled by noisy pseudo-labels during distribution drifts.
• State-of-the-Art: The authors assert that their method achieves state-of-the-art performance in dynamic test-time streaming settings, offering a flexible and practical solution for improving model robustness in non-stationary environments.

The work is supported by the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China and the Natural Science Foundation of Sichuan Province.