Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models

1. Problem Definition

The paper addresses Test-Time Adaptation (TTA) in the strict black-box setting.

• Context: Modern deep learning models are increasingly deployed as opaque APIs (e.g., commercial vision or language models) where users can only submit inputs and receive output probabilities.
• Constraints:
  • No Internal Access: No access to model parameters, gradients, intermediate features, or pre-training data.
  • Unsupervised: Adaptation must occur on unlabeled target data streams.
  • Cost & Latency: Each API call incurs monetary cost and network latency. Efficient methods must minimize queries (ideally one per sample) and avoid high latency.
• Limitations of Existing Methods:
  • Output Refinement (e.g., LAME): Limited adaptive capacity as they only modify final predictions without changing the input.
  • Input Modification (e.g., ZOO-based prompting): While powerful, Zeroth-Order Optimization (ZOO) requires hundreds of API calls per sample (high cost) and suffers from catastrophic instability in unsupervised settings (e.g., learning degenerate prompts that corrupt semantic content).
  • Gray-Box Methods: Require access to internal tokens or features, which is unavailable in strict black-box APIs.

2. Methodology: BETA Framework

The authors propose BETA (Black-box Efficient Test-time Adaptation), a framework that achieves stable adaptation using a local, lightweight white-box steering model to guide a powerful, inaccessible black-box target model.

Core Components:

1. Dual-Model Architecture:

   • Target Model ($f_B$): The powerful black-box API (frozen, no gradients).
   • Steering Model ($f_S$): A lightweight, fully accessible local model (e.g., ViT-Small) initialized from public checkpoints. It serves as a "gradient pathway."

2. Visual Prompt Learning:

   • Instead of updating weights, BETA learns an additive visual prompt $\delta$ added to the input image ($x' = x + \delta$).
   • The goal is to optimize $\delta$ to improve the target model's predictions without accessing its gradients.

3. Prediction Harmonization (The Core Innovation):

   • Problem: Naively transferring gradients from $f_S$ to $f_B$ fails because gradient similarity between different architectures is near zero. ZOO is too expensive and unstable.
   • Solution: BETA defines a harmonized prediction $p_H$ that fuses the outputs of both models:
     $$p_H(x') = \alpha \cdot p_S(x') + (1 - \alpha) \cdot p_B(x')$$
   • Optimization: The system minimizes the entropy of this harmonized distribution. Crucially, gradients are computed only through the steering model $f_S$.
   • Analytical Justification: The entropy of the mixture decomposes into the entropy of the steering model plus a Jensen-Shannon (JS) divergence term between the two models.
     • Minimizing $H(p_H)$ effectively minimizes the entropy of $p_S$ while the JS term acts as a constraint, anchoring the optimization direction to the black-box model's beliefs ($p_B$). This prevents the prompt from driving the steering model to be confident on classes the target model rejects.

4. Stabilization Mechanisms:
   To prevent the instability common in unsupervised prompt learning (e.g., prompt collapse), BETA employs:

   • Prompt Learning-Oriented Data Filtering: Updates are only performed on samples where the steering model's prediction entropy is below a threshold $\epsilon$. This filters out noisy, high-entropy samples.
   • Consistency Regularization: A KL-divergence loss is added to ensure the prompted image $x'$ yields predictions consistent with the clean image $x$ on the steering model. This anchors the prompt to the model's pre-trained knowledge, preventing degenerate solutions.

5. Joint Optimization:
   The final objective combines the harmonization loss, a steering loss (updating normalization layers of $f_S$), and the consistency regularization. The process is strictly online, performing a single gradient step per batch.

3. Key Contributions

1. First Systematic Study of Strict Black-Box TTA: The paper rigorously defines the strict black-box setting (API-only access) and exposes the limitations of existing post-hoc, augmentation, and ZOO-based approaches.
2. Novel Framework (BETA): Introduces a method that bypasses expensive API queries by using a local steering model with Prediction Harmonization. This creates a tractable gradient pathway without needing internal access to the target.
3. Theoretical Grounding: Provides an analytical derivation showing that the harmonized objective includes a JS divergence term that aligns the local gradient with the target's belief, explaining why this works despite architectural differences.
4. Stability Innovations: Combines consistency regularization and entropy-based filtering to solve the inherent instability of unsupervised prompt learning.

4. Experimental Results

The authors evaluated BETA on ImageNet-C (corruptions), ImageNet-S (Sketch), ImageNet-R (Rendition), and a commercial Clarifai API.

• Performance on Vision Models (ViT-B/16 & ViT-L/16):
  • On ImageNet-C (ViT-B/16), BETA achieved 62.6% accuracy (+7.1% gain over source), outperforming strong white-box methods like TENT (+4.1%) and CoTTA (+6.1%).
  • It significantly outperformed other black-box methods: ZOO variants collapsed on difficult corruptions (e.g., Contrast dropped to 4.1% with ZOO vs. 53.4% with BETA).
• Performance on Vision-Language Models (CLIP):
  • BETA achieved 63.4% accuracy on CLIP, surpassing specialized white-box VLM methods like TPT and DynaPrompt.
• Real-World API Efficiency (Clarifai):
  • Cost: BETA achieved comparable performance to ZOO at 250× lower cost ($0.4 vs. >$100).
  • Latency: BETA matches standard inference latency (1 API call) while ZOO methods require 16+ calls, causing severe slowdowns.
• Generalization:
  • BETA worked effectively on EuroSAT (satellite imagery) and Derm7pt (medical skin lesions) using zero-shot steering models, demonstrating robustness even when the steering model is not a perfect semantic match for the target domain.
• Robustness: The method remained stable under label imbalance, continual domain shifts, and small batch sizes.

5. Significance and Impact

• Practical Viability: BETA is the first method to make TTA practical for commercial API-based models. It solves the "query cost vs. adaptation quality" trade-off that previously made black-box TTA infeasible.
• Paradigm Shift: It demonstrates that a smaller, local model can effectively "steer" a massive, inaccessible model without modifying its weights, opening new possibilities for adapting proprietary AI systems.
• Future Applications: The authors suggest this "advisor" architecture (local steering + remote execution) is a prototype for adapting agent stacks and multi-component AI systems where only remote APIs are available.

In summary, BETA successfully bridges the gap between the theoretical need for test-time adaptation and the practical constraints of modern black-box AI deployment, offering a solution that is accurate, stable, and economically viable.