🤖 machine learning

Audio-Visual Continual Test-Time Adaptation without Forgetting

This paper proposes AVReCAP, a novel method for audio-visual continual test-time adaptation that mitigates catastrophic forgetting by dynamically retrieving and adapting optimal modality fusion layer parameters from a buffer using only test data, thereby significantly improving performance across non-stationary domains without access to source data.

Original authors: Sarthak Kumar Maharana, Akshay Mehra, Bhavya Ramakrishna, Yunhui Guo, Guan-Ming Su

Published 2026-07-21

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Sarthak Kumar Maharana, Akshay Mehra, Bhavya Ramakrishna, Yunhui Guo, Guan-Ming Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant that can see and hear the world, designed to recognize everything from a barking dog to a siren. You trained this robot in a quiet, sunny studio, and it's perfect there. But the real world is messy. Suddenly, your robot is deployed on a rainy street, then a foggy mountain, then a crowded concert. The lighting changes, the sounds distort, and the robot starts to get confused. This is the problem of "distribution shift": the world changes, but the robot's brain stays stuck in the past.

To fix this, scientists use something called "Test-Time Adaptation" (TTA). Think of this as giving the robot a quick brain upgrade while it's working. Instead of stopping to retrain the whole thing, the robot tweaks its own settings on the fly to handle the new rain or noise. But there's a catch: if the robot keeps tweaking itself too much, it might forget how to do the simple things it knew before. This is called "catastrophic forgetting." It's like a student who studies so hard for a new math test that they forget how to do basic addition. The big challenge for audio-visual robots is that they have to learn from two senses at once (sight and sound), and when both get messed up at the same time, the robot can get really lost.

This paper introduces a new method called AVReCAP to help these robots adapt without forgetting. The researchers discovered a clever trick: instead of trying to remember every single new situation the robot encounters, the robot should only tweak a very specific part of its brain—the "fusion layer" that combines sight and sound. They found that once this layer learns to handle one type of mess (like snow), it's actually really good at handling other similar messes (like fog) without needing to be retrained from scratch.

So, AVReCAP works like a smart librarian. As the robot encounters new environments, it takes a quick "snapshot" of the brain settings that worked best for that moment and stores them in a special memory buffer. When the robot faces a new challenge, it doesn't just keep guessing; it checks its library. If the new situation looks statistically similar to a past one (like "rainy" looking like "foggy"), it pulls out the old, proven settings and uses them as a starting point. If the situation is totally new, it learns a little bit and saves a new snapshot. This way, the robot can adapt instantly to new conditions without overwriting its old knowledge or getting confused by conflicting instructions.

The researchers tested this on datasets where they artificially corrupted the data with things like snow, wind, static noise, and crowd chatter. They found that AVReCAP significantly outperformed existing methods. While other approaches often saw their accuracy drop drastically (sometimes falling well below the original robot's performance) as they tried to adapt to a long sequence of new environments, AVReCAP stayed robust. In fact, on a difficult test with 15 different types of audio-visual corruption, the new method only lost about 2.9% of its performance compared to the original, whereas other methods lost nearly 28%. It suggests that by selectively retrieving the right "brain settings" from a memory bank rather than constantly rewriting them, we can keep our audio-visual robots smart, adaptable, and ready for whatever the real world throws at them.

Technical Summary: AVReCAP for Audio-Visual Continual Test-Time Adaptation without Forgetting

1. Problem Definition

The paper addresses Audio-Visual Continual Test-Time Adaptation (AV-CTTA) under strict source-free conditions. In real-world deployments (e.g., autonomous vehicles, robots), audio-visual models face evolving, unlabeled target domains where distributional shifts occur in the audio modality, the visual modality, or both simultaneously (bimodal shifts).

The core challenges identified are:

Catastrophic Forgetting: Existing Test-Time Adaptation (TTA) methods, when applied continuously across multiple domains, suffer from severe performance degradation. The model's accuracy often drops significantly below the original source model due to continual parameter updates that overwrite previously learned knowledge.
Error Accumulation & Modality Imbalance: In bimodal settings, shifts affect audio and visual streams differently, creating a "modality gap." Continual updates on pseudo-labels compound errors, leading to misaligned cross-modal fusion and progressive performance decline.
Source-Free Constraints: Adaptation must occur without access to source training data or explicit task boundaries, adhering to privacy and real-time constraints.

2. Methodology: AVReCAP

The authors propose AVReCAP (Audio-Visual Retrieval-based Continual Adaptation with Parameters), a framework that leverages the transferability of specific model parameters to mitigate forgetting.

2.1 Core Insight: Fusion Layer Transferability

Through empirical analysis (Section 4.1), the authors observe that adapting only the attention fusion layer (specifically the Query, Key, and Value projection matrices $W_Q, W_K, W_V$ of the joint encoder) on a single target domain yields parameters that generalize well to other unseen domains within the same corruption category and even across different categories. This suggests that these parameters capture domain-invariant correlations rather than overfitting to a specific domain.

2.2 Selective Parameter Retrieval

Instead of continuously overwriting the model's parameters, AVReCAP employs a shared buffer to store "snapshots" of the adapted fusion parameters and their corresponding input statistics.

Input Statistics: For each incoming batch, the method computes modality-specific mean ( $\mu$ ) and diagonal covariance ( $\Sigma$ ) for both audio and visual inputs, modeling them as Gaussian distributions.
Retrieval Mechanism: At each time-step $t$ $t$ , the method calculates the Kullback-Leibler (KL) divergence between the current input statistics and all stored buffer elements.
- If a buffer element $n$ is found where the distance $g^*(t, n) < \tau$ (a threshold), the corresponding parameters ( $W_Q, W_K, W_V$ ) are retrieved and plugged into the model.
- The model then performs a single adaptation step on these retrieved parameters using the current batch.
- The retrieved element's statistics and parameters are updated via an Exponential Moving Average (EMA) to smooth the transition.

2.3 Buffer Expansion and Memory Management

If no suitable match is found ( $g^*(t, n) \ge \tau$ for all $n$ ), the current adapted parameters and statistics are added to the buffer as a new element.

Merging: To prevent unbounded memory growth, the buffer has a fixed size budget $\eta$ . When the buffer is full, the two most statistically similar elements (lowest KL divergence) are identified and merged by averaging their statistics and parameters. This consolidates knowledge and maintains a fixed memory footprint.

2.4 Loss Function

The adaptation follows the READ [51] approach, optimizing only the fusion layer weights ( $W_Q, W_K, W_V$ ) using a combination of:

Confidence-based loss ( $\mathcal{L}_{conf}$ ): Encourages high-confidence predictions.
Negative Entropy loss ( $\mathcal{L}_{ne}$ ): Minimizes prediction entropy.
The encoders (audio and visual) remain frozen throughout the process.

3. Key Contributions

Comprehensive Study: The paper provides the first extensive study of source-free audio-visual CTTA under both unimodal and challenging bimodal corruptions.
Transferability Discovery: It demonstrates that the attention fusion layer exhibits strong intra- and cross-task transferability, enabling the selective retrieval of parameters from past time-steps rather than continuous overwriting.
AVReCAP Framework: A novel method utilizing a shared buffer and selective retrieval mechanism that significantly mitigates catastrophic forgetting. On the difficult VGGSound-2C benchmark (15 bimodal corruptions), AVReCAP reduces the performance drop to 2.9% compared to the source model, whereas the baseline READ method suffers a 27.9% drop.
State-of-the-Art Performance: AVReCAP outperforms existing TTA (TENT, EATA, SAR) and AV-TTA (READ, SuMi, PTA, BriMPR*) baselines across multiple datasets (Kinetics50, VGGSound) and corruption types.

4. Experimental Results

Experiments were conducted on Kinetics50 and VGGSound datasets with severity level 5 corruptions.

Unimodal Corruptions: AVReCAP achieves superior mean accuracy compared to baselines. For instance, on VGGSound-C with audio corruptions, it shows a relative improvement of 4.95% over the next best method (PTA).
Bimodal Corruptions: In the most challenging setting (Kinetics50-2C and VGGSound-2C), most baselines fail to maintain source-level performance. AVReCAP achieves a 1.25% improvement over READ on Kinetics50-2C and a substantial 6.28% improvement over the source model on VGGSound-2C.
Forgetting Mitigation: Figure 7 illustrates that AVReCAP minimizes catastrophic forgetting, maintaining accuracy close to the source model even after 15 sequential domain shifts, while other methods degrade significantly.
Robustness: The method remains stable under varying task orders (reverse, random shuffle) and small batch sizes (down to $B=8$ ).

5. Significance and Claims

The paper claims that AVReCAP represents a major step forward in the real-time deployment of audio-visual models in continually changing environments. By shifting the paradigm from "continuous overwriting" to "selective retrieval of generalizable fusion parameters," the method effectively balances adaptation to new domains with the preservation of source knowledge.

The authors emphasize that their approach adheres to strict privacy constraints (no source data access) and memory efficiency (fixed buffer size via merging), making it suitable for edge deployment. They acknowledge limitations, including the linear search overhead ( $O(N)$ ) for retrieval as the buffer grows and the assumption that models follow a specific late-fusion architecture (modality-specific encoders + joint encoder). However, the work establishes new SOTA baselines for robust audio-visual continual adaptation.