💻 computer science

GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation

This paper introduces a systematic benchmark demonstrating that intra-class diversity is a critical factor for effective test-time adaptation memory policies, leading to the development of the Guided Observational Test-Time Adaptation (GOTTA) framework, which combines class-balanced allocation with feature-space diversity to significantly improve model robustness under challenging non-i.i.d. streams and constrained memory budgets.

Original authors: Shyma Alhuwaider, Yasmeen Alsaedy, Merey Ramazanova, Silvio Giancola, Bernard Ghanem

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shyma Alhuwaider, Yasmeen Alsaedy, Merey Ramazanova, Silvio Giancola, Bernard Ghanem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a smart robot that was trained in a perfect, controlled classroom. Now, you send this robot out into the real world to do a job. The real world is messy, unpredictable, and constantly changing. The robot's old training doesn't quite fit anymore.

Test-Time Adaptation (TTA) is the robot's way of learning on the fly. As it encounters new, unlabeled data (like a rainy day after a sunny week, or a new type of car), it tries to adjust its brain to keep working correctly.

However, the robot has a problem: its memory is tiny. It can't remember every single thing it sees. It has to choose what to keep in its short-term memory to help it learn.

The Old Way: The "First-In, First-Out" Bucket

Most previous methods treated the robot's memory like a simple bucket.

The FIFO (First-In, First-Out) approach: "I'll just keep the newest things I saw and throw out the oldest."
The Class-Balanced approach: "I'll make sure I have an equal number of pictures of cats and dogs."

The Problem: Imagine the robot walks through a street where it sees 100 identical red cars in a row.

If it uses the FIFO method, it might keep 100 red cars and forget about the blue truck that passed by earlier.
If it uses the Class-Balanced method, it might keep 50 red cars and 50 red cars (because it thinks it needs more "car" examples), but it still misses the variety of the world. It's wasting its tiny memory space on duplicates.

The paper calls this redundancy. It's like trying to learn a language by reading the same sentence 100 times instead of reading 100 different sentences.

The New Idea: The "Diverse Collection"

The authors of this paper, GoTTA, argue that memory shouldn't just be a bucket; it should be a curated museum exhibit.

They propose a new system called Guided Observational Test-Time Adaptation (GOTTA). Instead of just asking "Is this a cat?" or "Is this new?", their system asks: "Is this different from what I already have?"

They use two clever strategies to pick the best memories:

The "Farthest Point" Strategy (FPS): Imagine you are picking fruits for a basket. You don't just grab the first apple you see. You look at the apples you already have, and you only pick a new one if it is very different (far away) from the ones already in the basket. This ensures your basket has a wide variety of apples, not just clones.
The "Cosine Diversity" Strategy (CDS): This looks at the "shape" or "angle" of the data. It ensures that even within the same category (like "dogs"), the robot remembers a Chihuahua, a Golden Retriever, and a Poodle, rather than 50 Golden Retrievers.

What They Found

The researchers tested this on various "messy" streams of data (like videos or images with weather changes) and found:

Less is More: When the robot has very little memory (a small bucket), the "Diverse Collection" strategy works much better than the old ways. It prevents the robot from wasting space on duplicates.
Handling Chaos: When the world changes rapidly (e.g., sunny to rainy to snowy), the old methods get confused because they keep seeing the same thing over and over. The new method keeps a diverse set of examples, helping the robot adapt faster.
It Works Everywhere: This new memory system works well with many different types of robot brains (adaptation algorithms). It's like a universal plug-in that makes any robot smarter.

The Big Takeaway

The paper concludes that how you choose what to remember is just as important as how you learn.

If you are trying to adapt to a changing world, you don't need a bigger memory; you need a smarter memory. By ensuring the robot remembers a diverse mix of experiences rather than just the most recent or most common ones, it stays robust and accurate, even when the world gets weird.

In short: Don't just save everything you see. Save the most interesting and different things, so you can learn the most from the least amount of space.

Technical Summary: GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation

Problem Definition

Test-Time Adaptation (TTA) enables pre-trained models to adapt online to unlabeled test streams experiencing distribution shifts without access to source data. While much research focuses on adaptation objectives (e.g., entropy minimization, pseudo-labeling), practical deployment faces challenges from non-stationary, temporally correlated, and class-imbalanced streams. In such settings, consecutive batches often contain redundant samples or exhibit severe label skew, leading to amplified prediction errors and accelerated forgetting.

Existing TTA methods often employ memory replay buffers to stabilize adaptation. However, these memory mechanisms are typically evaluated as inseparable components of specific algorithms, making it difficult to isolate which design choices (e.g., recency, class balance, diversity) drive performance. Furthermore, standard memory policies (e.g., FIFO, reservoir sampling) or even class-guided strategies often fail to prevent the retention of near-duplicate samples within the same class, wasting limited memory capacity and weakening the adaptation signal.

Methodology

The authors propose a systematic approach to decouple memory management from the adaptation algorithm, treating memory as a modular, first-class component of TTA.

1. Systematic Benchmarking

The study evaluates memory policies under unified conditions across:

Stream Types: i.i.d., non-i.i.d. (temporally correlated), continual, and practical video streams.
Policy Families:
- Uninformed: FIFO, Reservoir Sampling.
- Class-guided: Pseudo-label Balanced Reservoir Sampling (PBRS), CSTU (class balancing with timeliness/uncertainty).
- Diversity-aware: The proposed Guided Observational Test-Time Adaptation (GOTTA) family.

2. The GOTTA Framework

GOTTA introduces diversity-aware memory policies that jointly enforce class balance and intra-class geometric diversity. The core principle is that memory should act as a compact coreset of observed samples rather than a passive buffer. GOTTA utilizes two primary sampling mechanisms:

Farthest Point Sampling (FPS):
- Assigns incoming samples to pseudo-label partitions.
- Within a partition, it compares the candidate to the nearest stored neighbor.
- It employs a shared eviction score $H(m_i)$ based on age and uncertainty (adapted from CSTU) to decide which entry to evict.
- Diversity Check: A candidate is accepted only if its distance to the nearest neighbor in the partition exceeds a threshold $\epsilon$ (set to 0.005).
- FPSD (Dynamic): Extends FPS by periodically re-encoding stored samples through the current model to update representations and pseudo-labels, ensuring diversity metrics remain consistent with the evolving feature space.
Cosine Diversity Sampling (CDS):
- Focuses on angular diversity within class partitions.
- Uses normalized representations to compute cosine similarity.
- If a partition is full, it forms a candidate set including the new sample and identifies the most redundant element (highest maximum similarity to others) for eviction.
- Includes a lightweight redundancy filter to discard highly similar candidates immediately.

3. Experimental Setup

Datasets: CIFAR-10-C, ImageNet-C, and ITD (a video-based benchmark with inherent non-i.i.d. structure).
Protocols: Episodic and continual adaptation under varying memory budgets ( $M \in \{8, 16, 32, 64\}$ ) and stream skew levels (controlled by Dirichlet concentration parameter $\gamma$ ).
Baselines: Evaluated across diverse TTA methods including TENT, SHOT, PL, CoTTA, NOTE, RoTTA, EATA, SAR, and Energy.

Key Results

1. Impact of Memory Capacity and Stream Correlation

Constrained Budgets: Under tight memory constraints (e.g., $M=32$ ) and temporally correlated streams, diversity-aware memories (GOTTA) significantly outperform uninformed and class-guided baselines. For instance, on CIFAR-10-C with $M=32$ , FPS paired with the Norm method achieved 76.11% accuracy, surpassing FIFO (72.63%) and CSTU (72.75%).
Severe Under-provisioning: On ImageNet-C ( $M=64$ for 1,000 classes), where strict class balance is mathematically impossible, diversity-aware selection proved critical. CDS and FPS maintained competitive performance where other methods collapsed, demonstrating that per-instance diversity is a more robust criterion than strict class balancing in extreme scenarios.
Video Streams: On the ITD video benchmark, FPS paired with RoTTA achieved the best results among memory policies, highlighting the necessity of handling temporal redundancy in video data.

2. Diversity Threshold Sensitivity

Ablation studies on the diversity threshold $\epsilon$ revealed that $\epsilon = 0.005$ consistently provided the best balance between diversity and representativeness.
Smaller values ( $\epsilon = 0.001$ ) allowed redundant samples to accumulate, while larger values ( $\epsilon = 0.01$ ) overly restricted memory updates, reducing coverage of the data distribution.

3. Scalability

As memory capacity increased (e.g., to $M=64$ ), the performance gap between diversity-aware and simpler policies narrowed. This suggests that larger buffers can partially compensate for weaker selection strategies, but diversity-aware policies remain superior or competitive even as capacity scales.

Significance and Claims

The paper argues that memory management is a first-class component of robust Test-Time Adaptation. The authors claim that:

Diversity is Central: Effective memory management requires more than just retaining recent or class-balanced samples; intra-class feature diversity is essential to avoid redundant buffers and maintain representative adaptation signals, particularly under temporally correlated and label-skewed streams.
Modularity: Memory policies can be decoupled from adaptation objectives. GOTTA serves as a "drop-in replacement" for existing buffers, improving performance across various TTA algorithms without modifying the underlying adaptation loss or architecture.
Practical Necessity: In real-world deployment where memory is constrained and data is non-i.i.d., enforcing diversity alongside class balance is a critical design principle for preventing error accumulation and forgetting.

Limitations

The authors acknowledge that GOTTA relies on feature-space distances, the reliability of which depends on the quality of representations during adaptation. Additionally, the approach introduces higher curation costs compared to simple FIFO or reservoir memories. The evaluation is primarily focused on corruption benchmarks and controlled video streams, with broader validation on diverse deployment data left for future work.