🤖 machine learning

Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction

The paper proposes MOON, a training-free and model-agnostic test-time transduction method that leverages a Von Mises-Fisher mixture model with dynamic shrinkage driven by zero-shot priors to robustly handle class imbalance and prevent performance collapse in vision-language models.

Original authors: Jiazhen Huang, Zhiming Liu, Changhu Wang, Wei Ju, Ziyue Qiao, Xiao Luo

Published 2026-07-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiazhen Huang, Zhiming Liu, Changhu Wang, Wei Ju, Ziyue Qiao, Xiao Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that has read every book and seen every picture on the internet. It's a genius at understanding the world, but it's never been tested on a specific, tricky situation before. This is the world of Vision-Language Models (VLMs). These are AI systems that can look at a photo and describe it, or read a sentence and imagine the scene. They are incredibly powerful, but they usually work best when they are given a huge, perfectly balanced set of practice examples.

However, in the real world, things are messy. Imagine you hand this robot a stack of photos to sort, but the stack is weird: 99% are pictures of cats, and only one is a picture of a dog. Or maybe the photos keep changing as you look at them, with long stretches of only cars, then suddenly a flood of birds. This is called class imbalance. When the robot tries to learn from these unbalanced, messy piles of data on the fly (a process called test-time transduction), it often gets confused. It might start thinking everything is a cat because it saw so many, or it might get so scared of the rare dog that it refuses to guess correctly. The big question for scientists is: How do we teach these robots to stay calm and accurate when the data is unbalanced and chaotic, without needing to retrain them from scratch?

Enter MOON, a new method proposed by researchers Jiazhen Huang and their team. Think of MOON as a very wise, cautious librarian for our robot's brain. When the robot tries to sort a new batch of photos, it usually makes a quick guess based on what it learned before (the "zero-shot" guess). But in a messy pile where some items are rare and others are common, the robot's quick guesses can be noisy and unreliable.

MOON steps in with a clever trick called Dynamic Shrinkage. Imagine the robot is trying to find the center of a group of friends in a crowded room. If the group is huge and clear, the robot trusts its own eyes completely. But if the group is tiny, or if the room is full of strangers who look nothing like the friends, the robot gets shaky. MOON tells the robot: "Hey, if you aren't sure, don't trust your shaky guess too much. Pull your answer closer to the safe, reliable knowledge you already have."

The magic of MOON is that it doesn't use a fixed rule for how much to pull back. Instead, it dynamically adjusts.

At the individual level: If the robot is looking at a photo and feels very confused (high "entropy"), MOON says, "Ignore that guess; it's too noisy."
At the group level: If the robot sees a category of items that barely shows up in the pile (like that single dog in a sea of cats), MOON says, "Don't let that rare item trick you into changing your whole understanding. Stick to what you know about dogs from your training."

The researchers tested this on 11 different datasets, ranging from pictures of flowers and cars to scenes and textures. They found that while other methods often crashed or performed terribly when the data was unbalanced, MOON stayed steady. In fact, on the massive ImageNet dataset, MOON improved the robot's accuracy by 13.2% across 10 different scenarios, beating the previous best methods by a wide margin. It did this without needing any extra training data or complex tuning, and it was incredibly fast—processing thousands of samples in just a few milliseconds.

The paper explicitly argues against the idea that we can just treat all data as if it were perfectly balanced, or that we can use a single, static rule to fix mistakes. They show that methods which don't have this "anchoring" mechanism (a safe baseline to pull back to) tend to overfit to the noisy, local data and collapse. They also show that even methods that do have an anchor often fail because they use a "static" strength that doesn't change based on how rare or common a class is. MOON fixes this by making the shrinkage strength dynamic, changing in real-time based on how much evidence the robot actually has.

In short, the authors suggest that by modeling the robot's knowledge as points on a sphere (using something called a Von Mises-Fisher mixture) and gently pulling its guesses toward a safe anchor only when necessary, we can make these powerful AI systems much more robust in the real, messy world. They proved this through extensive experiments, showing that MOON is not just a theoretical idea, but a practical, efficient tool that works across many different types of images and AI models. It's a reminder that sometimes, the smartest thing an AI can do is know when to trust its gut and when to double-check its notes.

Technical Summary: Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction

1. Problem Definition

Vision-Language Models (VLMs), such as CLIP, have demonstrated remarkable zero-shot capabilities by aligning visual and textual concepts in a shared latent space. However, enhancing their performance at test time using only unlabeled data remains challenging, particularly under realistic deployment scenarios.

Existing test-time adaptation (TTA) and transductive methods often assume class-balanced distributions or uniform class marginals. In reality, test data streams frequently exhibit highly imbalanced class distributions (e.g., sparse batches, long-tailed distributions, or non-i.i.d. temporal correlations). Under these conditions, standard transductive methods—which typically rely on soft probabilistic clustering—suffer from performance degradation or even catastrophic collapse. This failure stems from two primary issues:

Overfitting to Local Statistics: Without a mechanism to anchor estimates to prior knowledge, methods overfit to the biased statistics of majority classes within a batch, amplifying errors in minority or absent classes.
Static Shrinkage Modeling: Existing approaches that do incorporate regularization (e.g., via KL-divergence anchors) often use a static hyperparameter for shrinkage strength. This assumes equal reliability across all classes and instances, failing to dynamically adjust to the varying confidence and presence of specific classes in a given batch.

2. Methodology: MOON

The authors propose MOON (Mixture of von Mises-Fisher Models with Dynamic Shrinkage), a training-free, model-agnostic method for realistic test-time transduction. The methodology is grounded in Penalized Likelihood Estimation (PLE) and operates on the unit hypersphere where normalized VLM embeddings reside.

2.1 Theoretical Foundation: PLE and Adaptive Shrinkage

The authors revisit transduction through the lens of PLE, formulating the objective as minimizing negative log-likelihood (NLL) plus a regularization term. They theoretically demonstrate that incorporating a KL-divergence anchor term (based on zero-shot priors) naturally yields an adaptive shrinkage behavior.

The Mechanism: The update for class statistics becomes a convex combination of the empirical estimate ( $\hat{\mu}_k$ ) and the prior anchor ( $\mu'_k$ ):
$\mu_k = \beta_k \hat{\mu}_k + (1 - \beta_k) \mu'_k$
where the shrinkage strength $\beta_k$ is determined by the ratio of the effective sample count ( $n_k$ ) to the anchor weight ( $\alpha$ ).
Implication: Classes with abundant support approach standard Maximum Likelihood Estimation (MLE), while rare or absent classes remain strongly regularized toward the prior, preventing negative transfer.

2.2 Core Components of MOON

MOON implements this theory using a Mixture of von Mises-Fisher (vMF) distributions, which are topologically native to the unit hypersphere of normalized embeddings.

Instance-Level Dynamic Shrinkage ( $\gamma$ ):
- To handle noisy predictions within a batch, MOON introduces an entropy-based weight $\gamma_i$ for each sample.
- Samples with high predictive entropy (uncertain predictions) are down-weighted in the likelihood calculation, suppressing unreliable assignments during parameter estimation.
Class-Level Dynamic Shrinkage ( $\alpha_k$ ):
- Instead of a static scalar $\alpha$ , MOON employs class-specific weights $\alpha_k$ derived from zero-shot priors.
- $\alpha_k$ is inversely proportional to the class confidence ( $\lambda_k$ ), defined as the geometric mean of average and maximum zero-shot confidence.
- Effect: Highly confident classes (likely present) allow the model to learn more from empirical data ( $\beta_k \to 1$ ), while low-confidence or outlier classes (likely absent) are forced to shrink strongly toward the anchor ( $\beta_k \to 0$ ), mitigating negative transfer from non-existent classes.
Optimization:
- The method uses a Block Successive Minimization (BSUM) algorithm, alternating between updating soft label assignments ( $z$ ) and mixture parameters ( $\mu, \kappa$ ).
- The updates are derived in closed-form, ensuring computational efficiency. The algorithm converges rapidly, often within a few iterations.

3. Key Contributions

New Theoretical Perspective: The paper systematically analyzes the brittleness of existing transductive methods under class imbalance. It proves that KL-anchored PLE inherently provides adaptive shrinkage, identifying the lack of anchoring and static shrinkage modeling as the root causes of failure in realistic settings.
Novel Methodology (MOON): A novel framework combining vMF mixtures with dynamic, data-driven shrinkage at both instance and class levels. This design effectively suppresses unreliable assignments and prevents harmful updates from outlier classes without requiring task-specific hyperparameter tuning.
Empirical Validation: Extensive experiments across 11 datasets and two realistic settings (batch adaptation with limited effective classes and online adaptation with non-i.i.d. streams) validate the approach.

4. Experimental Results

MOON was evaluated against state-of-the-art baselines (including TransCLIP, StatA, ADAPT, and various online TTA methods) using CLIP ViT-B/16 and other backbones.

Performance: MOON consistently outperforms baselines. On ImageNet under batch adaptation with severe class imbalance (Very Low $K_{eff}$ ), MOON improves zero-shot CLIP performance by 16.2% (reaching 82.8%), significantly outperforming the strongest baseline (StatA) by 8.8%. It achieves state-of-the-art results across 10 out of 11 datasets in various imbalance scenarios.
Efficiency: MOON is training-free and highly efficient. It processes thousands of samples in tens of milliseconds, requiring only 3.3% of the inference latency of standard CLIP. It avoids expensive gradient backpropagation and inner-loop optimizations required by some Gaussian-based methods.
Robustness: The method demonstrates robustness across different batch sizes, model architectures (ResNet, ViT), and VLM scales (up to 1.1B parameters). It maintains performance even in "Random" scenarios where class sparsity fluctuates unpredictably.
Convergence: Analysis shows MOON converges rapidly (often within 3 iterations), whereas methods without anchoring mechanisms tend to diverge or collapse over iterations.

5. Significance and Claims

The paper claims that MOON offers a practical, effective, and efficient solution for deploying VLMs in real-world scenarios where class distributions are unknown and imbalanced.

Practicality: By operating under a black-box assumption and requiring no task-specific tuning, MOON is ready for seamless deployment.
Reliability: It addresses the critical issue of negative transfer in imbalanced settings, ensuring that VLMs do not degrade when encountering sparse or skewed data streams.
Generality: The use of vMF distributions and the theoretical grounding in PLE suggest that the approach is a generalized solution for enhancing VLMs, independent of the specific underlying visual encoder.

The authors conclude that by systematically revisiting transduction through penalized likelihood estimation and introducing dynamic shrinkage, MOON successfully bridges the gap between theoretical robustness and practical efficiency in test-time adaptation.