💻 computer science

PRiSM: Prototype Regularization for Few-Shot VLMs

The paper introduces PRiSM, a training-free prototype regularization method with a novel multi-term loss and efficient Majorize-Minimize optimizer, which significantly improves the robustness of few-shot Vision-language Models against realistic challenges like class imbalance and varying numbers of classes where current state-of-the-art methods fail.

Original authors: Ghassen Baklouti, Omprakash Chakraborty, Jose Dolz, Ismail Ben Ayed

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ghassen Baklouti, Omprakash Chakraborty, Jose Dolz, Ismail Ben Ayed

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to recognize animals. You don't want to spend years feeding it millions of pictures; instead, you want to show it just a few examples of a new creature, like a "fluff-puff," and have it figure out the rest. This is the world of Vision-Language Models (VLMs). Think of these models as a giant library where pictures and words are already sorted into a shared filing system. The robot knows that the word "dog" and a picture of a dog live in the same neighborhood of this library.

Usually, to teach the robot a new trick, scientists use a method called few-shot learning. They give the robot a tiny "support set"—just a handful of labeled examples (say, 4 or 16 pictures of a specific animal)—and ask it to adjust its internal map to recognize that animal in new, unseen photos. For a long time, scientists tested these methods using a very neat, tidy setup: they made sure every animal in the test had the exact same number of examples. It was like a classroom where every student raised their hand exactly once. But in the real world, life is messy. Some animals are everywhere (like pigeons), while others are rare (like snow leopards). This paper asks a simple but crucial question: What happens to our robot's brain when we stop pretending the world is perfectly balanced and start feeding it the messy, uneven reality of the real world?

The Problem: The "Majority Vote" Trap

The researchers, a team from ÉTS Montreal, discovered that our current robot teachers are surprisingly fragile. They found that when you test these models on realistic data where some classes are common and others are rare, the models start to fail spectacularly. In fact, they found a weird paradox: giving the robot more examples often made it worse.

Imagine you are trying to learn the names of all the students in a school. If you only get to meet 4 students, and 3 of them are from the same popular clique, you might start thinking that everyone in the school looks like that clique. If you then meet 16 students and 15 of them are still from that same clique, your confusion gets even worse. You've learned the "majority" so well that you've completely forgotten how to spot the "minority."

The paper shows that current "training-free" methods (which are supposed to be smart and efficient) fall into this trap. They rely on a "prototype"—a mental average of what a class looks like. When the data is unbalanced, the mental average gets dragged toward the majority class, blurring the lines between different animals. The more data you add, the stronger this "drag" becomes, causing the robot to misidentify rare animals as common ones.

The Solution: PRiSM (The "Fairness Coach")

To fix this, the authors introduce a new tool called PRiSM (Prototype Regularization for Few-Shot VLMs). Think of PRiSM as a strict but fair coach that steps in after the robot has taken a first guess.

Here is how PRiSM works, using a playground analogy:

The Initial Mess: The robot looks at the few pictures it has and draws a "mental map" of where each animal belongs. Because there are too many pictures of the common animals, the map is squished and messy; the rare animals are pushed off the edge.
The Coach's Intervention: PRiSM doesn't throw away the robot's work. Instead, it applies a set of rules (a "regularizer") to tidy up the map.
- Rule 1 (Stay True): "Don't drift too far from what you saw." It keeps the robot's new guesses close to the actual pictures it was shown.
- Rule 2 (Respect the Original): "Don't forget what you already knew." It ensures the robot doesn't completely forget its original, pre-trained knowledge.
- Rule 3 (Keep Your Distance): "Make sure the groups don't overlap." This is the most important part. PRiSM actively pushes the "mental clusters" of different animals apart. If the robot is confusing a "cat" with a "dog" because there were too many dogs in the training set, PRiSM physically pushes the "cat" cluster away from the "dog" cluster to create clear space between them.

The team also invented a special, super-fast math trick (called a block Majorize-Minimize optimizer) to do this cleaning job. It's like having a GPS that instantly calculates the perfect speed to drive without needing to test different speeds first. This makes the whole process incredibly fast and efficient, requiring no extra "test" data to tune the settings.

What They Found

The results were surprising and powerful. The authors tested PRiSM on 10 different datasets, ranging from recognizing flowers and cars to spotting textures and scenes.

The "More Data" Paradox Confirmed: In their realistic, unbalanced tests, they saw that standard methods (like Tip-Adapter and APE) actually got worse as they increased the number of training shots from 2 to 16. For example, on some datasets, adding more pictures caused the accuracy to drop by over 30%.
The Magic Fix: When they added PRiSM on top of these failing methods, the accuracy skyrocketed. In the most extreme cases of imbalance, PRiSM turned a failing system into a top-performer. For instance, on one dataset with severe imbalance, PRiSM boosted a method's accuracy by over 40 percentage points (jumping from roughly 21% to over 60%).
It Works Everywhere: PRiSM wasn't just a band-aid for weak models. It even helped the strongest existing methods (like ProKeR) perform slightly better, proving that the problem wasn't the robot's brain, but the messy way the classes were arranged.

The authors suggest that the main reason these models fail in the real world isn't that they aren't smart enough, but that the way we test them (assuming perfect balance) hides a critical flaw: prototype bias. When the data is uneven, the robot's "mental average" gets skewed, and PRiSM is the tool that straightens it out.

Why This Matters

This paper suggests that for AI to be truly useful in the real world, we need to stop testing it in the "perfect classroom" and start testing it in the "messy playground." By introducing a benchmark that mimics real-world imbalance and a tool (PRiSM) that fixes the resulting confusion, the authors show that we can build AI that is robust enough to handle the uneven distribution of data we actually encounter every day. They didn't just find a better way to tune a robot; they found a way to make the robot fairer, ensuring it pays attention to the rare and the common alike.

Technical Summary: PRiSM - Prototype Regularization for Few-Shot VLMs

Problem Statement

The paper identifies two critical limitations in current training-free few-shot adaptation methods for Vision-Language Models (VLMs):

Unrealistic Evaluation Protocols: Standard benchmarks rely on class-balanced support and query sets. This assumption masks severe prototype bias that occurs in real-world scenarios where data distributions are naturally skewed (class imbalance) and the set of observed classes may be incomplete.
Degradation with Increased Data: Contrary to standard few-shot assumptions, the authors observe that under realistic class imbalance, increasing the number of shots (labeled samples) often degrades the performance of state-of-the-art training-free methods. Instead of refining decision boundaries, additional samples from imbalanced distributions amplify the bias toward majority classes, leading to distorted prototypes and increased inter-class confusion.

Existing methods (e.g., Tip-Adapter, APE, GDA, ProKeR) often require task-specific hyperparameter tuning via validation sets—a practice that violates the few-shot assumption when such sets are unavailable or expensive to annotate. Furthermore, these methods lack mechanisms to correct the inherent bias in prototype estimation caused by skewed support distributions.

Methodology: PRiSM

The authors propose PRiSM (Prototype Regularization for Few-Shot VLMs), a plug-and-play framework designed to rectify biased class prototypes without requiring a validation set.

1. Objective Function

PRiSM optimizes a novel multi-term loss function to learn regularized prototypes $w$ from a baseline classifier's prototypes $a$ . The objective $L(w, \phi)$ combines three terms:
$L(w, \phi) = \beta \sum_{c} \|w_c - \mu_c(\phi)\|_2^2 + \gamma \sum_{c} \|w_c - a_c\|_2^2 - \frac{\lambda}{2(C-1)} \sum_{c \neq c'} \|w_c - w_{c'}\|_2^2$
Where:

Support-Feature Alignment ( $\beta$ ): Encourages the rectified prototype $w_c$ to align with the mean visual features $\mu_c(\phi)$ of the support set, ensuring consistency with the observed data distribution.
Baseline Fidelity ( $\gamma$ ): Anchors the rectified prototype to the original baseline prototype $a_c$ , preventing substantial deviation from the pretrained zero-shot knowledge.
Inter-Class Separation ( $\lambda$ ): A pairwise regularizer that maximizes the distance between different class prototypes, reducing interference and improving separability under imbalanced conditions.

The optimization involves two variable blocks: the corrected prototypes $w$ and the vision encoder parameters $\phi$ .

2. Optimization Strategy: Block Majorize-Minimize (MM)

To solve the non-convex objective efficiently, the authors introduce a block Majorize-Minimize (MM) optimizer that alternates between:

Prototype Rectification: Updating $w$ while keeping $\phi$ fixed. Since the objective is quadratic with respect to $w$ , the authors derive a valid blockwise Lipschitz constant $\rho$ (a bound on the Hessian's spectral norm). This allows for an implicit step size of $1/\rho$ , eliminating the need for computationally expensive grid searches for learning rates. The bound is computed efficiently using the Gershgorin circle theorem.
Feature Adaptation: Updating $\phi$ while keeping $w$ fixed. This is performed via standard back-propagation using LoRA (Low-Rank Adaptation) to ensure parameter efficiency, updating only a small subset of the vision encoder's layers.

Key Contributions

Realistic Benchmarking: The authors expose the limitations of standard few-shot protocols by introducing a benchmark that varies class imbalance and the effective number of classes using Dirichlet sampling. They demonstrate that standard methods degrade as the number of shots increases under severe imbalance.
PRiSM Framework: A validation-free, plug-and-play prototype regularization module that can be deployed on top of any existing training-free baseline. It effectively mitigates majority-class dominance while preserving the discriminative structure of the VLM embedding space.
Efficient Optimization: The derivation of a blockwise Lipschitz constant for the prototype update, enabling a principled, validation-free step size selection that is computationally efficient and theoretically grounded.
Empirical Validation: Extensive experiments showing that PRiSM significantly improves the performance of diverse baselines (Tip-Adapter, APE, GDA, ProKeR), particularly in scenarios with severe class imbalance and high numbers of classes.

Experimental Results

Experiments were conducted on 10 diverse image classification datasets (e.g., Oxford-Pets, EuroSAT, Food-101) using CLIP with a ViT-B/16 backbone.

Severe Imbalance (High Effective Classes): Under severe imbalance, baseline methods often degrade as the number of shots increases (e.g., Tip-Adapter dropped from ~63% to ~33% accuracy when moving from 2-shot to 16-shot). PRiSM reversed this trend, improving Tip-Adapter's mean accuracy by +35.05% in the 16-shot setting and +23.22% in the 4-shot setting.
Near-Balanced Settings: Even in near-balanced conditions, PRiSM provided consistent gains, with Tip-Adapter improving by +29.79% in the 16-shot setting.
Low Effective Classes: In regimes with fewer effective classes, PRiSM still yielded significant improvements, particularly for weaker baselines, confirming that prototype bias persists even with reduced label spaces.
Comparison: PRiSM allowed simpler methods (like Tip-Adapter) to reach performance levels comparable to or exceeding more complex state-of-the-art methods (like ProKeR) once prototypes were regularized.
Learning Rate Analysis: The proposed Lipschitz-based step size was shown to match the optimal performance region across datasets without requiring exhaustive hyperparameter tuning, validating the theoretical derivation.

Significance and Claims

The paper claims that the primary bottleneck in few-shot VLM adaptation is not necessarily the adaptation mechanism itself, but the quality of the class prototypes used for prediction, which are often biased by imbalanced data.

The authors argue that:

Prototype Bias is a Common Failure Mode: Imbalance-induced bias acts as a systematic failure mode across diverse adaptation methods. Rectifying this bias can systematically elevate the performance of diverse approaches.
Validation-Free Adaptation is Feasible: By deriving a theoretically grounded step size and using a multi-term loss that balances alignment, fidelity, and separation, PRiSM achieves competitive performance without the unrealistic assumption of access to a labeled validation set.
Realism in Evaluation: The proposed benchmark highlights that current state-of-the-art claims may be inflated by unrealistic class-balanced assumptions. The authors advocate for evaluation protocols that reflect real-world skewed distributions to foster the development of more robust adaptation techniques.

The paper concludes that PRiSM offers a simple yet effective correction that regularizes prototype geometry, preserving semantic fidelity while preventing collapse toward dominant classes, thereby enhancing robustness in challenging few-shot regimes.