🤖 AI

Generalizable Vision-Language Few-Shot Adaptation with Predictive Prompts and Negative Learning

The paper introduces SCAN, a parameter-free framework for vision-language few-shot adaptation that enhances performance by employing query-specific negative routing, LLM-generated contrastive prompts, and adaptive fusion weights to effectively address query-specific class confusion and distribution shifts.

Original authors: Sriram Mandalika

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sriram Mandalika

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly rigid, robot how to recognize different types of birds. You only have a few photos of each bird to show it (this is called "few-shot learning"). The robot already knows a lot about the world because it read the entire internet, but it still gets confused when two birds look very similar, like a Red-tailed Hawk and a Red-shouldered Hawk.

The paper introduces a new method called SCAN (Selective Confusion-Aware Negatives) to fix this confusion. Here is how it works, broken down into three simple ideas:

1. The "Smart Security Guard" (Query-Adaptive Routing)

The Old Way: Imagine a security guard at a museum who, whenever a visitor walks in, yells, "You are NOT a painting, you are NOT a statue, you are NOT a vase..." to every single object in the museum, all at once. It's a loud, messy noise that doesn't help the visitor figure out what they are actually looking at. The robot does this too: it tries to tell every image what it isn't, even if it's impossible for that image to be confused with those things.

The SCAN Way: SCAN acts like a smart security guard who looks at the visitor first. If the visitor looks like a Hawk, the guard only whispers, "You are definitely not a Red-shouldered Hawk," because that's the only thing you might be confused with. The guard ignores all the other birds (like Penguins or Flamingos) because you clearly aren't them.

Why it helps: By only focusing on the specific things the image is likely to be confused with, the robot makes a much sharper decision without getting distracted by irrelevant noise. It does this without needing any extra memory or training.

2. The "Expert Art Critic" (LLM-Bootstrapped Prompts)

The Old Way: When describing a bird to the robot, old methods might just say, "This is a photo of a Red-tailed Hawk." If the robot sees a similar bird, it might think, "Well, that's also a photo of a bird, so maybe it's the same one." It's too vague.

The SCAN Way: SCAN hires an AI Art Critic (a Large Language Model) to write a much better description. Instead of just naming the bird, the critic says: "This is a Red-tailed Hawk, which you can tell apart from a Red-shouldered Hawk because the Hawk has a rusty tail and a white belly, while the other one has a barred tail."

Why it helps: It gives the robot specific "clues" to look for that distinguish similar items. It's the difference between saying "Don't eat that" and saying "Don't eat that mushroom; it looks like the edible one, but it has red spots."

3. The "Intuitive Balance Scale" (Adaptive Fusion)

The Old Way: The robot has two ways of thinking: looking at the picture (Visual) and reading the description (Text). Usually, humans have to manually decide how much to trust the picture versus the text. It's like trying to balance a scale by guessing how heavy each side is. If you guess wrong, the robot gets confused.

The SCAN Way: SCAN has a magic scale that automatically figures out the balance. If the pictures are very clear and distinct, the scale tips to trust the picture more. If the pictures are blurry but the names are very different, the scale tips to trust the text more. It does this by looking at the few examples you gave it and deciding the best mix on the fly, without any human guessing.

The Results: How Well Did It Work?

The authors tested this "Smart Guard" system on 11 different challenges, from recognizing flowers to spotting cars and textures.

The Score: On average, SCAN was 4.6% more accurate than the previous best methods when given 16 examples.
The Big Wins: It shined the brightest on the hardest tasks (like telling apart very similar birds or cars), where it improved accuracy by up to 7.7%.
The Tough Stuff: Even when the data was messy (like if 50% of the labels were wrong, like a teacher marking the wrong answers on a test), SCAN still performed better than the competition. It was also very good at recognizing things it had never seen before in slightly different lighting or styles (like a sketch of a cat instead of a photo).

Summary

In short, SCAN teaches the robot to stop shouting "No!" at everything and start whispering "No!" only to the things that matter. It uses a smart critic to explain why things are different, and it automatically knows whether to trust its eyes or its brain. This makes it much better at learning new things from just a few examples.

Technical Summary: SCAN – Selective Confusion-Aware Negatives for Few-Shot Vision-Language Models

Problem Statement

Few-shot adaptation of Vision-Language Models (VLMs) like CLIP is fundamentally constrained by how negative class signals are handled during inference. Existing methods typically apply uniform negative suppression across all classes for every query, regardless of the specific confusion patterns of that query. This approach ignores the reality that the most damaging confusions are query-specific and shift depending on the support-set geometry. Consequently, negative signals are often diluted across irrelevant classes rather than concentrated on the classes a specific image is most likely to be confused with. Furthermore, current methods rely on generic negative text templates and require manual tuning of the trade-off between visual and textual modalities, limiting their adaptability to fine-grained tasks and distribution shifts.

Methodology: SCAN Framework

The authors propose SCAN (Selective Confusion-Aware Negatives), a parameter-free framework designed to refine few-shot VLM adaptation through three targeted mechanisms:

1. Query-Adaptive Negative Routing

Instead of applying a fixed negative penalty to all classes, SCAN dynamically routes negative suppression at inference time.

Mechanism: For each query image, the system identifies the top- $K$ most confusable classes based on their positive scores.
Routing: Negative suppression is restricted exclusively to this confusion set $N(q)$ . A per-query confusion weight $\beta_c(q)$ is computed using a softmax over the positive scores of the confusable classes.
Benefit: This concentrates the penalty where it matters most, sharpening per-query decision boundaries without introducing additional parameters or training overhead.

2. LLM-Bootstrapped Pairwise Contrastive Prompts

SCAN replaces generic negative templates (e.g., "not a {CLASS}") with discriminative textual supervision.

Generation: For each class, the system identifies its top- $k$ visually confusable neighbors in the CLIP embedding space. An LLM is prompted offline to generate descriptions characterizing the target class against these specific neighbors (e.g., "a photo of $c$ , distinguishable from $c'$ by [visual attribute]").
Integration: These pairwise contrastive descriptions serve as the negative text weights ( $w^-_c$ ), providing grounded discriminative language that goes beyond simple negation of class names.

3. Parameter-Free Adaptive Modality Fusion

The framework eliminates the need for manual hyperparameter tuning regarding the balance between visual and textual branches.

Estimation: The fusion weight $\lambda$ is estimated directly from the support-set geometry.
Logic: It calculates the visual Fisher discriminability ( $\sigma_v$ , ratio of between-class to within-class variance) and textual discriminability ( $\sigma_t$ , derived from pairwise cosine similarity of text weights).
Adaptation: The weight is computed as $\lambda = \text{clip}(\frac{\sigma_v}{\sigma_v + \sigma_t})$ . This naturally favors the visual branch when features are discriminative and the textual branch when class names are semantically close.

Key Contributions

The paper outlines three primary contributions:

Query-Conditioned Negative Routing: A mechanism that concentrates negative suppression on the classes each test image is most likely to be confused with, rather than applying uniform penalties.
LLM-Bootstrapped Contrastive Prompts: The introduction of offline-generated, pairwise contrastive descriptions that characterize classes against their nearest semantic neighbors, replacing generic negative templates.
Adaptive Modality Fusion: A parameter-free method to infer the optimal visual-textual weighting solely from support-set geometry, removing manual tuning requirements.

Experimental Results

SCAN was evaluated across 11 standard benchmarks (including ImageNet, Caltech101, Flowers102, and fine-grained datasets like Stanford Cars and FGVC Aircraft) and four Out-of-Distribution (OOD) variants of ImageNet.

Few-Shot Performance: SCAN consistently outperforms prior prompt-based (CoOp, CoCoOp) and adapter-based (CLIP-Adapter, Tip-Adapter-F, TaskRes) methods.
- At 16-shot, SCAN achieves an average improvement of 4.61% over the strongest baseline (SimNL).
- Gains are most pronounced on fine-grained datasets where inter-class confusion is severe: 7.70% improvement on DTD and FGVC Aircraft, and 5.90% on Stanford Cars.
Distribution Shift Robustness: SCAN demonstrates strong generalization across ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R, showing an average improvement of 2.95% over SimNL. The largest gain (+3.20%) is observed on ImageNet-Sketch.
Label Noise Resilience: Under significant label corruption (up to 50% noise on 16-shot ImageNet), SCAN maintains robust performance. Notably, its accuracy under 50% noise (67.68%) exceeds the clean baseline performance of the strongest competing method (SimNL at 66.52%).

Significance and Claims

The paper claims that SCAN addresses the fundamental limitation of treating all negative signals as equal. By shifting the paradigm to query-specific negative routing, the method effectively sharpens decision boundaries in high-confusion scenarios without increasing inference-time parameters.

The authors assert that targeted improvements to the negative branch and textual supervision can yield substantial, consistent gains across diverse domains. The framework's ability to generalize under distribution shifts and maintain robustness under severe label noise suggests that query-adaptive routing is intrinsically noise-tolerant. The work positions SCAN as a plug-and-play enhancement compatible with existing adapter-based few-shot pipelines, offering a path toward more reliable few-shot vision-language models in real-world settings where data is scarce and noisy.