⚡ electrical engineering

Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models

This paper proposes a distribution-conditioned attribute selection method that extracts visual descriptors directly from target images rather than relying on class-name-conditioned LLM descriptions, thereby significantly improving zero-shot classification robustness across distribution shifts while offering an interpretable summary of dataset characteristics.

Original authors: Gautam Rajendrakumar Gare, Jia Shi, Zhiqiu Lin, Deepak Pathak, John Galeotti, Deva Ramanan

Published 2026-07-22

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Gautam Rajendrakumar Gare, Jia Shi, Zhiqiu Lin, Deepak Pathak, John Galeotti, Deva Ramanan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective, the Description, and the Sketch

Imagine you are a detective trying to identify a suspect from a lineup of photos. You have a powerful assistant, an AI, who is incredibly good at matching words to pictures. This AI has seen millions of photos and knows that "dogs" usually have fur and "strawberries" are usually red. This is the world of Vision-Language Models: computers that can look at an image and understand the words describing it, or look at a word and find the matching image.

But here's the tricky part: sometimes the photos you have aren't the usual, high-definition, colorful snapshots. Maybe they are black-and-white sketches, or paintings, or photos taken in weird lighting. If your AI assistant relies only on a textbook definition of a "strawberry" (red, round, with seeds), it might get confused when it sees a strawberry drawn as a gray line on a piece of paper. It might think, "This isn't red, so it can't be a strawberry!"

For a while, researchers thought the best way to help the AI was to ask a super-smart language bot (like a very advanced chatbot) to write a detailed description of every object. They thought, "If we tell the AI that strawberries are red and have seeds, it will do a great job." But this paper asks a simple, nagging question: Is the AI actually looking at the picture to make its decision, or is it just blindly trusting the description we gave it? The authors of this study decided to put this idea to the test, and what they found changes how we should teach these digital detectives.

The Great "Red Strawberry" Mistake

The story starts with a popular trick used by scientists. To make AI better at recognizing things without showing it thousands of examples (a process called "zero-shot learning"), they ask a language model to describe a class of objects. For "strawberry," the model might say, "It is red, has green leaves, and is covered in seeds." They then feed this description to the vision AI along with the class name.

The paper reveals a shocking flaw in this method. When the AI sees a normal photo of a strawberry, it gets it right about 60% of the time. But the authors realized the AI wasn't actually looking at the visual evidence of the strawberry's redness; it was just relying on the word "strawberry" to do the heavy lifting. To prove this, they played a game of "remove the name." They took the description ("red, has seeds") and fed it to the AI without telling it the word "strawberry."

The result was a disaster. The accuracy didn't just drop a little; it crashed from 59.5% down to 15.5%. It was as if the AI suddenly forgot what a strawberry looked like the moment you stopped saying the word. The description was useless on its own.

Why? Because the language model that wrote the description had never seen the specific photos the AI was looking at. It was describing a "perfect" strawberry from its training data. But in the real world, specifically in a dataset called ImageNet-Sketch, the strawberries were all black-and-white line drawings. The language model insisted, "Strawberries are red!" while the AI looked at a gray sketch and thought, "This isn't red, so it's not a strawberry." The description was fighting against the image, not helping it.

The Solution: Let the Image Speak

The authors' diagnosis was simple: the attributes (the descriptions) were being chosen based on the name of the object, not the image of the object. They argued that if you want to describe a group of pictures, you shouldn't ask a book what a strawberry looks like; you should ask the pictures themselves.

So, they built a new method. Instead of asking a language bot to guess the features, they let the AI look at a few actual images from the target group (like the sketchy strawberries) and ask, "What words from our giant list of possible descriptions match these specific pictures?"

They took a huge pool of potential words (like "red," "round," "sketchy," "colorless," "dotted") and scored them against the actual images. For the sketchy strawberries, the words "red" and "ripe" got low scores because the pictures weren't red. But words like "heart-shaped," "dotted," and "colorless" got high scores because they actually matched the drawings.

The Results: A New Kind of Detective

When they used these image-selected words to help the AI, the results were much more impressive, especially when the class name was removed.

The Old Way (LLM Descriptors): Without the class name, accuracy was 15.5%.
The New Way (Image-Selected): Without the class name, accuracy jumped to 23.8% using the same pool of words.

This proved that the method of selection was the hero, not just the vocabulary. By letting the images choose the words, the AI could handle weird situations much better.

The paper also showed that this method is incredibly fast and efficient.

Speed: It took less than one minute to set up the new system.
Comparison: A competing method called CoOp, which tries to "tune" the AI by adjusting invisible numbers, took 14 hours to train and still performed worse when there was only one image per class (60.13% vs 57.15%).
Few-Shot Learning: With just one image per class, the new method beat the 14-hour training method. This is huge because it means you don't need a supercomputer or days of waiting to get a smart, readable system.

Why This Matters: Reading the Data's Mind

The coolest part of this discovery is that the selected words act like a summary of the data itself. Because the words are chosen based on what the images actually look like, you can use them to describe how a dataset has changed.

For example, if you compare the words chosen for normal photos versus the words chosen for the sketchy photos, the difference tells a story. The normal photos might highlight "red" and "green," while the sketchy photos highlight "colorless," "gray," and "cartoon." You can literally subtract one list of words from the other to get a sentence that explains the difference between the two groups of images.

The authors also showed this works for things that are hard to name. If you have a picture of a weird fruit with a scientific name that the AI doesn't know (like Cristidiscoidea), the AI usually fails. But if you show it a few pictures and let it pick the visual words (like "spiky," "yellow," "small"), it can recognize the object even without knowing the fancy name.

The Bottom Line

This paper suggests a major shift in how we build AI. We shouldn't just ask language models to describe things based on their names; we should let the images teach us what words matter.

What they ruled out: They proved that simply adding a language-generated description to a prompt doesn't actually help the AI "see" the image better; the AI was just guessing based on the class name.
What they found: Selecting attributes directly from the images makes the AI more robust, faster, and more interpretable.
The Catch: While this method is great for weird or shifted data, it doesn't completely replace the need for class names in standard situations. The best results still come from combining the class name with the image-selected words. But for understanding why an AI made a mistake, or for handling data that looks nothing like the training set, letting the image choose the words is the winning strategy.

In short, if you want your AI to understand the world, don't just tell it what things should look like. Show it what they actually look like, and let it tell you the story.

Technical Summary: Distribution-Conditioned Attribute Selection for Vision-Language Models

Problem Statement
Current approaches to interpretable zero-shot classification often employ Large Language Models (LLMs) to generate textual descriptors for class names (e.g., prompting an LLM to describe a "strawberry" as "red" and "having seeds"). These descriptors are then averaged to form prompts for Vision-Language Models (VLMs) like CLIP. The paper identifies a critical flaw in this paradigm: these descriptors are conditioned solely on the class name and general knowledge, not on the target image distribution.

The authors demonstrate that when the class name is removed from the prompt, leaving only the LLM-generated descriptors, ImageNet accuracy collapses from 59.5% to 15.5%. This indicates that the descriptors themselves provide little visual evidence; the performance relies heavily on the class name acting as a semantic anchor. Furthermore, under distribution shift (e.g., ImageNet-Sketch), these label-conditioned descriptors fail catastrophically because they describe the canonical concept (e.g., "red strawberries") rather than the actual data (e.g., "colorless line drawings"), actively misleading the classifier.

Methodology: Distribution-Conditioned Attribute Selection
The proposed solution shifts the conditioning variable from the class name to the target images. The method, termed Distribution-Conditioned Attribute Selection, operates as follows:

Attribute Pooling: A large, fixed pool of candidate attribute strings is aggregated from existing datasets (VAW, LSA) and LLM-generated descriptors. Explicit class names and duplicates are removed from this pool.
Image-Conditioned Scoring: Using a frozen CLIP model, the method encodes the target image collection and the attribute text pool. For each image, it computes a vector of scaled cosine similarities between the image embedding and all attribute text embeddings.
Class-Attribute Pairing: For each class, a score is estimated for how well each attribute describes the class. This is done either by averaging the attribute scores over the class's training images or by training a simple linear probe on the attribute features.
Selection: The top- $k$ attributes with the highest scores for each class are selected to form the class's prompt set.
Classification: During inference, a test image is classified by matching its embedding against the selected attribute prompts. The prompts can be used with or without the class name.

Crucially, the attribute encoders remain frozen, and no image-text pairs are used for training; the only learnable component is a linear probe used for ranking, which requires minimal labeled data.

Key Contributions

Diagnosis of the Class-Name Confound: The authors establish that LLM-generated descriptors derive their utility almost entirely from the class name. Removing the name causes a 44-point accuracy drop, proving that standard descriptor prompts do not provide genuine attribute-level evidence.
Image-Conditioned Selection Mechanism: The paper proposes a training-light procedure that selects attributes based on the target image distribution rather than the label. By holding the attribute pool constant and changing only the selection mechanism, the authors show that conditioning on images recovers 3.9 percentage points of accuracy over the LLM pool alone (15.5% to 19.4%) and achieves 23.8% with a larger pool (VAW+LSA) in an attribute-only setting.
Interpretable Feature Space: The selected attributes serve as a readable summary of the dataset. The authors demonstrate that subtracting the mean attribute profiles of two datasets yields a word-based description of distribution shift (e.g., identifying "colorless" and "cartoon" as dominant shifts in ImageNet-Sketch).

Results

Attribute-Only Performance: Without the class name, the proposed method achieves 23.8% accuracy on ImageNet (using VAW+LSA pool and top-5 attributes), significantly outperforming the 15.5% baseline of LLM descriptors. With the attribute budget uncapped and a weighted voting scheme, accuracy reaches 45.5%.
Distribution Shift: The method maintains its relative advantage across four shifted ImageNet variants (V2, Sketch, A, R), with the largest margin observed on ImageNet-R.
Few-Shot Learning: With only one image per class, the method achieves 60.13% accuracy, outperforming CoOp (57.15%) by 3 points. It fits in under a minute compared to CoOp's 14 hours. However, the method plateaus at ~61.4% accuracy as the number of shots increases, whereas gradient-based methods like CoOp continue to improve, indicating the selection approach is best suited for extreme few-shot regimes.
Applications:
- Shift Description: The method successfully quantifies domain gaps in words (e.g., describing ImageNet-R as "cartoon" and "painting").
- Color Shift: In a fruit classification task with color variants, the method mitigates CLIP's tendency to latch onto color, recovering 4 points of accuracy lost to color shift.
- Unparseable Vocabularies: For scientific names (iNaturalist) that CLIP's text encoder cannot parse, the method improves accuracy from 3% to 7.25% by replacing the name with visual attributes.
- Localization: The method enables class-name-free object localization by computing attribute similarities on image patches.

Significance and Claims
The paper argues that for interpretable zero-shot classification, attributes must be derived from the data distribution, not the label. The authors claim that their method provides a strong, human-readable alternative to prompt tuning in the extreme few-shot regime, requiring no learned soft prompts.

The significance extends beyond the specific algorithm: the authors propose a protocol change for the field. They argue that any work claiming attribute-level interpretability via descriptor prompts must report class-name-free accuracy, as the standard protocol (including the class name) conflates attribute evidence with prompt ensembling. Furthermore, for concept-bottleneck pipelines, the paper suggests that concept pools should be selected against the deployment distribution rather than generated from labels to ensure robustness under distribution shift.

The authors remain modest regarding limitations: attribute-only performance does not yet match class-name prompting (23.8% vs 60.8%), and the method does not claim to solve medium-shot learning where gradient-based methods excel. Additionally, the method's performance is bounded by the quality and relevance of the pre-existing attribute pool.