💬 NLP

Visual Word Sense Disambiguation with CLIP through Dual-Channel Text Prompting and Image Augmentations

This paper presents an interpretable Visual Word Sense Disambiguation framework that leverages CLIP with dual-channel text prompting and image augmentations to resolve lexical ambiguity, achieving significant performance improvements on the SemEval-2023 dataset while demonstrating that precise, CLIP-aligned prompts are more effective than noisy external signals.

Original authors: Shamik Bhattacharya, Daniel Perkins, Yaren Dogan, Vineeth Konjeti, Sudarshan Srinivasan, Edmon Begoli

Published 2026-02-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shamik Bhattacharya, Daniel Perkins, Yaren Dogan, Vineeth Konjeti, Sudarshan Srinivasan, Edmon Begoli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a game of "Guess the Picture." Someone gives you a word that has two very different meanings, like the word "bank."

Does it mean the place where you keep your money?
Or does it mean the muddy edge of a river?

If I just say "bank," you might be confused. But if I say "river bank," your brain instantly picks the river picture. This paper is about teaching a computer to do the same thing, but with a twist: instead of just reading more words, the computer has to look at 10 different pictures and pick the one that matches the meaning of the word.

Here is how the researchers solved this puzzle, explained simply:

The Problem: The Computer's "Blurry Glasses"

Large language models (the smart AI brains) are great at reading, but they sometimes get confused when a word has multiple meanings. If the sentence is short, the computer might not know which "bank" you mean. It's like trying to identify a person in a foggy room; you can see a shape, but you aren't sure if it's your friend or a stranger.

The Solution: A Two-Part Strategy

The researchers built a system using a tool called CLIP (which is like a super-smart translator that speaks both "English" and "Image"). They tried to make the computer's "glasses" clearer using two main tricks:

1. The "Dual-Channel" Prompt (The Verbal Hint)

Instead of just feeding the computer the raw sentence (e.g., "bank erosion"), they gave it a set of hints to help it focus. They used two types of hints:

The "Semantic" Channel: These are like dictionary definitions but written as short, clear phrases (e.g., "the concept of a river bank").
The "Photo" Channel: These are prompts that describe what the picture looks like (e.g., "a photo of a river bank with erosion").

Think of this as a detective giving the computer a list of clues: "Look for a river, not a building." By mixing these clues together, the computer gets a much sharper idea of what it's looking for.

2. The "Image Augmentation" (The Kaleidoscope)

For the pictures, the computer doesn't just look at the image once. The researchers used a technique called augmentation.
Imagine you are looking at a painting. To understand it better, you might:

Zoom in on the details.
Look at it from the side.
Squint your eyes to see the shapes.
Turn it upside down.

The computer did the same thing. It took each of the 10 candidate images and created 28 different "versions" of them (cropped, flipped, brightened, etc.). It then averaged all these views together to get a "super-view" of the image. This helps the computer ignore distractions like a weird shadow or a person standing in the corner.

The Experiment: What Worked and What Didn't?

The researchers tested these tricks on a dataset called SemEval-2023, which is basically a giant test bank of tricky words and pictures.

What Worked Best:

The Verbal Hints (Prompts): This was the winner. Giving the computer better descriptions (the "Dual-Channel" hints) made it much smarter. It was like giving the detective a better map. This improved the computer's accuracy significantly without slowing it down much.
The "Kaleidoscope" (Image Augmentation): Looking at the pictures from many angles helped a little bit, but it was expensive. It took a lot of computing power to process 28 versions of every image, and the improvement in accuracy was very small. It was like using a sledgehammer to crack a nut.

What Didn't Work:

Adding More Languages: The researchers tried translating the hints into Spanish, French, and German, hoping that different languages would clarify the meaning. Instead, it made things worse. It was like trying to solve a puzzle while wearing noise-canceling headphones that were playing static; the extra information just created confusion.
Adding Dictionary Definitions: They tried feeding the computer the official dictionary definition of the word. While it helped a tiny bit, relying too much on the dictionary actually made the computer forget the context of the sentence.

The Final Result

By combining the smart verbal hints with a moderate amount of image tweaking, the researchers built a system that got better at picking the right picture.

Before: The computer got the right answer about 58% of the time.
After: The computer got the right answer about 62% of the time.

The Takeaway

The main lesson from this paper is that quality beats quantity.

Giving the computer a few, very precise verbal clues (prompts) was far more effective than throwing a massive amount of extra data (like translations or dictionary definitions) at it.
Trying to see the image from every possible angle (aggressive augmentation) cost too much time and energy for very little gain.

In short, the best way to help an AI understand a tricky word is to give it a clear, focused description of what to look for, rather than overwhelming it with too many options or too much noise.

Technical Summary: Visual Word Sense Disambiguation with CLIP through Dual-Channel Text Prompting and Image Augmentations

Problem Statement
Lexical ambiguity remains a persistent challenge for Large Language Models (LLMs), particularly when contextual clues are insufficient to distinguish between polysemous (related meanings) or homonymous (unrelated meanings) words. While traditional Word Sense Disambiguation (WSD) relies on surrounding text, it often fails in low-context scenarios. Visual Word Sense Disambiguation (VWSD) addresses this by leveraging visual inputs to resolve ambiguity, aiming to identify which image from a candidate set best depicts the intended sense of a target word within a specific context. Existing state-of-the-art approaches often rely on complex multimodal fusion or large external generators, making it difficult to isolate the specific contributions of prompting strategies versus visual processing enhancements.

Methodology
The authors propose an interpretable VWSD framework built upon the CLIP (Contrastive Language-Image Pre-training) architecture. The core objective is to project ambiguous text and candidate images into a shared multimodal space to determine alignment via cosine similarity. The proposed architecture enhances the standard "vanilla" CLIP approach through three primary mechanisms:

Dual-Channel Text Prompting: Instead of using raw context phrases, the model enriches textual embeddings using an ensemble of two prompt types:
- Semantic Prompts: Narrowly defined phrases describing the target word from specific semantic angles (e.g., "bank related to erosion").
- Photo Prompts: Phrases that explicitly link the text to visual descriptions (e.g., "a photo of bank erosion").
  These prompts are generated, embedded, averaged within their respective channels, and fused via a weighted sum ( $\beta_p$ and $\beta_s$ ) to create a robust textual representation.
WordNet Integration: To further diversify input, the system incorporates WordNet synonyms for both the target word and the context word. These synonyms are used to generate additional photo prompts, which are infused into the mean photo embedding to refine the final textual vector.
Image Augmentation Pipeline: To improve the robustness of visual embeddings, the model replaces single-view encoding with a comprehensive test-time augmentation strategy. This includes:
- Geometric Transformations: Horizontal mirroring, center cropping, zooming, and random rotations.
- Photometric Transformations: Brightness, contrast, and saturation jitter, along with mild blur.
- Multi-Crop Strategies: Deterministic crops (grid patches, mid-quadrant crops) to capture off-center objects.
  Each candidate image is transformed into multiple views (e.g., 28 views), encoded independently, and averaged to produce a stable, view-averaged image embedding.

The final prediction is made by calculating the cosine similarity between the enriched textual embedding and the enriched image embeddings, selecting the image with the highest score.

Key Contributions and Experimental Findings
The paper evaluates its approach on the SemEval-2023 VWSD dataset, comparing the proposed method against a vanilla CLIP baseline and various ablation configurations.

Performance Gains: Enriching embeddings with dual-channel prompting and image augmentations improved the Mean Reciprocal Rank (MRR) from 0.7227 to 0.7590 and the Hit Rate from 0.5810 to 0.6220 on the test set.
Ablation Insights:
- Prompting vs. Augmentation: The ablation studies reveal that dual-channel prompting provides strong, low-latency performance (MRR ~~0.75, Hit@1 ~0.625) with an end-to-end latency of ~63ms. In contrast, aggressive image augmentation alone yielded only marginal gains (MRR ~0.72) while incurring a significant computational cost (~~710ms latency).
- Diminishing Returns: Combining both prompting and augmentation did not yield additive benefits; the combined approach performed worse than prompting alone in terms of efficiency and offered limited performance gains, suggesting diminishing returns from visual diversity when semantic guidance is already present.
External Knowledge Limitations:
- WordNet Definitions: Integrating WordNet definitions showed that while a weighted average (15% definition, 85% context) offered slight improvements, replacing the context entirely with definitions degraded performance.
- Multilingual Translation: Experiments translating prompts into Spanish, French, and German consistently degraded performance. The authors attribute this to translation errors and the "dilution" of semantic specificity when averaging embeddings across languages.
Baseline Comparison: A baseline using BERT for text and BLIP for image captioning failed (0% hit rate), underscoring the necessity of a unified embedding space provided by CLIP for meaningful cross-modal comparison.

Significance and Claims
The paper positions its significance not in achieving the absolute highest accuracy compared to the top SemEval-2023 systems (which achieved ~84-85% Hit@1 using larger, more complex models), but in providing an interpretable framework that isolates the contributions of specific design choices.

The authors claim that:

Precision over Noise: For CLIP-based VWSD, precise, monolingual, CLIP-aligned prompts are more effective than noisy external signals like multilingual translations or excessive WordNet definitions.
Efficiency vs. Complexity: Semantic prompting is a computationally efficient strategy that offers significant disambiguation power, whereas aggressive visual augmentation introduces substantial overhead with limited returns.
Modularity: The study demonstrates that a modular approach, avoiding complex fusion mechanisms (like GNNs or vision-LLM stacks), can effectively evaluate how low-cost design choices impact validation metrics under strict latency budgets.

The authors conclude that while a gap remains between their results and state-of-the-art systems, their work establishes a foundation for understanding component-level contributions to disambiguation, prioritizing interpretability and sustainable inference costs.