Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
The paper proposes GIFT, a gaze shift-guided cross-modal fusion method that mitigates Vision-Language Model hallucinations by dynamically amplifying attention to salient visual regions and user queries based on positive attention changes, thereby reducing visual attention sinks and improving performance with low computational overhead.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Problem: The "Daydreaming" AI
Imagine you have a very smart, well-read assistant who loves to tell stories. You show them a photo of a cat sitting on a mat and ask, "What is the cat doing?"
Because this assistant has read millions of books about cats, they might confidently say, "The cat is chasing a red mouse!" even though there is no mouse in the picture. They are hallucinating. They are relying too much on what they think should be there (their "prior knowledge") rather than looking closely at what is actually there (the visual input).
Current methods try to fix this by telling the AI, "Look harder at the picture!" But the paper argues these methods have two flaws:
- They look at the wrong spots: Sometimes the AI gets distracted by background noise (like a blurry wall) instead of the cat. This is called the "Visual Attention Sink." It's like a student staring at a smudge on their textbook page instead of the math problem.
- They forget the question: If you just tell the AI "Look at the picture!" without reminding them what to look for, they might get confused. They need to balance looking at the image with understanding your specific question.
The Solution: GIFT (The "Gaze Shift" Tracker)
The authors propose a new method called GIFT (Gaze Shift-Guided Cross-modal Fusion Enhancement).
To understand GIFT, imagine a human detective looking at a crime scene photo while reading a witness statement.
- Old Way (Static Gaze): The detective takes a snapshot of the whole photo and averages their attention. They might get distracted by a random chair in the corner because it's in the middle of the photo.
- The GIFT Way (Gaze Shift): The detective reads the witness statement word by word.
- When they hear the word "Red," their eyes jump (shift) to the red fire hydrant.
- When they hear the word "Dog," their eyes jump to the dog.
- When they hear a word that doesn't matter (like "the" or "and"), their eyes don't move much.
GIFT does exactly this for the AI. It watches how the AI's "eyes" (attention) move when it reads the user's question. It ignores the parts where the AI's gaze is steady or drifting aimlessly. It only pays attention to the moments when the AI's gaze shifts positively toward a new part of the image because a specific word in the question triggered it.
How It Works (The Three Steps)
1. Mapping the "Gaze Shifts" (Pre-computation)
Before the AI starts writing its answer, GIFT runs a quick simulation. It asks: "As the AI reads the word 'motorcycle' in your question, where do its eyes jump to in the image?"
It creates a Saliency Map (a heat map).
- The Magic: Because it only tracks changes (shifts), it naturally ignores the "Visual Attention Sink" (the background noise). If the AI's eyes didn't move to a spot when reading the question, that spot isn't on the map. It's like filtering out the static on a radio to hear the music clearly.
2. Guiding the Answer (Decoding)
Now, the AI starts generating the answer. GIFT uses that heat map to guide the process:
- Boost the Visuals: If the heat map says "The motorcycle is important," GIFT turns up the volume on the AI's attention to the motorcycle part of the image.
- Boost the Question: Crucially, GIFT also turns up the volume on the question itself. It ensures the AI doesn't get so obsessed with the image that it forgets what you asked. It keeps the conversation balanced.
3. The Result
The AI is now like a detective who is perfectly focused on the relevant clues and remembers the exact question. It is much less likely to invent a "red mouse" that isn't there.
The Results: Faster and Smarter
The paper tested this on several AI models (like LLaVA and Qwen) and found:
- Less Hallucination: The AI made up fewer fake objects. On some tests, it improved accuracy by over 20%.
- Still Smart: It didn't lose its ability to reason or answer general questions. It didn't become a "dumb" image recognizer; it just became a more honest one.
- Fast: It didn't slow the AI down much. It only added about 13% to the time it takes to generate an answer, which is very efficient compared to other methods that can be 10x slower.
Summary Analogy
Think of the AI as a tour guide in a museum.
- The Problem: The guide knows so much history that when you ask about a painting, they start reciting facts about a different painting they think you meant, or they invent details about the frame that aren't there.
- The Old Fix: "Look at the painting!" (But the guide keeps looking at the floor or the ceiling).
- The GIFT Fix: The guide has a special pair of glasses. When you say "The blue vase," the glasses instantly highlight the blue vase in their vision and dim the rest of the room. At the same time, the glasses remind the guide, "Don't forget, the guest asked about the blue vase, not the red one."
By tracking exactly where the guide's attention moves when hearing your words, GIFT ensures they describe exactly what is in front of them, without making things up.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.