Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

Imagine you are talking to a super-smart robot friend who has seen millions of pictures and read billions of books. You show it a photo of a sunny park, and it starts describing the scene. But sometimes, this robot gets a little too creative. It might confidently tell you there's a "giant red balloon" floating above the trees, even though the photo is completely empty there. This is called "hallucination," and it's a big headache for the scientists building these AI systems. If a robot thinks it sees things that aren't there, we can't really trust it to help us with important tasks.

For a long time, researchers thought the problem was that the robot wasn't "looking" hard enough at the picture. They believed the AI was just daydreaming because its eyes were wandering, so they tried to force it to pay more attention to the image. But what if the robot is looking? What if it's staring right at the spot where the fake balloon should be, but it's just misinterpreting what it sees? This paper, titled "Same Attention, Different Truths," dives into that exact mystery. It uses a clever trick called "Logit Lens" (think of it as an X-ray for the robot's brain) to peek inside the AI's mind while it's talking. The goal is to figure out why the robot lies about what it sees, so we can teach it to tell the truth without needing to retrain it from scratch.

The Mystery: Staring Hard, But Getting It Wrong

The authors started by testing a popular idea: that AI hallucinations happen because the model doesn't pay enough attention to the image. They watched the AI as it described pictures, tracking exactly where its "gaze" (attention) landed. They found something surprising: the AI paid just as much attention to the real objects (like a chair) as it did to the fake ones (like a bowl that wasn't there).

It's like a detective who is staring intensely at a blank wall, convinced there's a secret message hidden there. The detective isn't ignoring the wall; they are looking right at it! The problem isn't how much they look, but what they think they see. The paper calls this "Same Attention, Different Truths." The AI is focusing its energy, but the connection between what it sees and what it says is broken.

The X-Ray: Reading the AI's Mind

To understand what was going on, the researchers used a tool called Logit Lens. Imagine the AI's brain as a multi-layered cake. As the AI processes an image, the information travels through these layers. The Logit Lens lets the researchers peek at the "filling" of the cake at different layers and ask, "If we stopped the AI right here, what word would it say?"

When they did this, they found a clear split:

• Real Objects: When the AI was looking at a real chair, the "filling" of its brain correctly decoded the image as "chair." The visual evidence matched the word.
• Fake Objects: When the AI was looking at a blurry patch of ground and decided to call it a "bowl," the Logit Lens showed that the brain didn't actually see a bowl. The visual evidence was messy and uncertain, but the AI insisted on saying "bowl" anyway.

Two Types of Liars

The paper discovered that the AI lies for two very different reasons, like two different types of students taking a test.

1. The "Blurry Vision" Liar (Visual Uncertainty)
Sometimes, the image is just confusing. Maybe there's a dark, fuzzy shape that looks a bit like a bowl. The AI stares at this blurry spot, gets confused, and guesses, "I bet that's a bowl!"

• The Fix: The researchers found that if they simply covered up (masked) that blurry, confusing spot, the AI stopped lying. It realized, "Oh, I can't see anything there, so I won't guess." This is called High-Attention Regions Masking (HARM). It's like telling the detective, "Don't look at that blurry wall; look at the clear window instead."

2. The "Daydreaming" Liar (Contextual Prior)
Other times, the image is clear, but the AI is too influenced by what it expects to see. Imagine the AI is describing a kitchen. Even if there is no microwave in the picture, the AI might say, "There's a microwave," because in its training, kitchens usually have microwaves. It's so used to the idea of a microwave that it hallucinates one.

• The Twist: If you cover up the part of the image the AI is looking at, it doesn't stop lying! It just shifts its gaze to another spot and keeps saying "microwave." The AI is following a script: "I must look at something before I say 'microwave'."
• The Fix: For this type, the researchers used a method called Visual Evidence Enhanced Decoding (VEED). They forced the AI to pay extra attention to the actual visual evidence it had already found (which, in this case, didn't show a microwave) and used that to override its daydreaming. It's like reminding the detective, "Check your notes again; the notes say there is no microwave, so stop guessing."

The Solution: A Detective's Toolkit

Based on these findings, the authors built a simple, "training-free" toolkit. This means they didn't have to re-teach the AI from scratch; they just added a smart check-up step before the AI gives its final answer.

1. The Check-Up: As the AI generates a sentence, the system uses the Logit Lens to check: "Does the picture actually support this word?" If the answer is no, it flags the word as a hallucination.
2. The Diagnosis: It then figures out why the AI lied. Is it because the image was blurry (Type 1) or because the AI was daydreaming (Type 2)?
3. The Cure:
   • If it's Type 1, it masks the blurry part and asks the AI to try again.
   • If it's Type 2, it boosts the signal of the real visual evidence to drown out the daydream.

The Results

The team tested this method on several different AI models and found it worked incredibly well. It reduced the number of fake objects the AI invented more than any other method they tested, without making the AI forget to mention the real objects.

In short, this paper shows that AI hallucinations aren't just about the robot not looking hard enough. Sometimes, it's looking right at the problem but misinterpreting a blurry clue, and other times, it's just too eager to follow a story it's heard before. By understanding these two different "lies," the researchers found a way to catch them and fix them, making our robot friends a little more honest and a lot more reliable.