🤖 AI

First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models

This paper proposes First Logit Boosting (FLB), a simple, training-free technique that mitigates object hallucination in Large Vision-Language Models by preserving initial visual information throughout generation to counteract long-term decay, all while incurring negligible inference overhead.

Original authors: Jiwoo Ha, Jongwoo Baek, Jinhyun So

Published 2026-04-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiwoo Ha, Jongwoo Baek, Jinhyun So

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, artistic robot named VisionBot. VisionBot is great at looking at a picture and describing what it sees. If you show it a photo of a dog, it might say, "A fluffy dog is sleeping on a rug."

But VisionBot has a weird habit: sometimes, as it keeps talking, it starts hallucinating. It might look at the same dog photo and suddenly say, "A fluffy dog is sleeping on a rug, and there is a unicorn eating a pizza next to it."

The problem is that the unicorn and pizza aren't there. VisionBot just made them up because it got bored of looking at the dog and started guessing based on what it thinks usually happens in stories.

The Problem: "Long-Term Decay"

The researchers found that VisionBot gets this way because of how it remembers things.

At the very beginning: When it starts talking, it looks right at the picture. It's super focused. "I see a dog."
A few sentences later: As it keeps talking, its "eyes" (attention) drift away from the picture. It starts relying more on its internal storybook (language patterns) than the actual photo.
The result: The further it gets into the sentence, the more likely it is to invent things that aren't there. This is called Long-Term Decay.

The Old Solutions (The "Heavy" Fixes)

Before this new paper, people tried to fix VisionBot in two ways:

Retraining: Teaching the robot from scratch with millions of new examples. This is like sending the robot back to school for a whole year. It's expensive and slow.
Double-Checking: Making the robot look at the picture twice for every word it says (once normally, once with a "distorted" picture) to compare notes. This is like asking a friend to read your essay while you write it. It works, but it makes the robot twice as slow.

The New Solution: First Logit Boosting (FLB)

The authors of this paper came up with a clever, lightweight trick called First Logit Boosting (FLB). Think of it as giving VisionBot a permanent anchor.

Here is how it works, using a simple analogy:

1. The "Anchor" Effect (Direct Visual Grounding)

Imagine you are describing a photo of a man in a hat.

The very first word VisionBot says is usually something like "The" or "A".
At that exact split second, the robot is looking hard at the photo. It knows, 100%, that the subject is a "man."
FLB's Trick: The robot saves that initial "Man" feeling (the math behind that first word) in a little pocket. Then, as it keeps writing the sentence, it secretly adds that "Man" feeling back into every new word it generates.
The Result: Even when the robot gets bored and starts drifting, that "Man" anchor pulls it back to the photo. It stops the robot from forgetting that the subject is a man and not a woman or a unicorn.

2. The "The" Effect (Implicit Visual Referencing)

The researchers noticed something funny. The first word VisionBot usually picks is "The".

When a sentence starts with "The," it implies, "I am talking about something specific that we already know."
When it starts with "A," it implies, "I am introducing something new."
FLB's Trick: By boosting the "The" feeling, the robot is more likely to say, "The man is juggling..." instead of "A woman is juggling..."
Why this helps: "The" forces the robot to stick to the things it has already seen and confirmed. It acts like a safety belt, keeping the robot from wandering off into imaginary territory.

Why is this a Big Deal?

It's Free: You don't need to retrain the robot or buy new hardware. It's just a software tweak.
It's Fast: Unlike the old "Double-Check" method, FLB doesn't slow the robot down at all. It's like adding a GPS to a car without making the car heavier.
It Works Everywhere: They tested it on different robot brains, and it stopped the hallucinations (the fake unicorns and pizzas) without making the sentences sound robotic or broken.

Summary

Imagine you are telling a story about a photo. As you talk, you might forget the details and start making things up.
First Logit Boosting is like having a friend whisper in your ear at the start of the story: "Remember, it's a man in a hat!" and then that friend keeps whispering that reminder gently throughout the whole story so you never forget the truth.

It's a simple, smart, and fast way to keep AI honest about what it's actually seeing.

1. Problem Statement

Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks like image captioning and Visual Question Answering (VQA). However, they suffer from object hallucination, where the model generates descriptions of objects that are not present in the input image.

The paper identifies two primary limitations in existing mitigation strategies:

Training-based & External Methods: Approaches like Reinforcement Learning from Human Feedback (RLHF) or using external grounding models are effective but incur high data costs, computational overhead, or structural complexity.
Training-free Methods (e.g., Contrastive Decoding): While efficient, methods like Visual Contrastive Decoding (VCD) suffer from long-term decay. As text generation progresses, the model's attention drifts away from visual evidence and becomes dominated by language priors, causing hallucinations to increase in later tokens. Furthermore, these methods often require dual forward passes per decoding step (one for the original input, one for a perturbed input), doubling inference time.

2. Methodology: First Logit Boosting (FLB)

The authors propose First Logit Boosting (FLB), a simple, training-free technique designed to mitigate long-term decay without additional inference passes.

Core Mechanism

FLB operates on the observation that the first generated token in an LVLM contains the strongest visual grounding because it is generated immediately after visual tokens, before positional decay (caused by Rotary Positional Embeddings, RoPE) weakens cross-modal attention.

The method involves three steps:

Logit Storage: At the first decoding step ( $t=0$ ), the model generates logits based on the image and prompt. These logits ( $l_0$ ) are stored.
Logit Reuse: For all subsequent decoding steps ( $t > 0$ ), the stored logits $l_0$ are added to the current step's logits.
Weighted Boosting: The contribution of the first logit is scaled by a temporal weighting function $w_t$ that increases over time to counteract the growing influence of language priors:
$w_t = \gamma(1 - e^{-\lambda t})$
where $\gamma$ is the maximum scaling coefficient and $\lambda$ controls the rate of increase.

The final probability distribution is calculated as:
$p(y_t) \propto \text{softmax}(\text{logit}_\theta(y_t | v, x, y_{<t}) + w_t \cdot l_0)$

Adaptive Plausibility Constraint

To prevent the reuse of first-token logits from causing unnatural token selections (e.g., inserting "The" in the middle of a sentence), FLB applies an adaptive constraint. It filters out candidate tokens that fall below a certain probability threshold ( $\beta$ ) relative to the maximum probability in the original distribution.

3. Key Contributions & Theoretical Insights

The paper identifies two complementary effects driving FLB's success:

Direct Visual Grounding Effect:
- The first token's logit inherently carries the strongest visual evidence. By reusing this logit, FLB acts as a persistent "visual anchor," continuously reinforcing the visual signal throughout the generation process and preventing the decay of visual attention.
The "The" Effect (Implicit Visual Referencing):
- Empirical analysis shows that the first token is frequently an article like "The."
- Sentences starting with "The" exhibit significantly lower hallucination rates compared to those starting with "A" or other tokens.
- "The" acts as an implicit visual reference, encouraging the model to refer back to entities already established in the context (which are visually grounded) rather than hallucinating new, ambiguous objects.
- Entropy analysis reveals that tokens following "The" have lower prediction entropy, indicating higher model confidence and reduced uncertainty.

4. Experimental Results

The authors evaluated FLB on multiple benchmarks (CHAIR, AMBER, MMHalBench, ConvBench) using backbone models like LLaVA-1.5 and InstructBLIP.

Hallucination Mitigation: FLB significantly outperformed state-of-the-art training-free baselines (VCD, ICD, M3ID).
- On the AMBER benchmark, FLB reduced the CHAIR score (proportion of hallucinated objects) from ~11.5 (Baseline) to 6.1 for LLaVA-1.5, a substantial improvement over VCD (9.9) and ICD (9.1).
- On CHAIR, FLB achieved the lowest hallucination rates (CHAIRs: 43.5 vs. Baseline 57.5) while maintaining high recall of ground truth objects.
Inference Efficiency: Unlike Contrastive Decoding methods that require two forward passes, FLB requires only one forward pass. Consequently, FLB maintains inference speeds nearly identical to the baseline, whereas CD methods are roughly 2x slower.
Quality Preservation: Qualitative evaluations using GPT-4V confirmed that FLB does not degrade sentence fluency, grammaticality, or expression diversity. The generated captions remained natural and detailed.
Generalization: FLB demonstrated robustness across different model architectures (MLP projector vs. Q-Former) and task types (captioning, multi-turn conversation, diverse Q&A).

5. Significance

Efficiency: FLB offers a highly practical solution for real-time multimodal systems by eliminating the computational overhead associated with dual-pass decoding methods.
Simplicity: It requires no retraining, no external models, and no complex architectural changes, making it easily deployable on existing LVLMs.
Mechanism Insight: The paper provides a novel understanding of how "long-term decay" affects LVLMs and how linguistic initialization (specifically the "The" token) can serve as a stabilizing mechanism for visual grounding.
State-of-the-Art Performance: It sets a new benchmark for hallucination mitigation, achieving superior results in reducing object hallucinations while preserving the accuracy and coverage of ground truth objects.

In conclusion, First Logit Boosting represents a significant step forward in making LVLMs more reliable for safety-critical applications by effectively addressing the root cause of hallucination (long-term decay) with minimal computational cost.