💬 NLP

Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding

The paper introduces ReVisiT, a training-free decoding method that mitigates hallucinations in Large Vision Language Models by dynamically projecting relevant visual semantics from vision tokens into the text generation process, achieving superior performance with reduced computational cost.

Original authors: Beomsik Cho, Jaehyung Kim

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Beomsik Cho, Jaehyung Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Daydreaming" Artist

Imagine you hire a brilliant artist (a Large Vision-Language Model, or LVLM) to describe a photo you show them. You show them a picture of a red apple on a table.

Ideally, the artist looks at the photo and says, "That's a red apple."
But sometimes, the artist gets distracted by their own internal thoughts. They might say, "That's a red apple... and also a banana and a pizza," even though those things aren't in the picture. This is called hallucination.

The paper asks: Why does this happen?
The researchers discovered that the artist actually does see the apple correctly in their "mind's eye" (the visual data), but when they start speaking, they get confused by the noise of their own vocabulary. They know the truth, but they forget to say it because they are too focused on what words usually go together (like "apple" and "pie"), rather than what is actually in the photo.

The Discovery: The "Hidden Notes"

The team looked closely at how these AI models work. They found that the model breaks the image down into tiny chunks called vision tokens. Think of these tokens as little sticky notes attached to different parts of the image.

The Sticky Notes are Smart: Even when the AI makes a mistake (hallucinates a banana), the sticky notes on the image still contain the correct information about the apple. The visual truth is there, just buried.
The Translation Problem: When the AI tries to turn these sticky notes into words, it gets messy. If you ask the AI, "What word does this sticky note represent?" without any rules, it might guess "sky," "blue," or "texture." It's too vague.
The Magic Filter: The researchers found that if you put a filter on the question—asking the AI to only choose from a specific list of relevant words (like "apple," "fruit," "red")—the sticky notes suddenly become very clear. The AI can finally say, "Ah! This note definitely means 'apple'!"

The Solution: ReVisiT (Referencing Vision Tokens)

Based on this, the authors created a new method called ReVisiT. It's like giving the artist a "cheat sheet" while they are speaking, but without needing to retrain the artist or hire a new one.

Here is how ReVisiT works, step-by-step:

The Context Check: As the AI starts writing a sentence, ReVisiT looks at what the AI is currently thinking about.
The Filter: It creates a small, relevant list of words based on that context (e.g., if the sentence is about a kitchen, the list includes "cup," "spoon," "apple," but not "airplane").
The Match-Up: ReVisiT looks at all the "sticky notes" (vision tokens) from the image and asks: "Which of these notes matches our current list of words the best?" It picks the single best match.
The Nudge: It takes that winning sticky note and gently pushes the AI's next word choice toward it. It's like whispering, "Hey, remember that note? It says 'apple,' not 'banana'."

Why It's Cool

No Training Required: You don't need to teach the AI anything new. You just change how it speaks while it's talking.
Super Fast: Because it's just a quick math check and a "nudge," it doesn't slow the AI down. In fact, it's almost as fast as the standard way of speaking.
Works Everywhere: They tested it on many different AI models (from small to very large) and on many different tasks (describing images, answering questions, spotting objects). It consistently reduced the "daydreaming" (hallucinations) and made the descriptions more accurate.

The Analogy: The Librarian and the Book

Imagine the AI is a student taking a test.

The Vision Tokens are the textbooks the student has open on their desk. The student has the right answers in the books.
The Hallucination happens because the student is trying to guess the answer from memory while ignoring the books, or the books are open to the wrong page.
ReVisiT is a librarian who walks up, looks at the question, points to the exact page in the textbook that has the answer, and says, "Read this part." The student then reads the correct answer from the book instead of guessing.

Summary

The paper proves that AI models already "see" the truth inside their visual data, but they often fail to speak it because they get lost in their own word associations. ReVisiT is a simple, free tool that acts as a bridge, forcing the model to look at its own visual notes and use them to correct its speech, resulting in fewer lies and more accurate descriptions.

Technical Summary: ReVisiT – Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding

Problem Statement

Large Vision-Language Models (LVLMs) integrate visual perception with language understanding but frequently suffer from object hallucinations, where the model generates text describing objects not present in the input image. While the visual information is encoded in "vision tokens" within the model's hidden states, the mechanism by which these tokens influence the decoding process remains under-explored. Existing approaches often treat vision tokens as static auxiliary contexts or rely on complex training pipelines and multi-pass inference to mitigate hallucinations. The central problem addressed is how to effectively leverage the intrinsic semantic information within vision tokens during the decoding process without architectural modifications or additional training.

Methodology: ReVisiT

The authors propose ReVisiT (Referencing Vision Tokens), a training-free, model-agnostic decoding strategy designed to guide text generation by dynamically referencing the most relevant vision tokens. The method operates in three primary steps at each decoding timestep $t$ :

Context-Aware Vocabulary Subset Construction:
Instead of operating over the full vocabulary, ReVisiT dynamically constrains the vocabulary space to a subset $V_t^{cons}$ . This subset is defined adaptively based on the vanilla output distribution $p_\theta$ of the LVLM, retaining tokens with probabilities above a threshold $\alpha \cdot \max(p_\theta)$ . This constraint reduces semantic distraction and focuses the decoding on plausible candidates.
Vision Token Selection via Constrained Divergence:
The method projects the hidden states of vision tokens ( $h_i^j$ ) from candidate decoder layers $j$ onto the constrained vocabulary subset $V_t^{cons}$ . It then computes the Jensen-Shannon Divergence (JSD) between the vanilla output distribution and each projected vision token distribution. The vision token $(i^*, j^*)$ with the minimum JSD is selected as the most contextually relevant reference signal.
- Optimization: To ensure efficiency, vision token projections over the full vocabulary are pre-computed once before decoding. During decoding, these cached projections are sliced and masked to fit the dynamic subset $V_t^{cons}$ , avoiding additional forward passes.
Logit Refinement:
The vanilla output distribution is refined by fusing it with the distribution of the selected vision token. This is achieved via log-space addition (equivalent to a Product-of-Experts combination in probability space):
$l_{ReVisiT}(y_t) = \log p_\theta(y_t | h_{T+t-1}^L, V_t^{cons}) + \log p_\theta(y_t | h_{i^*}^{j^*}, V_t^{cons})$
The final distribution is normalized over the constrained subset, and the next token is sampled from this refined distribution.

Key Contributions

Empirical Insights into Vision Tokens:
The paper provides quantitative and qualitative evidence that vision tokens intrinsically encode interpretable object-level semantics. The authors demonstrate that while these signals are often latent under unconstrained decoding (where hallucinated objects may be selected despite ground-truth objects having high probabilities), they become explicit and well-aligned when projected over semantically coherent vocabulary subsets.
ReVisiT Algorithm:
The introduction of a simple, training-free decoding method that utilizes context-aware constrained divergence to select and reference vision tokens. This approach refines token-level logits to better incorporate visual semantics without requiring external modules, architectural changes, or multi-pass inference.
Comprehensive Validation:
The method is validated across six model sizes (ranging from 7B to 32B parameters) spanning three distinct architectures (LLaVA-1.5, Qwen2.5-VL, and InternVL). The evaluation covers five benchmarks, including hallucination-specific datasets (HallusionBench, CHAIR, POPE) and general-purpose tasks (VQAv2, MMMU).

Experimental Results

Hallucination Reduction: ReVisiT consistently reduces hallucinations across all tested architectures. On the CHAIR benchmark, it achieves competitive or superior F1 scores compared to state-of-the-art baselines (e.g., DoLa, VCD, M3ID) while maintaining or improving object coverage. On HallusionBench, it achieves the best scores on four out of five metrics for several models.
Generalization: The method demonstrates robustness on discriminative tasks (POPE, VQAv2) and knowledge-intensive reasoning (MMMU), indicating that the visual grounding signals surfaced by ReVisiT generalize across different task types.
Efficiency: Unlike baseline methods that require additional forward passes (e.g., VCD, CODE) or intermediate layer projections (DoLa), ReVisiT increases inference latency by only 0.6% to 2.0%. It achieves up to a 2 $\times$ reduction in computational cost compared to multi-pass baselines.

Significance and Claims

The paper claims that ReVisiT demonstrates that vision tokens intrinsically encode interpretable semantics that can be unlocked through semantically coherent constraints. By explicitly exposing these semantics in the language space, the method provides an interpretable guidance signal for generation.

The authors position ReVisiT as a scalable and broadly applicable solution that improves visual grounding without the computational overhead of existing intra-alignment or post-alignment methods. They emphasize that the method's effectiveness across diverse model sizes and architectures suggests that the proposed context-aware constrained divergence captures a fundamental decoding signal rather than a benchmark-specific heuristic.

Limitations Acknowledged:
The authors note that ReVisiT relies heavily on the semantics encoded within the model's internal vision tokens, which may lead to an overemphasis on perceptually grounded elements at the expense of implicit commonsense reasoning. Furthermore, as a decoding-time method, it cannot correct factual errors stemming from pretraining data if visual evidence is insufficient, nor does it incorporate external knowledge retrieval, though it remains compatible with Retrieval-Augmented Generation (RAG) pipelines. The evaluation is currently limited to open-source models in the 7–32B parameter range.