💬 NLP

LookBack: Where and How to Score LVLM Responses via Visual Reference Usage

This paper introduces LookBack, a training-free method that improves LVLM response scoring by augmenting token likelihood with a visual lookback score to better detect hallucinations and select grounded outputs, addressing the limitations of existing confidence-based metrics.

Original authors: Beomsik Cho, Jinhyeong Kim, Dongseok Lee, Jaehyung Kim

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Beomsik Cho, Jinhyeong Kim, Dongseok Lee, Jaehyung Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a super-smart robot that has both eyes and a brain. This robot, called a Large Vision-Language Model (or LVLM for short), can look at a picture and then write a story or answer a question about it. It's like having a friend who can see the world and describe it perfectly. But here's the catch: sometimes this robot gets too confident in its own voice. It might describe a scene with such perfect grammar and flow that it sounds totally real, even if it's making things up that aren't in the picture at all. This is called "hallucinating."

To fix this, scientists usually try to pick the best answer from a list of many guesses the robot makes. They often use a simple trick: they pick the answer that sounds the most confident or likely based on the robot's own internal math. Think of it like a teacher picking the essay that sounds the most fluent. But what if the robot is just a smooth talker who is confidently wrong? That's the problem this paper tackles: how do we tell if the robot is actually looking at the picture, or if it's just daydreaming while sounding smart?

The researchers behind this study, from Yonsei University, realized that the robot's usual "confidence meter" is broken when it comes to pictures. They found that even if you take the picture away and just ask the robot to guess, it still gives the same high confidence scores to its answers. It's like a student who can recite a history essay perfectly from memory, but if you ask them about a specific photo in their textbook, they might confidently describe a dinosaur in a classroom because they are just good at guessing words, not looking at evidence.

To solve this, the team invented a new method called LOOKBACK. Imagine you are grading that student's essay again. Instead of just checking if the sentences flow well, you ask: "Did you actually look at the photo when you wrote this word?" LOOKBACK is a special tool that checks every single word the robot writes to see if the robot's "eyes" (its internal attention) were actually looking at the picture at that exact moment.

Here is how it works in a fun way:

The Confidence Check: First, LOOKBACK checks how sure the robot is about a word, just like the old methods did.
The Lookback Check: Then, it asks, "Did you look at the picture to say this?" If the robot says "panda" and its internal eyes were staring right at a panda in the image, that word gets a huge boost. If the robot says "panda" but was just guessing because it loves the word "panda," it gets a penalty.
The Final Score: The tool combines these two checks. It gives the highest score to answers that are not only fluent and confident but also prove they were actually looking at the visual evidence.

The team tested this idea on four different challenges involving pictures and questions, using three different smart robot brains. They found that LOOKBACK was much better at picking the correct answer than the old methods. In fact, when they had the robots generate 25 different guesses and used LOOKBACK to pick the winner, it consistently chose the right answer more often than any other method they tried.

What's really cool is that LOOKBACK doesn't need to learn anything new or use any extra robots to help. It just uses the signals the main robot is already sending while it thinks. It's like giving the robot a mirror so it can check its own work without needing a teacher to stand over it. The study suggests that by simply asking the robot to "look back" at the picture while it speaks, we can stop it from confidently lying about what it sees, making these AI friends much more reliable for real-world tasks.

Technical Summary: LOOKBACK – Where and How to Score LVLM Responses via Visual Reference Usage

1. Problem Statement

Large Vision-Language Models (LVLMs) integrate visual perception with language generation but suffer from a specific type of hallucination: producing fluent, plausible responses that are ungrounded in the input image. While existing methods for improving reliability, such as Best-of-N (BoN) selection, rely on scoring candidate responses to select the best one, current scoring mechanisms are insufficient for LVLMs.

The paper identifies a critical gap in output-space confidence (e.g., token likelihood or Self-Certainty). Diagnostics reveal that these confidence scores are largely insensitive to the visual input. Specifically, removing the image from the input context barely changes the confidence-based selection of the top-ranked response. This indicates that output-space confidence primarily captures textual plausibility (linguistic fluency) rather than visual grounding (agreement with the image). Consequently, relying solely on confidence fails to distinguish between responses that are linguistically fluent but visually hallucinated versus those that are faithful to the image.

2. Methodology: LOOKBACK

To address this, the authors propose LOOKBACK, a training-free, model-intrinsic response scoring method. LOOKBACK augments token likelihood with a visual lookback score, a lightweight measure of how strongly each generated token refers back to the visual input. The method operates in three stages:

A. Visual Lookback Score ( $A_t$ )

The authors leverage the internal attention mechanisms of the LVLM. Since LVLMs process images as explicit vision tokens within the causal context, each generated token prediction can directly attend to these vision tokens.

Definition: For a generated token $y_t$ , the visual lookback score $A_t$ is the fraction of attention weight directed from the prediction step to the vision tokens ( $P_v$ ) across all layers and attention heads.
Significance: $A_t$ serves as a proxy for visual reference usage. It is high for tokens describing objects, attributes, or relations present in the image and low for generic function words.

B. Lookback-Calibrated Token Score ( $u_t$ )

LOOKBACK combines the model's output-space confidence ( $p_t$ ) with the visual lookback score ( $A_t$ ) to form a calibrated token score:
$u_t = \log(p_t) + \alpha \log(A_t)$

Here, $\alpha$ is a hyperparameter controlling the strength of the calibration.
This formulation ensures that tokens with high confidence but low visual attention (hallucinations) receive a lower score, while tokens that are both confident and visually grounded are favored.

C. Visual Relevance Weighting and Aggregation

Not all tokens contribute equally to visual grounding. LOOKBACK aggregates token scores using a visual relevance distribution ( $q_\lambda$ ) rather than a uniform average.

Distribution: $q_\lambda$ is derived by maximizing an entropy-regularized relevance objective, concentrating probability mass on tokens with high visual lookback scores ( $A_t$ ).
Final Score: The final response score $S(y)$ is the expected value of the calibrated token scores under this distribution:
$S(y) = \sum_{t=1}^{T} q_\lambda(t) \cdot u_t$
This ensures that the final score is heavily influenced by tokens that are both confident and strongly tied to the visual input.

3. Key Contributions

Diagnostic Insight: The paper provides empirical evidence that standard output-space confidence metrics (like Self-Certainty) are largely image-insensitive, failing to detect visual hallucinations in Best-of-N selection settings.
Training-Free Scoring: LOOKBACK introduces a method to score LVLM responses without requiring external reward models, additional training, or extra inference passes. It utilizes only internal signals (token probabilities and attention weights) available during generation.
Token-Level Complementarity: The work demonstrates that output-space confidence and visual lookback scores capture complementary aspects of generation: confidence favors linguistic predictability, while lookback favors visual reference.
Efficient Aggregation: By weighting tokens based on visual relevance, LOOKBACK prioritizes diagnostic content words (nouns, attributes) over generic fluency words in the final scoring.

4. Experimental Results

The authors evaluated LOOKBACK across four benchmarks (VQAv2, CHAIR, AMBER, HallusionBench) and three representative LVLMs (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL3-8B).

Performance: LOOKBACK consistently outperformed existing baselines, including language-side methods (Self-Certainty, Universal Self-Consistency) and vision-side methods (CLIPScore, VAUQ).
- It achieved the highest model-wise average performance across all three LVLMs.
- It improved Best-of-N selection accuracy by approximately 4.97% relative gain over random selection and consistently beat the Self-Certainty baseline.
Scaling: As the candidate pool size ( $N$ ) increased from 1 to 25, LOOKBACK maintained a consistent advantage over baselines, whereas other methods showed diminishing returns or instability.
Efficiency: LOOKBACK incurred negligible computational overhead compared to generation time. It was significantly faster than methods requiring perturbation (VAUQ) or external encoders (CLIPScore), adding only a modest post-generation scoring cost.

5. Significance and Claims

The paper claims that LOOKBACK provides a lightweight, model-internal solution for improving the reliability of LVLM responses. Its significance lies in:

Bridging the Gap: It directly addresses the disconnect between image-conditioning and image-sensitivity in current scoring metrics.
Practicality: By being training-free and requiring no auxiliary models, it is applicable to resource-constrained settings and black-box scenarios (provided internal attention weights are accessible).
Generalizability: The authors suggest the principle of "source-aware scoring" (checking if high-confidence generation steps refer back to the intended source) could extend beyond vision to other grounded generation tasks, such as Retrieval-Augmented Generation (RAG) or tool use.

Limitations Acknowledged:
The authors note that LOOKBACK requires access to internal attention weights, limiting its use with strictly black-box APIs. Furthermore, visual lookback is a proxy for reference usage, not a guarantee of factual correctness; a model can attend to image tokens while still generating incorrect or biased content. Therefore, LOOKBACK is recommended as a response-selection aid to be used alongside task-specific validation and human oversight.