💻 computer science

Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation

This paper introduces EAGLE, a lightweight black-box framework that explains autoregressive token generation in multimodal large language models by attributing decisions to specific visual regions and quantifying the balance between visual evidence and language priors, thereby offering superior faithfulness, localization, and hallucination diagnosis with reduced computational cost.

Original authors: Ruoyu Chen, Xiaoqing Guo, Kangwei Liu, Siyuan Liang, Shiming Liu, Qunli Zhang, Laiyuan Wang, Hua Zhang, Xiaochun Cao

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruoyu Chen, Xiaoqing Guo, Kangwei Liu, Siyuan Liang, Shiming Liu, Qunli Zhang, Laiyuan Wang, Hua Zhang, Xiaochun Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly mysterious, robot friend named EAGLE. This robot is great at looking at pictures and describing them in sentences (like, "A cat is sitting on a banana tree"). But sometimes, it makes things up (hallucinations), or it's hard to tell why it said what it said.

This paper introduces EAGLE, a new tool designed to be a "detective" for these AI robots. It answers two big questions:

Where did the robot look? (Which part of the picture made it say "cat"?)
What did it rely on? (Did it actually see the cat, or did it just guess because it's a common sentence?)

Here is a simple breakdown of how it works, using some fun analogies.

1. The Problem: The "Black Box" Mystery

Imagine you ask a chef to cook a dish. They hand you a delicious meal, but they won't tell you which ingredients they used or in what order. You might suspect they used salt, but maybe they used sugar instead.

MLLMs (Multimodal Large Language Models) are like that chef. They look at an image and write a story.
The Issue: Sometimes they lie (hallucinate). If they say, "There is a dog in the picture," but there isn't one, we need to know why they thought that. Old tools tried to guess by looking at the chef's brainwaves (internal math), but that's messy and often wrong.

2. The Solution: The "Lego Block" Detective

EAGLE doesn't need to look inside the robot's brain. It treats the robot like a black box (you can't see inside, you just put things in and get things out).

How it works:
Imagine the picture is a giant puzzle made of 100 tiny Lego blocks (small pieces of the image).

Step 1: The Blindfold Test. EAGLE covers up most of the Lego blocks, leaving only a few visible. It asks the robot: "Can you still describe the picture?"
Step 2: The "Insight" Score (The "Enough" Test). EAGLE tries to find the smallest group of Lego blocks that, if shown to the robot, would make it say the exact same sentence as before.
- Analogy: If you show the robot just the "cat's ear" and "tail," and it still says "Cat," those are the Insight blocks. They are enough to do the job.
Step 3: The "Necessity" Score (The "Need" Test). EAGLE then takes a full picture and starts removing blocks one by one. It watches to see which block, when removed, makes the robot stop saying "Cat."
- Analogy: If you remove the "cat's body" and the robot suddenly says "I don't see anything," that block was Necessary.

By combining these two tests, EAGLE creates a Heat Map. It highlights the specific parts of the image that actually mattered.

3. The Twist: "Guessing" vs. "Seeing"

This is the cleverest part. EAGLE also figures out if the robot is seeing the object or just guessing based on what it knows about language.

The Scenario: The robot says, "The man is holding a coffee cup."
The Test: EAGLE slowly reveals the picture to the robot.
- If the robot is a "Visual Learner": As soon as the coffee cup is revealed, the robot's confidence skyrockets. It needs to see the cup to say the word.
- If the robot is a "Language Guesser": The robot says "coffee cup" even when the picture is completely black! It's just guessing because "man" and "coffee cup" often go together in stories.
The Result: EAGLE can tell you: "Hey, this word was based on what it saw (Visual Evidence), but that other word was just a lucky guess (Language Prior)."

4. Why is this a Big Deal?

The paper tested EAGLE on several popular AI models and found it to be the best detective so far.

It's More Accurate: Old tools often pointed to the wrong parts of the image (like highlighting the sky when the robot was talking about a car). EAGLE points to the car.
It Catches Lies: When a robot hallucinates (makes up a dog that isn't there), EAGLE can pinpoint exactly which weird patch of pixels confused the robot. It's like finding the "smoking gun" that caused the lie.
It Saves Money: It doesn't need a super-expensive computer to run. It's lightweight and efficient.

Summary

Think of EAGLE as a truth-telling translator for AI.

Before, we had to trust the AI blindly.
Now, EAGLE lets us say, "Okay, I see why you said that. You saw the banana tree, so you mentioned bananas. But you didn't see a cat, so why did you say there was one? Let's look at this blurry spot that confused you."

It makes AI more transparent, trustworthy, and easier to fix when it makes mistakes.

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant success in vision-language tasks but suffer from a lack of transparency and reliability. Key challenges include:

Opacity: It is difficult to trace how specific visual inputs influence generated text tokens, especially given the autoregressive nature of generation.
Hallucinations: MLLMs frequently generate factually incorrect outputs (e.g., hallucinating objects), undermining trust in safety-critical domains.
Limitations of Existing Methods:
- Attention Visualization: Often fails to capture complex cross-modal interactions.
- Gradient-based Methods: Sensitive to noise, cumulative effects in long sequences, and heavily influenced by textual priors rather than visual evidence.
- Activation-based Methods: Lack a direct causal link between inputs and outputs and often reflect intermediate layer preferences misaligned with human intuition.
- Subset Selection (VPS): While effective for object detection, existing subset-selection methods cannot be directly transferred to the autoregressive token generation of MLLMs.

2. Methodology: The EAGLE Framework

The authors propose EAGLE (Explaining Autoregressive Generation by Language priors or Evidence), a lightweight, black-box attribution framework. It operates without requiring access to internal model gradients or weights.

A. Core Workflow

Image Sparsification: The input image is decomposed into semantically meaningful sub-regions using SLICO superpixel segmentation, creating a set of candidate regions $V$ .
Greedy Search for Attribution: The problem is formulated as an ordered subset selection task. The goal is to find a minimal set of regions that maximizes the probability of generating target tokens.
Objective Function: EAGLE introduces a unified objective function combining two scores:
- Insight Score (Sufficiency): Measures the probability of generating target tokens given a specific subset of regions $S$ . It identifies regions sufficient for generation.
- Necessity Score (Indispensability): Measures the drop in generation probability when regions $S$ are removed (i.e., using $V \setminus S$ ). It identifies regions essential for generation.
- Optimization: A greedy search strategy is employed to order the sub-regions based on their marginal contribution to the combined objective, creating a ranked list of influential visual regions.

B. Modality Relevance Analysis

Beyond spatial attribution ("Where"), EAGLE analyzes "What" the model relies on:

Language Priors vs. Perceptual Evidence: By tracking how token logits evolve as salient regions are progressively revealed, EAGLE calculates an Influence Score.
Mechanism: If a token's probability remains stable regardless of visual input changes, it relies on language priors. If the probability fluctuates significantly with the introduction of visual regions, it relies on perceptual evidence.
Token-Agnostic: Unlike gradient methods that require pre-selecting "visual" tokens, EAGLE works on any token and explicitly diagnoses the source of its generation.

3. Key Contributions

EAGLE Framework: A lightweight, black-box attribution method for autoregressive token generation that attributes any selected set of tokens to compact perceptual regions with low GPU memory cost.
Unified Objective Function: A novel formulation combining Insight (sufficiency) and Necessity (indispensability) scores, optimized via greedy search to balance interpretability and efficiency.
Modality Disentanglement: A mechanism to quantify the reliance of each token on language priors versus visual evidence, offering fine-grained interpretability.
Hallucination Diagnosis: The ability to pinpoint the minimal set of visual regions responsible for hallucinations and demonstrate that removing them corrects the error.

4. Experimental Results

The method was evaluated on open-source MLLMs (LLaVA-1.5, Qwen2.5-VL, InternVL3.5) across MS COCO (Image Captioning), MMVP (VQA), and RePOPE (Object Hallucination).

A. Faithfulness (Sentence & Word Level)

Performance: EAGLE consistently outperforms state-of-the-art baselines (LLaVA-CAM, IGOS++, TAM) in Insertion, Deletion, and Average Highest Score metrics.
- Image Captioning: Improved Insertion by ~20% and Deletion by ~13% on average compared to baselines.
- Word-Level: Surpassed TAM by 36.42% and 42.63% on the Pointing Game (box and mask levels), demonstrating superior object grounding.
Efficiency: EAGLE requires significantly less GPU memory (e.g., 17.68 GB for Qwen2.5-VL 7B vs. 96.90 GB for IGOS++), making it practical for large models. While slower in inference time than gradient methods, it is the only method showing potential for real-world interpretability requirements.

B. Hallucination Diagnosis

Localization: EAGLE accurately identifies the specific visual regions causing hallucinations (e.g., confusing a surfboard for a snowboard).
Correction: By removing only the minimal set of identified "hallucination-prone" regions, the model's output is corrected.
- Achieved ~80% Correction Success Rate under a 10% region removal budget (CSR@10%), significantly outperforming baselines.
- Reduced the Average Minimal Correction Region (AMCR) by large margins (e.g., 65.4% improvement on LLaVA-1.5).

5. Significance

Trust & Safety: Provides a reliable tool to diagnose why MLLMs fail, specifically in safety-critical domains like healthcare and autonomous driving, by revealing the exact visual triggers for errors.
Interpretability: Moves beyond "where the model looks" to "what the model relies on," distinguishing between genuine visual understanding and reliance on language priors.
Scalability: The black-box, low-memory design allows the method to be applied to a wide range of current and future MLLMs without architectural modifications.
Practicality: Demonstrates that removing minimal visual interference can effectively mitigate hallucinations, offering a pathway for model refinement and debugging.

In conclusion, EAGLE represents a significant step forward in making MLLMs transparent, reliable, and safe by providing a faithful, efficient, and granular explanation of their autoregressive generation process.