💻 computer science

Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow

This paper proposes an Adaptive Information Flow (AIF) method that enhances Vision-Language Models by dynamically modulating attention during inference to ensure text tokens focus only on important visual tokens, thereby correcting misalignment between visual perception and answer generation.

Original authors: Chengxin Liu, Wonseok Choi, Chenshuang Zhang, Tae-Hyun Oh

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chengxin Liu, Wonseok Choi, Chenshuang Zhang, Tae-Hyun Oh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but slightly distracted student named Vision-Language Model (VLM). This student is incredibly good at looking at a picture and reading a question about it. They can see almost everything in the image.

However, there's a problem: sometimes, even though they see the right thing, they give the wrong answer.

The Problem: The "Distracted Student"

Think of the VLM as a student taking a test with a picture in front of them.

The Question: "What cartoon character is on the clock?" (The answer is Mickey Mouse).
The Student's Eye: They look at the clock, but their eyes also wander to the background, the table, the window, and the dust motes in the air.
The Brain's Confusion: Because the student is trying to process all those visual details at once, their brain gets noisy. They see Mickey Mouse, but they also see a blurry chair and a shadow. The "noise" from the irrelevant stuff drowns out the clear signal of Mickey Mouse. So, they guess "Bugs Bunny" or "Donald Duck" by mistake.

In technical terms, the paper calls this "misalignment between seeing and perceiving." The model sees the image, but its internal "attention" is spread too thin across irrelevant parts of the picture.

The Solution: The "Smart Filter"

The researchers (Chengxin Liu and colleagues) realized that the student doesn't need to stop looking at the picture; they just need to stop listening to the wrong parts of the picture while they are thinking about the answer.

They invented a method called Adaptive Information Flow (AIF). Here is how it works, using a simple analogy:

1. The "Noise Meter" (Token Dynamics)

Imagine the student is reading the question and looking at the picture. As they do this, their brain lights up in different spots.

Some parts of the picture (like the Mickey Mouse clock) make their brain light up brightly and consistently.
Other parts (like the background wall) make their brain light up randomly and weakly.

The researchers created a "Noise Meter" (called Token Dynamics) that measures how much a specific part of the image is actually helping the student answer the question. If a part of the image is just causing random noise, the meter says, "Ignore this!"

2. The "Silence Button" (Causal Mask Modulation)

Usually, the student is forced to listen to every single visual detail while they speak their answer. It's like trying to solve a math problem while someone is shouting random numbers in your ear.

The researchers' method adds a Silence Button.

Before the student gives their final answer, the system checks the "Noise Meter."
It identifies the "noisy" parts of the image (the background, the irrelevant objects).
It hits the Silence Button, effectively cutting the connection between the student's brain and those noisy visual parts.
Now, the student can only "hear" the important parts (Mickey Mouse).

The Result: A Clearer Answer

Once the noise is cut off, the student's brain focuses entirely on the relevant evidence.

Before: "I see a clock, a table, a shadow... maybe it's Bugs Bunny?" -> Wrong Answer.
After: "I see a clock. The clock has Mickey Mouse on it. The answer is Mickey Mouse." -> Correct Answer.

Why This Is Special

Usually, to fix a student who makes mistakes, you have to send them back to school for retraining (teaching them new lessons), which takes a long time and costs a lot of money.

This paper's method is like giving the student a pair of noise-canceling headphones right before the test.

No Retraining: The student's brain (the model's weights) doesn't change at all.
Instant Fix: It happens in a split second during the test.
Works Everywhere: It helps with counting objects, reading text in images (OCR), and spotting things that aren't actually there (hallucinations).

Summary

The paper shows that Vision-Language Models often fail not because they are blind, but because they are too chatty with irrelevant details. By simply blocking the flow of information from the "boring" parts of the image to the "thinking" part of the brain, the models suddenly become much smarter, more accurate, and less likely to hallucinate.

It's a bit like telling a detective: "Don't look at the whole messy crime scene; just focus on the fingerprint on the gun." Suddenly, the case is solved.

1. Problem Statement

Vision-Language Models (VLMs) have achieved remarkable success in multimodal tasks. However, a critical misalignment exists between "seeing" (capturing relevant image regions via attention) and "perceiving" (generating correct answers based on those regions).

The Issue: Recent studies show that while VLMs often attend to the correct image regions, they frequently output incorrect answers.
Root Cause: The authors identify suboptimal information flow during the decoding process. Specifically, text tokens distribute excessive attention to irrelevant visual tokens (background noise), creating a "spatially dispersed" attention pattern. This introduces noise into the textual decoding path, distracting the model and leading to hallucinations or errors.
Limitations of Existing Solutions:
- Visual Cropping: Effective for fine-grained perception but significantly increases inference time and fails on relational reasoning.
- Retraining: Requires massive computational resources and data.
- Post-hoc Correction: Often acts after the fact without fixing the internal interaction pathways.

2. Methodology: Adaptive Information Flow (AIF)

The proposed method, Adaptive Information Flow (AIF), is a training-free approach that modulates the causal mask during inference to block information flow between text tokens and irrelevant visual tokens.

Core Concept: Token Dynamics

The authors observe that visual tokens corresponding to important object regions exhibit distinct activation patterns across different layers of the Large Language Model (LLM), whereas irrelevant tokens show irregular or random activation.

Token Dynamics ( $D_{v_i}$ ): Defined as the sequence of maximum attention values a visual token $v_i$ receives from text tokens across all LLM layers ( $L$ ).
Entropy Measurement: To quantify the "importance" of a visual token, the authors compute an entropy value based on these dynamics.
- High Entropy: Indicates high randomness/irregularity in activation across layers $\rightarrow$ Low Importance (likely background/noise).
- Low Entropy: Indicates consistent, distinct activation patterns $\rightarrow$ High Importance (likely the target object).

The AIF Pipeline

One-Step Decoding: The model performs a single decoding step to collect attention statistics and compute token dynamics for all visual tokens.
Entropy Calculation: Compute the entropy ( $Ent_{v_i}$ ) for each visual token using the dynamics data.
Adaptive Masking:
- The method treats masking as an optimal thresholding problem.
- It tests different mask ratios (e.g., masking tokens with the highest entropy first).
- It selects the mask ratio that maximizes the shift in attention distribution (measured by entropy) away from the baseline, effectively centralizing attention on relevant regions.
Causal Mask Modulation: A modified causal mask is generated where the connections between text tokens and masked (irrelevant) visual tokens are blocked (set to $-\infty$ $- \infty$ ).
- Crucial Detail: Connections between visual tokens remain intact to preserve image context; only the cross-modal flow to text is restricted.
Final Inference: The model re-runs inference (or continues) using the modulated causal mask to generate the final answer.

3. Key Contributions

Diagnosis of Misalignment: Demonstrated that the gap between "seeing" and "perceiving" is caused by noisy information flow where text tokens attend to irrelevant visual regions.
Training-Free Modulation: Proposed a novel, architecture-agnostic method to modulate information flow at inference time without retraining or changing model weights.
Token Dynamics-Based Entropy: Introduced a metric to identify important visual tokens based on their activation stability across layers, proving it is a superior proxy for token importance compared to raw attention scores.
Comprehensive Evaluation: Validated the method across diverse tasks including Visual Question Answering (VQA), OCR, Visual Grounding, Counting, and Hallucination reduction.

4. Experimental Results

The method was evaluated on representative open-source VLMs (LLaVA-1.5 and Qwen2.5-VL) across multiple benchmarks:

General VQA: Significant improvements on RealWorldQA (+4.9% for LLaVA-1.5, +6.0% for Qwen2.5-VL) and MMStar.
OCR Understanding: Consistent gains on TextVQA and SeedBench2-Plus, demonstrating effectiveness in fine-grained tasks.
Visual Grounding: Outperformed specialized grounding models (e.g., Grounding-DINO) on RefCOCO datasets, with an average improvement of 2.3 points.
Counting: Improved performance on CountBench by ~2-3%.
Hallucination Reduction: Reduced object hallucinations on the POPE dataset (e.g., LLaVA-1.5 accuracy rose from 85.4% to 88.7%).
Comparison: The method outperformed both training-free baselines (like ViCrop) and training-based methods (like CCA) without the computational cost of retraining.

5. Significance and Impact

Efficiency: The method adds negligible computational overhead (roughly equivalent to generating one token) compared to the massive costs of retraining or multi-step cropping.
Interpretability: It provides a new lens for understanding VLMs, showing that controlling the structure of information flow is as critical as the model's capacity.
Generalizability: Being model-agnostic and training-free, it can be applied to any decoder-based VLM to instantly boost performance and reduce hallucinations.
Paradigm Shift: Moves the focus from "adding more data/parameters" to "optimizing how modalities interact internally," suggesting that better alignment can be achieved by pruning noise rather than adding complexity.

In summary, the paper argues that VLMs often fail not because they cannot see, but because they cannot filter out visual noise during the reasoning process. By adaptively blocking irrelevant visual-text interactions via causal mask modulation, the model is forced to focus on evidence, leading to more accurate and grounded outputs.