💻 computer science

Learning to See What You Need: Gaze Attention for Multimodal Large Language Models

This paper introduces Gaze Attention, a novel mechanism for Multimodal Large Language Models that dynamically selects task-relevant visual regions during generation to significantly reduce computational overhead while maintaining or improving performance through the use of learnable context tokens.

Original authors: Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Overwhelmed Librarian"

Imagine a multimodal Large Language Model (MLLM) as a super-smart librarian who is trying to describe a picture to you. Currently, when this librarian looks at a photo, they try to read every single word written on every single page of a massive encyclopedia that represents that photo.

Even if you only ask, "What is the dog doing?", the librarian still frantically scans the entire book, including pages about the sky, the grass, and a cat in the corner. This is called dense attention. It's inefficient, wastes a lot of energy (computational power), and can actually make the librarian's focus "diluted" or scattered because they are trying to process too much irrelevant information at once.

Humans don't do this. When we describe a scene, our eyes naturally shift (or "gaze") to the specific part we are talking about. If we talk about the dog, we look at the dog. If we talk about the cat, we look at the cat. We don't stare blankly at the whole room with equal intensity.

The Solution: "Gaze Attention"

The authors of this paper created a new system called Gaze Attention. Think of it as teaching the librarian to act more like a human with a flashlight.

Instead of reading the whole book, the librarian now:

Divides the book into chapters: The image is broken down into small, manageable chunks (called "gaze regions").
Creates a "Cliff's Notes" summary: For each chapter, the librarian writes a tiny, lightweight summary card (a "descriptor") that says what that part of the image is about.
Shines the flashlight: When the model needs to generate the next word (like "dog"), it quickly checks the summary cards, picks the one that matches "dog," and only reads the details from that specific chapter. It ignores the rest of the book for that moment.

This happens dynamically. As the sentence changes from "The dog is running" to "The cat is sleeping," the model instantly shifts its "flashlight" to the cat's chapter.

The Safety Net: "Context Tokens"

There was a worry: If the librarian only looks at the specific chapter about the dog, might they forget the general vibe of the whole room?

To fix this, the authors added a few special "Context Tokens." Imagine these as a panoramic photo of the whole room taped to the librarian's desk. Even when the librarian is zooming in on the dog, they can still glance at this panoramic photo to remember, "Oh right, this is a park scene." This ensures the model keeps a "big picture" awareness without having to read every single detail of the whole image every time.

The Results: Faster, Smarter, and Cheaper

The paper tested this new method on many different tasks, from answering questions about single photos to understanding long videos.

Efficiency: The model achieved the same (or better) results as the old "read everything" method but used up to 90% fewer visual tokens to do it. It's like finishing a book report by reading only the relevant chapters instead of the whole novel.
Focus: When the researchers looked at where the model was looking, they saw it was very focused and precise, much like human eye movements. The old method looked like a blurry, scattered mess.
Video: This was especially helpful for videos. Instead of trying to remember every frame of a long movie, the model learned to jump to the specific frames and spots where the action was happening.

What the Paper Does Not Claim

It is important to note what this paper does not say:

It does not claim the model can now "see" in the human sense or have consciousness.
It does not claim this will immediately be used in medical diagnosis or self-driving cars (though it could be useful there in the future, the paper only tests it on standard image and video question-answering benchmarks).
It does not say the model is perfect; the authors admit that if the "chapters" (regions) are cut up in a way that splits an object in half, the system might get confused.

Summary

In short, Gaze Attention teaches AI models to stop staring blankly at the whole picture and start looking where they actually need to look. By mimicking how humans shift their gaze, the models become faster, use less computer power, and can describe scenes more accurately by focusing on the details that matter at that exact moment.

Technical Summary: Gaze Attention for Multimodal Large Language Models

Problem Statement

Current Multimodal Large Language Models (MLLMs) process visual inputs by attending uniformly to all visual tokens at every generation step. This "dense attention" mechanism leads to two primary issues:

Diluted Focus: The model processes irrelevant visual regions alongside task-relevant ones, failing to mimic the selective nature of human visual perception, where gaze shifts dynamically to fixate on regions pertinent to the intended description.
Computational Overhead: Attending to the entire sequence of visual tokens (which can be massive in high-resolution images or long videos) creates significant computational and memory burdens, particularly in the Key-Value (KV) cache.

Existing approaches to mitigate this, such as token compression or KV-cache eviction, typically determine which visual information to retain before generation begins (e.g., based on a one-time relevance estimate from the input question). These static methods often discard fine-grained details or fail to adapt when subsequent generation steps require information from different, previously discarded regions. They lack the ability to dynamically select visual regions at each decoding step.

Methodology: Gaze Attention

The authors propose Gaze Attention, a mechanism that enables MLLMs to selectively attend to task-relevant visual regions dynamically during the generation process. The method consists of three core components:

1. Gaze Regions and Descriptors

Instead of treating visual tokens as a flat sequence, the method spatially groups visual key vectors (from the KV cache) into compact gaze regions.

Partitioning: Visual keys are reshaped into their original spatial (and temporal, for video) layout and partitioned into regions of size $r_h \times r_w \times r_t$ .
Descriptors: Each region is summarized by a lightweight descriptor ( $d_g$ ), obtained via mean-pooling the key vectors within that region. This reduces the candidate set from the full token sequence to the region level.
Dynamic Routing: At each decoding step $j$ , the query token $q_j$ computes similarity scores with all region descriptors. The model selects the top- $K$ most relevant regions ( $G_j$ ) and restricts the attention mechanism to attend only to the key-value pairs within these selected regions.
Stability: To address training instability inherent in learning sparse attention from scratch, a progressive Top-K schedule is employed. Training begins with a dense setting (selecting all regions) and gradually reduces $K$ as training proceeds, allowing the model to learn discriminative routing signals without being forced into premature sparse decisions.

2. Learnable Context Tokens

To prevent the loss of global scene information caused by localized attention, the authors introduce learnable context tokens.

A small set of learnable tokens is appended to each image or video frame at the LLM input.
These tokens form compact summary keys ( $K_C$ ) that remain visible throughout the generation process, regardless of which specific gaze regions are selected.
This ensures the model retains a holistic visual awareness (global context) while performing fine-grained, selective attention on local regions, analogous to how humans perceive a scene as a whole before focusing on details.

3. Unified Attention Formulation

The final attention mechanism combines three sources of Key-Value pairs for any given query:

Textual tokens.
Global context tokens ( $K_C$ ).
Visual tokens from the dynamically selected gaze regions ( $K_{selected}$ ).

This formulation allows the model to maintain linguistic coherence and global visual awareness while allocating the majority of its visual computation budget to task-relevant regions.

Key Contributions

Dynamic Selective Attention: Unlike prior methods that fix visual attention based on pre-computed importance scores, Gaze Attention routes queries to relevant regions at every generation step, adapting to the evolving needs of the text being generated.
Efficiency without Global Loss: The method achieves significant computational savings by restricting attention to a subset of regions while preserving global context through learnable tokens, avoiding the "blurring" or "discarding" of details common in token compression or static eviction methods.
Emergent Gaze Behavior: The model learns to shift its "gaze" to match the subject of the generated token (e.g., focusing on a dog when generating the word "dog") solely through next-token prediction, without explicit gaze supervision.

Results

The authors evaluated Gaze Attention on 13 image and 6 video understanding benchmarks, comparing it against dense-attention baselines and existing KV-cache eviction methods (e.g., InfiniPot-V, HERMES).

Performance: Gaze Attention matches or surpasses dense-attention baselines across all benchmarks. In some cases, it outperforms baselines even with significantly reduced KV cache sizes.
Efficiency: The method reduces the number of visual KV entries involved in attention computation by up to 90%.
- In image tasks, it maintains performance with only ~16% of the dense attention FLOPs.
- In video tasks, it outperforms dense-attention baselines constrained to the same KV budget.
Comparison to Eviction Methods: Existing eviction methods (InfiniPot-V, HERMES) suffer significant performance drops when aggressively reducing KV cache size, particularly in single-image settings, because they cannot adapt their selection once the generation begins. Gaze Attention remains robust under similar or more aggressive compression.
Visualization: Attention map visualizations confirm that Gaze Attention produces concentrated, interpretable attention patterns focused on relevant objects, whereas dense attention yields diffuse, ambiguous patterns.

Significance and Claims

The paper claims that enabling MLLMs to selectively attend to relevant visual regions is a promising alternative to dense attention. By mimicking human gaze behavior—dynamically shifting focus based on the current generation step—Gaze Attention offers a path to:

Reduce Computational Costs: Significantly lowering FLOPs and memory usage (up to ~80% reduction in attention FLOPs and ~80% in KV cache memory) without sacrificing performance.
Improve Interpretability: Generating concentrated attention maps that align with the semantic content of the generated text.
Enhance Adaptability: Allowing models to access different visual details at different stages of generation, overcoming the limitations of static, pre-computed visual selection.

The authors note that this approach bridges the gap between efficient attention mechanisms (typically studied for text) and selective visual attention for multimodal understanding, demonstrating that high performance can be achieved with a fraction of the visual context processed.