💻 computer science

Attention Alignment Between Humans and Vision-Language Models

This study reveals that while LSTM-based decoders in vision-language models achieve superior alignment with human spatial attention compared to Transformer decoders, the latter exhibit sharper spatial concentration and better task differentiation, with a trade-off emerging where models with lower fixation alignment (CNN-Transformers) better predict synthetic neural activity in early visual cortex.

Original authors: Isaac R. Christian, Udith Haputhanthrige, Hanna Hornfeld, Declan Campbell, Samuel Nastase, Taylor Webb, Michael Graziano

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Isaac R. Christian, Udith Haputhanthrige, Hanna Hornfeld, Declan Campbell, Samuel Nastase, Taylor Webb, Michael Graziano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to look at a picture the way a human does. You want the robot's eyes to land on the same spots a person would when they look at a photo of a busy street or a family gathering.

This paper is like a scientific experiment where researchers built several different "robot brains" to see which one looks most like a human. They didn't just build one; they built a whole family of robots by mixing and matching different parts, like swapping out the camera lens (the Encoder) and the part of the brain that decides what to focus on (the Decoder).

Here is the breakdown of their findings, using simple analogies:

1. The Two Main Parts of the Robot

Think of the robot's vision system as a camera and a director:

The Encoder (The Camera): This is the part that takes the picture and breaks it down into pieces. The researchers tested two types:
- CNN (The Local Scanner): Like a person looking at a photo one small square at a time, noticing details like edges and textures.
- ViT (The Global Glancer): Like a person stepping back to see the whole picture at once, understanding the big picture immediately.
The Decoder (The Director): This is the part that decides, "Okay, now that I see the picture, what should I say, and where should I look next?" They tested two types:
- LSTM (The Single-Threaded Director): This director tries to hold all the visual information in one hand and squeeze it into a single "focus point" for every word it speaks.
- Transformer (The Team of Directors): This director has a team of specialists (called "heads") who all look at the picture at the same time, each focusing on a different aspect.

2. The Big Discovery: The Director Matters More Than the Camera

The researchers found that who is directing the show matters way more than what kind of camera is being used.

The "LSTM Director" (Single-Threaded): This robot was the best at mimicking human eye movements. When humans looked at a picture, this robot's attention map looked very similar to where humans looked. It got about 85–87% of the way to a perfect human match.
- The Catch: While it looked at the right places, it was a bit "sloppy." Its attention was spread out like a wide, fuzzy spotlight. It didn't zoom in sharply on specific details, and it didn't change its focus much whether the task was "describe the scene" or "guess what the person is looking at."
The "Transformer Director" (Team of Specialists): This robot was sharper and more precise. It could focus its attention tightly on specific spots, just like a laser beam. It also changed its focus dramatically depending on the task (e.g., looking at a face for a social task vs. looking at the whole room for a description).
- The Catch: Despite being sharper, it was much worse at matching human eye movements. It only reached about 40–59% of the human match. It was looking at the right kind of spots, but not the exact spots humans chose.

The Verdict: If you want a robot that looks at a picture exactly where a human does, you want the "LSTM Director." If you want a robot that is precise and adaptable but looks at different spots than humans, you want the "Transformer Director."

3. The "Camera" Still Helps

Even though the Director was the most important part, the Camera still made a difference.

The CNN Camera (the local scanner) consistently helped the robot look more like a human than the ViT Camera (the global glancer).
The best combination overall was the CNN Camera paired with the LSTM Director. This robot was the closest match to human behavior, reaching nearly 87% of the human limit.

4. The "Brain Damage" Test

To see how robust these robots were, the researchers simulated a "brain injury" by blocking out the left side of the robot's vision (like a condition called hemispatial neglect, where people ignore one side of space).

The Transformer robots (the team of directors) were very tough. Even when half their vision was blocked, they could still make sense of the picture because their team members could share the load.
The LSTM robots (the single-threaded director) struggled more. Because they relied on one big "summary" of the image, blocking part of the view hurt them more.

5. The Surprise: Looking vs. Thinking

Finally, the researchers asked a tricky question: "Does a robot that looks like a human also think like a human?"

They used a special tool to simulate how a human brain would react to the pictures the robots were looking at.

The Surprise: The robot that looked most like a human (the LSTM) was not the one that best predicted brain activity.
Instead, the CNN-Transformer robot (which looked less like a human) was actually better at predicting what parts of the brain would light up.
What this means: There is a difference between "where we look" (behavior) and "how our brain processes the image" (neural activity). A robot can look at a picture in a human-like way but process the information in a non-human way, and vice versa.

Summary

To look like a human: Use a "Local Scanner" camera (CNN) with a "Single-Threaded Director" (LSTM). It's a bit fuzzy but hits the right spots.
To be sharp and adaptable: Use a "Global Glancer" camera (ViT) with a "Team of Directors" (Transformer). It's precise but looks at different spots than humans.
The Lesson: Just because a robot's eyes move like a human's doesn't mean its brain works like a human's. The two things are related, but they aren't the same.

Technical Summary: Attention Alignment Between Humans and Vision-Language Models

Problem Statement

Visual perception is governed by the interplay of bottom-up sensory mechanisms and top-down goals. While Vision-Language Models (VLMs) implement both components, it remains unclear how specific architectural choices in the encoder (bottom-up) and decoder (top-down) shape the alignment of model attention with human visual fixation. Previous research has largely focused on bottom-up saliency or treated training distribution as the primary driver of alignment, often overlooking how architectural biases independently and jointly influence top-down attention mechanisms. This study aims to dissociate these contributions by systematically varying encoder and decoder architectures while controlling for training data, asking whether specific architectural configurations produce attention maps that better mimic human spatial selection and task adaptation.

Methodology

Experimental Design

The authors employed a $2 \times 2$ factorial design to train four image captioning models, isolating the effects of encoder and decoder families:

Encoders: ResNet-101 (CNN) vs. ViT-B/16 (Vision Transformer).
Decoders: LSTM with additive soft attention vs. 4-layer Transformer with multi-head cross-attention.
Baseline Models: Two state-of-the-art VLMs (Molmo 7B-D and Qwen3.5 9B) were evaluated without fine-tuning.

Data and Tasks

Stimuli: 200 naturalistic images from the COCO 2017 validation set.
Human Data: Eye movements were recorded from 49 participants using an EyeLink 1000 Plus (2000 Hz) during two distinct tasks:
1. Describe Task: General scene description.
2. Social Task: Inferring the attention state of a specific actor within the scene.
Ground Truth: Human fixation heatmaps were generated by convolving gaze samples with a Gaussian kernel and downsampling to $256 \times 256$ .

Training Protocol

All four custom models underwent a consistent four-phase curriculum:

Pretraining: Decoder trained on COCO 2014 captions (encoder frozen).
Fine-tuning (Describe): Encoder unfrozen; trained on COCO 2014.
Fine-tuning (Social): Encoder unfrozen; trained on a lab-collected social captioning dataset.
Evaluation: Attention maps were extracted and compared against human heatmaps for the corresponding task.

Evaluation Metrics

Fixation Alignment: Spearman correlation between model attention maps and human fixation heatmaps, normalized against the human-human noise ceiling.
Spatial Spread: Normalized Shannon entropy to measure the diffuseness of attention (concentration vs. distribution).
Task Adaptation: Correlation between attention maps generated for the Social task versus the Describe task to assess task-specific reconfiguration.
Robustness: Simulation of hemispatial neglect via progressive ablation of the left-side spatial grid (7% to 100%) to measure caption degradation.
Neural Prediction: A banded ridge encoding model predicted TRIBE-simulated fMRI responses using model attention maps ("where") and CLIP embeddings ("what"). Variance partitioning was used to isolate unique and shared contributions.

Key Results

1. Decoder Architecture Dominates Alignment

The choice of decoder was the primary predictor of alignment with human fixations, accounting for a 40–50 percentage point difference:

LSTM Decoders: Achieved the highest alignment, reaching 80–87% of the human noise ceiling (e.g., CNN-LSTM: 85–87%).
Transformer Decoders: Showed significantly lower alignment, reaching 40–59% of the ceiling (e.g., ViT-Transformer: 39–52%).
Encoder Contribution: CNN encoders provided a secondary advantage of 5–20 percentage points over ViT encoders, depending on the decoder family. The CNN-LSTM combination was the most aligned overall.
SOTA Models: Molmo 7B-D (76–82%) and Qwen3.5 9B (53–69%) fell between the extremes of the trained models but did not surpass the custom-trained CNN-LSTM.

2. Dissociation Between Alignment and Spatial Properties

A critical finding was the trade-off between alignment accuracy and spatial characteristics:

LSTM Models: While highly aligned with human fixation locations, their attention maps were spatially diffuse (Shannon entropy $\approx 0.999$ ), spreading weights across nearly all 196 spatial positions. They also showed low task differentiation, maintaining similar attention patterns across the Describe and Social tasks.
Transformer Models: Despite lower fixation alignment, they exhibited sharper spatial concentration and stronger task differentiation (e.g., ViT-Transformer showed the most distinct attention maps between tasks).

3. Robustness to Lesioning

In hemispatial neglect simulations, robustness tracked with the degree of distributed processing:

ViT-Transformer: The most distributed architecture was the most resilient to spatial ablation, maintaining caption quality even with significant receptive field loss.
LSTM Models: Showed the greatest degradation under lesioning, particularly at high severity levels where all patches received equal weight.

4. Neural Predictive Accuracy vs. Behavioral Alignment

The study found a dissociation between behavioral alignment and neural predictive power:

CNN-Transformer: Despite ranking third in fixation alignment, this model's attention maps best predicted synthetic brain activity (highest whole-brain $R^2$ ) in the TRIBE simulation.
LSTM Models: Although best aligned to human gaze, they were not the best predictors of neural responses.
Variance Partitioning: "What" (visual/semantic content via CLIP) explained the majority of neural variance, concentrated in language and temporal regions. "Where" (spatial attention) uniquely explained variance primarily in early visual cortex and dorsal attention networks. The two signals were largely shared in higher-order regions, suggesting less separation between "what" and "where" pathways than traditionally assumed.

Significance and Claims

The paper claims that decoder architecture is the dominant factor in determining how well a VLM's attention aligns with human spatial selection, with LSTM decoders outperforming Transformers in behavioral alignment but failing to capture the spatial sharpness and task adaptability of human attention. Conversely, Transformer decoders, while less aligned behaviorally, offer superior spatial acuity and robustness.

The authors highlight a fundamental trade-off:

Top-down mechanisms (Decoders): LSTM decoders act as a bottleneck, forcing a broad, diffuse sampling that mimics human fixation distribution but lacks task-specific sharpness. Transformer decoders distribute attention across heads, allowing for specialization and task adaptation but resulting in lower overall fixation overlap.
Bottom-up mechanisms (Encoders): CNNs consistently outperform ViTs in alignment, reinforcing the idea that local processing biases may be beneficial for mimicking human visual selection.

Crucially, the study demonstrates that fixation alignment and neural predictive accuracy are not interchangeable metrics. A model can be less aligned with human eye movements yet better predict neural responses, suggesting that different architectural biases capture different facets of human-like processing. The findings suggest that current VLMs do not fully capture the full spectrum of human attention, which requires both the diffuse sampling of LSTMs and the sharp, task-adaptive concentration of Transformers.