Attention, not scale, drives human-AI alignment in multimodal language prediction

Technical Summary: Attention, not Scale, Drives Human-AI Alignment in Multimodal Language Prediction

Problem Statement
While recent advances in generative AI have demonstrated that large language models (LLMs) can simulate human behavior in unimodal tasks, it remains unclear to what extent current vision-language models (VLMs) replicate human predictive processing in ecological, multimodal settings. Human language comprehension routinely integrates visual context to constrain the space of upcoming semantic content, a process indexed by eye movements in the Visual World Paradigm (VWP). However, it is uncertain whether adding visual context to computational models improves their alignment with human predictions, or if this alignment is driven primarily by model scale (parameter count) or specific architectural features like attention mechanisms. Furthermore, the extent to which model attention patterns correspond to human gaze during multimodal prediction is largely unexplored.

Methodology
The authors conducted a large-scale, web-based experiment involving 600 human participants and five state-of-the-art pretrained systems (FLAVA, LLaMA with visual adapter, GPT-4, CLIP variants, and BLIP).

• Experimental Design: Participants viewed 100 six-second movie clips from The Prestige and The Usual Suspects. In a "Multimodal Paradigm," they watched video clips with synchronized audio; in "Unimodal" conditions, they received audio-only or visual-only stimuli. Participants rated the relevance of the stimulus for predicting a specified target word on a 0–100 scale. Eye movements were tracked via webcam throughout the task.
• Computational Models: The study evaluated models across three paradigms:
  1. Unimodal (Text-only): Models processed subtitles or prompts without visual input.
  2. Multimodal (Text + Image): Models processed text synchronized with video frames.
  3. Prompt-based: For closed models like GPT-4, the human instructions were used as prompts to generate deterministic predictability estimates (temperature set to 0).
• Models Tested:
  • FLAVA: A 1.3B parameter vision-language encoder.
  • LLaMA-Adapter: A 7B parameter text-only model with a visual adapter layer.
  • GPT-4: Both text-only and text-image versions.
  • CLIP: Variants with ResNet (no attention) and Vision Transformer (ViT, with attention) backbones of varying sizes.
  • BLIP: A model featuring cross-attention mechanisms.
• Analysis Metrics:
  • Alignment: Pearson correlations between model predictability scores and human relevance ratings, corrected against a "human ceiling" (leave-one-participant-out correlation).
  • Attention Overlap: Comparison of model attention heatmaps (from transformer layers) with human eye-tracking heatmaps using Spearman correlation, Normalized Scanpath Saliency (NSS), and Information Gain (IG).
  • Semantic Alignment: Representational Similarity Analysis (RSA) to link model internal representations with human semantic judgments of visual cues.
  • Ablation: Attention weights in CLIP ViT were manipulated (uniform and patch-shuffled) to test the causal role of learned attention.

Key Results

1. Visual Context Enhances Alignment: Adding visual context significantly increased the alignment between model and human predictions across all architectures (average $\Delta r \approx 0.18$). Multimodal FLAVA achieved the highest raw correlation ($\rho = 0.42$), explaining approximately 72% of the explainable human variance, outperforming larger models like GPT-4 and LLaMA in the multimodal setting. Crucially, parameter size did not correlate with improved alignment; smaller, carefully architected models outperformed larger ones.
2. Attention Mechanisms are Critical: Models with Vision Transformer (ViT) backbones (incorporating attention) significantly outperformed those with ResNet backbones (no attention) in predicting human judgments, particularly when visual cues were informative (high human predictability). Permutation tests showed a consistent ViT advantage across all percentiles of predictability.
3. Attention Ablation Confirms Necessity: When learned attention in CLIP ViT was replaced with uniform or patch-shuffled attention, the positive relationship between model confidence and human ratings reversed, indicating that the specific content of learned attention, not just the mechanism's existence, drives alignment.
4. Model Attention Mirrors Human Gaze: Attention maps from BLIP's cross-attention layers (specifically layer 9) showed the strongest overlap with human eye-tracking data, explaining up to 70% of the variance in the human ceiling. This alignment was highest when semantically salient visual cues were present.
5. Cross-Modal Integration: BLIP's cross-attention, which explicitly integrates linguistic context into visual attention, outperformed purely visual attention mechanisms (BLIP-Vision) and contrastive models (CLIP) in aligning with human gaze and semantic judgments. Cross-attention allowed the model to track anticipatory human fixations on semantic cues (e.g., objects, actions, settings).
6. Entropy Reduction: Visual input reliably reduced the predictive entropy of models (e.g., FLAVA $\Delta H = 0.26$ bits), confirming that visual context constrains the model's predictive distribution similarly to how it constrains human cognition.

Significance and Claims
The paper argues that selective attention to informative cues, rather than sheer model scale, is the principal driver of human-AI alignment in multimodal language prediction.

• Architectural Constraints over Scale: The findings challenge the notion that scaling parameters alone yields human-like behavior. Instead, the study suggests that architectural features enabling the effective integration of modalities (specifically cross-attention) are more critical for grounding language in visual context.
• Multimodal Grounding: The results suggest that models trained jointly on multimodal data (like FLAVA and BLIP) may achieve better behavioral alignment than models trained on text alone and adapted post-hoc, supporting the idea that embodied or structural alignment is necessary for effective multimodal processing.
• Attention as a Mechanism: The study provides evidence that transformer attention mechanisms, particularly cross-modal attention, function similarly to human attention by selectively focusing on semantically relevant visual information to constrain language prediction. The overlap between model attention and human gaze suggests that these models are not merely processing data statistically but are engaging in a form of predictive processing that mirrors human cognitive strategies.
• Limitations and Scope: The authors note that while their models approximate human behavior, the alignment is not perfect. They highlight that models like CLIP tend to focus on peripheral areas when predictive cues are absent, whereas humans and cross-attention models centralize attention, suggesting that current architectures still lack the full flexibility of human uncertainty handling.

In conclusion, the paper posits that the path toward human-like AI in multimodal settings lies not in simply increasing model size, but in refining architectural mechanisms—specifically attention—that allow for the dynamic, semantically guided integration of visual and linguistic information.