Comparing Human Gaze and Vision-Language Model Attention in Safety-Relevant Environments

Technical Summary: Comparing Human Gaze and Vision-Language Model Attention in Safety-Relevant Environments

Problem Statement
Safety-aware artificial intelligence (AI) systems, particularly in autonomous driving and robotics, require the ability to identify scene regions that humans perceive as potentially risky. While eye-tracking technology provides a direct measure of human visual attention and reveals a "negativity bias" (a disproportionate sensitivity to threatening stimuli), existing computational models often focus on general visual saliency rather than safety perception. A critical gap exists in understanding whether modern large vision-language models (VLMs), which possess semantic understanding, can spatially align their attention with human gaze in safety-relevant environments without requiring training on eye-tracking data.

Methodology
The study employed a two-pronged approach involving human data collection and AI inference:

1. Human Eye-Tracking Dataset Construction:

   • Stimuli: 33 scene images representing diverse everyday environments (streets, corridors, public spaces, green areas) with varying levels of potential risk (e.g., vehicles, obstacles, flood signs).
   • Participants: 10 university students with normal or corrected-to-normal vision.
   • Apparatus: Pupil Invisible wearable glasses (200 Hz binocular gaze tracking, 30 fps scene camera) were used to record gaze while participants viewed images on a 24-inch monitor for 10 seconds each.
   • Processing: Raw gaze data were filtered for validity, mapped from the scene camera's field of view to the stimulus resolution (1,536 × 1,024 pixels), and converted into individual fixation density maps using Gaussian smoothing ($\sigma = 30$ px). These were averaged to create 33 population-level human gaze heatmaps.

2. AI Saliency Map Generation:

   • Model: GPT-4o was selected for its multimodal reasoning capabilities.
   • Pipeline: An automated Python script prompted GPT-4o via the OpenAI Vision API to act as a "visual-saliency expert." The model was instructed to return a structured JSON object containing 20–25 normalized fixation points $(x, y)$ with saliency weights $(w)$, predicting where humans would look during free viewing.
   • Rendering: The JSON output was rasterized into filled disks ($r=40$ px) on a canvas, followed by Gaussian smoothing ($\sigma = 60$ px) to generate continuous saliency maps comparable to human heatmaps. No eye-tracking data was used to train the model.

3. Evaluation:

   • Spatial alignment between human and AI heatmaps was quantified using four metrics across all 33 images:
     • Pearson Correlation ($r$): Linear spatial agreement.
     • Normalised Scanpath Saliency (NSS): Correspondence of AI values at human fixation points.
     • Kullback–Leibler (KL) Divergence: Distributional difference between the two maps.
     • AUC-Judd: Ability to discriminate fixated vs. non-fixated locations.
   • A cross-model comparison was performed using Gemini Pro, Gemini Flash, and Claude.

Key Results

• GPT-4o Performance: The model demonstrated meaningful spatial alignment with human gaze without prior eye-tracking training.
  • AUC-Judd: $0.806 \pm 0.076$ (substantially above the chance baseline of 0.5).
  • NSS: $0.988 \pm 0.323$ (positive score indicating above-average values at human fixation points).
  • Pearson $r$: $0.515 \pm 0.117$ (moderate but consistent linear correlation).
  • KL Divergence: $1.766 \pm 0.844$.
• Cross-Model Comparison: All tested models (GPT-4o, Gemini Pro, Gemini Flash, Claude) exceeded the AUC-Judd chance baseline and achieved positive NSS scores.
  • Gemini Pro achieved the highest scores in Pearson correlation ($0.571$), NSS ($1.131$), and AUC-Judd ($0.840$), suggesting superior spatial localization.
  • GPT-4o achieved the lowest KL divergence ($1.766$), indicating its predicted saliency distribution shape most closely matched the human attention distribution, despite slightly lower spatial localization scores than Gemini Pro.
• Scene Variability: Alignment was strongest in scenes where risk-related content was spatially localised (e.g., Image 05, Image 02) and weaker in scenes with diffuse human gaze patterns.

Key Contributions

1. Dataset Creation: The study constructed a dedicated eye-tracking dataset covering 33 diverse everyday scene types with varying risk levels, providing population-averaged human gaze heatmaps as a ground truth for safety perception.
2. Zero-Shot Alignment Demonstration: It empirically demonstrated that large VLMs, prompted to identify risks, can generate spatial predictions that align with human attention patterns without requiring specific eye-tracking training data.
3. Evaluation Framework: The paper applies a comprehensive set of complementary saliency metrics to quantify the spatial correspondence between AI-generated risk maps and human gaze in safety-relevant contexts.

Significance and Claims
The authors claim that large vision-language models can identify regions of potential risk that broadly correspond to where humans direct visual attention in safety-relevant scenes. This suggests that VLMs implicitly capture human tendencies to prioritize risky regions, likely due to training on human-generated content, even though the models do not experience psychological "negativity bias" themselves.

The significance of these findings lies in the potential for VLMs to serve as a scalable tool for approximating human attentional patterns in safety-aware AI applications. This capability could support the development of systems (such as autonomous robots or vehicles) that align with human perception of risk without the need for dedicated, expensive eye-tracking hardware during deployment. The authors note that while the results are promising, they are modest; the alignment is not perfect, and the study acknowledges limitations regarding the small participant sample and the sparse nature of the AI's JSON-based output compared to dense human gaze data. Future work is suggested to involve larger, more diverse samples and dynamic video stimuli to further validate these findings.