💬 NLP

Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues

This paper reveals that Vision-Language Models significantly underperform humans in inferring gaze direction because they erroneously rely on head orientation rather than eye appearance, a bias attributed to training data rather than model architecture.

Original authors: Zory Zhang, Pinyuan Feng, Bingyang Wang, Tianwei Zhao, Suyang Yu, Qingying Gao, Hokin Deng, Ziqiao Ma, Yijiang Li, Dezhi Luo

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zory Zhang, Pinyuan Feng, Bingyang Wang, Tianwei Zhao, Suyang Yu, Qingying Gao, Hokin Deng, Ziqiao Ma, Yijiang Li, Dezhi Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Head-First" Guessing Game

Imagine you are playing a game where you have to guess which object a person in a photo is looking at. There are three items on the table: a bunny, a kettle, and some tulips.

If the person is looking at the bunny, but their head is turned slightly toward the kettle, what do you do?

A human looks closely at the person's eyes, sees the pupils are pointed at the bunny, and says, "They are looking at the bunny."
Current AI (Vision-Language Models) often ignores the eyes. Instead, it looks at the direction of the person's face (their head) and says, "They are looking at the kettle."

The paper argues that these AI models are "lazy" guessers. They rely on a shortcut: If the head is facing an object, the eyes must be looking there too. They fail to do the hard work of actually reading the eyes.

How the Researchers Tested This

The researchers built a special "gaze lab" to test this. They took 1,360 real photos of people sitting at a table with different objects. They set up three specific scenarios to trick the AI:

The "Agreeing" Scenario: The person looks at the bunny, and their head is also facing the bunny. (Easy for everyone).
The "Natural" Scenario: The person looks at the bunny, and their head is facing naturally where they feel comfortable (usually the bunny).
The "Trick" Scenario (Incongruent): This is the key. The person keeps their head facing the kettle, but they move their eyes to look at the bunny.

The Result:

Humans got this right almost every time (about 89% accuracy). They looked at the eyes.
AI Models (like GPT-4o, GPT-5.2, Qwen3) performed very poorly, often barely better than random guessing. When the head and eyes disagreed, the AI almost always guessed the object the head was facing, not the object the eyes were looking at.

Why Did the AI Fail? (The Detective Work)

The researchers didn't just say "AI is bad." They played detective to figure out why. They ruled out several excuses:

Is it because the photos are too blurry? No. When they made the photos super high-definition, the AI still failed.
Is it because the AI can't name the objects? No. The AI could correctly identify the "bunny" and the "kettle" when asked.
Is it because the AI is too small? No. Bigger, more powerful models failed just as much as smaller ones.

The Real Culprit: The Training Data
The researchers believe the problem is what the AI learned from the internet.

The Analogy: Imagine a child who has only ever seen people looking at things they are facing. In the real world, it is rare to see someone staring sideways while their face points forward.
Because the internet data is full of "head and eyes agree" examples, the AI learned a rule: "Head direction = Gaze direction." It never learned that eyes can move independently of the head.

The "Coin Flip" Test

To prove the AI wasn't just confused, the researchers ran a specific test. They asked: Does the AI care at all about the eyes?

They found that the AI's behavior was like a coin flip or a biased scale:

When the head and eyes agreed, the AI was confident.
When they disagreed, the AI didn't try to figure out the eyes. It just defaulted to the head.
Even when the researchers tried to "teach" the AI to look at the eyes by adding instructions like "Focus on the eyes," the AI mostly ignored them and went back to guessing based on the head.

The "Proof of Concept" Experiment

To show that this isn't a permanent flaw in the AI's brain (architecture), but just a bad habit from its training, the researchers did a small experiment:

They took one of the AI models and re-trained it specifically on their "Trick" photos (where head and eyes disagreed).

Before re-training: The model was terrible at the trick scenario (15% accuracy).
After re-training: The model got much better (34% accuracy). It learned to stop relying solely on the head and actually look at the eyes.

The Lesson: The AI can learn to read eyes, but it needs specific data that teaches it to do so. The standard internet data it usually learns from doesn't have enough of these "trick" examples.

Summary

The Problem: Current AI models are terrible at telling where someone is looking if their head is turned in a different direction. They mistake the direction of the face for the direction of the eyes.
The Cause: They were trained on data where heads and eyes usually point the same way, so they learned a lazy shortcut.
The Fix: We need to feed AI more data where people look sideways or up while their head stays still, forcing the AI to learn how to actually read eyes.

The paper concludes that until we fix this data problem, AI will struggle to understand human non-verbal cues, which is essential for robots and computers to interact naturally with people.

1. Problem Statement

Vision-Language Models (VLMs) are increasingly expected to understand nonverbal communication cues, such as gaze, to facilitate human-robot interaction and social AI. While specialized gaze-estimation models exist, they lack the general reasoning capabilities of VLMs. However, it remains unclear whether VLMs can accurately infer gaze targets (what a person is looking at) in zero-shot settings, particularly when distinguishing between eye appearance (the actual direction of the eyes) and head orientation (the direction the face is pointing).

The core problem investigated is whether VLMs possess the core capacity to infer gaze based on subtle eye cues or if they rely on a heuristic bias where they assume a person is looking in the direction their head is facing, even when the eyes are looking elsewhere.

2. Methodology

A. Dataset and Stimuli Construction

The authors constructed a controlled dataset of 1,360 real-world photographs featuring a single person seated at a table with 2–4 objects.

Controlled Variables:
- Gaze Target: The specific object the person is looking at.
- Head Target: The direction the person's head is oriented.
- Conditions:
  - Congruent: Head and eyes point to the same object.
  - Incongruent: Head points to one object, but eyes look at a different object (the critical test case).
  - Natural: Head orientation is unconstrained but comfortable.
- Manipulations: View angle (Front, Left, Right), Object Proximity (1–3, where 3 is closest), and Number of Objects (2–4).
Novelty: The images are real-world photos (not synthetic) to ensure statistical regularities of light and shade, but they are novel enough to minimize training data overlap.

B. Experimental Design

Models Evaluated: The study tested 111 VLMs in a pre-pilot, with a main study focusing on top performers: GPT-4o, GPT-5.2, Qwen3-VL, InternLM-XComposer2, and GLM-4.6V.
Baselines: Included specialized models like Moondream2 (hybrid with specialized gaze function) and GazeLLE (fine-tuned vision model).
Task: A multiple-choice question format asking, "Which object is this person looking at?" with randomized options.
Evaluation Protocol: Each VLM saw 1360 images 16 times with different prompt templates (totaling ~21,760 trials per model) to ensure statistical power.

C. Characterization Framework

The authors proposed a series of empirical tests to categorize model behavior into five hypothesized types (visualized in Figure 2 of the paper):

Gaze-competent: Human-level precision.
Eye-aware: Below-human precision but not heavily biased by head orientation.
Head-dominant: Cannot resolve gaze precisely; heavily relies on head orientation as a heuristic.
Head-only: Largely ignores eye appearance.
Eye-head ambivalent: Randomly flips between head and gaze targets.

Key Tests:

Test 1 (Performance Gap): Compare VLM accuracy against human participants.
Test 2 (Head Bias): Does the model choose the head target more often than the gaze target when they differ? Does performance drop significantly in the Incongruent condition?
Test 3 (Gaze Responsiveness): Does the model's error distance increase as the distance between the head target and gaze target increases?
Test 4 (Coin Flipping): Does the model's choice distribution change based on the number of objects between the head and gaze targets?

D. Mitigation Experiment (Proof-of-Concept)

To determine if the bias is architectural or data-driven, the authors fine-tuned GazeLLE on their specific stimulus set (7:1:2 train/val/test split).

3. Key Results

A. Performance Gap

Humans: Achieved 89% accuracy on the pilot stimuli.
VLMs: Most models performed near random chance (~42%). Even the best models (GPT-5.2, GPT-4o) achieved only 30–46% accuracy on the main study stimuli (Natural Condition).
Specialized Models: Moondream2 and GazeLLE also failed to reach human levels in the zero-shot setting, though GazeLLE performed better than general VLMs.

B. The Head Orientation Bias

Dominant Failure Mode: Most VLMs exhibited a strong Head-Dominant bias.
- In the Incongruent Condition, models frequently selected the object the head was facing rather than the object the eyes were looking at.
- Test 2.1: GPT-5.2 and Qwen3 chose the Head Target over the Gaze Target significantly more often than chance (62% and 57% respectively).
- Test 2.3: Performance dropped drastically when Head Target $\neq$ Gaze Target (e.g., GPT-5.2 dropped from 45% accuracy in Congruent to 28% in Incongruent).
Proximity Effect: Models performed worse when objects were closer together, indicating they use visual information but struggle with fine-grained discrimination.
View Effect: Models showed lower sensitivity to eye appearance when viewed from the side compared to the front, unlike humans who maintain high accuracy.

C. Cause Analysis

The authors ruled out several alternative explanations:

Resolution: Increasing resolution (up to 1024x1024) did not improve accuracy.
Object Naming: Models could identify objects correctly; the error was in linking gaze to the object.
Architecture/Size: Larger models (more parameters) and newer releases did not show significant improvements.
Conclusion: The bias is likely data-driven. Training data (internet images) likely over-represents cases where head and gaze align and under-represents "sidelong glances" or incongruent cues, failing to teach the model the social nuance of eye-head decoupling.

D. Fine-Tuning Results

Fine-tuning GazeLLE on the specific dataset significantly mitigated the bias.
Incongruent Condition: Accuracy jumped from 15.65% (pre-trained) to 34.15% (fine-tuned).
Congruent Condition: Improved modestly (62.82% $\to$ 70.00%).
Implication: The model can learn to rely on eye appearance if exposed to sufficient counter-examples, confirming the data distribution hypothesis.

4. Key Contributions

Empirical Characterization: Introduced a rigorous framework to categorize VLM gaze inference behaviors, moving beyond simple accuracy benchmarks to understand how models fail (e.g., Head-Dominant vs. Eye-Aware).
Dataset Creation: Released a high-quality, controlled dataset of 1,360 real-world images specifically designed to decouple head orientation from gaze direction.
Identification of Bias: Provided strong evidence that current state-of-the-art VLMs rely on a "head-orientation heuristic" rather than analyzing eye appearance, leading to a significant performance gap compared to humans.
Data-Centric Insight: Demonstrated via fine-tuning that this bias is not an inherent architectural limitation but a result of training data distribution, offering a clear path for improvement.

5. Significance and Future Directions

Social AI: For robots and conversational agents to interact naturally, they must understand gaze to resolve reference ambiguity and infer intent. Current VLMs are unreliable for this task.
Evaluation Methodology: The paper advocates for "hypothesis-driven evaluation" and controlled behavioral probing (similar to cognitive science) rather than just benchmarking on internet images, where head heuristics often suffice.
Training Recommendations: Future VLMs require training data that explicitly captures the interplay between body/head orientation and eye appearance, specifically including "incongruent" cases where the head and eyes point in different directions.
Ethical Considerations: The authors warn that while current models lack this skill, the potential for machines to interpret nonverbal cues without consent is a societal risk that must be managed as the technology improves.

In summary, the paper reveals that despite their advanced capabilities, current VLMs are "blind" to the subtle cues of human gaze, defaulting to a simpler, often incorrect, head-orientation heuristic. This limitation is solvable through targeted data curation and fine-tuning.