🤖 AI

When Eyes Betray AI: Social Gaze Consistency as a Semantic Cue for AI-Generated Image Detection

This paper introduces "Social Gaze Consistency" as a novel, high-level semantic cue for detecting AI-generated images by analyzing the mutual coherence of gaze, head-eye alignment, and pupil placement between interacting individuals, demonstrating through controlled datasets and cross-architecture validation that this approach effectively overcomes the limitations of existing low-level artifact detection methods.

Original authors: Kim Jihyeon, Sohee Kim, Soosan Lee, Souhwan Jung, James Matthew Rehg, Hyesong Choi

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kim Jihyeon, Sohee Kim, Soosan Lee, Souhwan Jung, James Matthew Rehg, Hyesong Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Perfect" Fake

Imagine a world where forgers have become so good at painting that you can no longer tell a fake painting from a real one by looking at the brushstrokes, the texture of the canvas, or the way the light hits the paint.

For a long time, computers detecting AI images worked like art inspectors looking for these "brushstrokes." They looked for tiny digital glitches, weird pixel patterns, or strange frequencies (the "low-level" clues). But modern AI generators (like FLUX.1 or DALL·E 3) have learned to paint so perfectly that these tiny glitches are gone. The "brushstrokes" are now invisible.

The New Idea: The "Social Eye Contact" Test

The authors of this paper say: "If we can't look at the paint, let's look at the story."

They introduce a new way to spot fakes called Social Gaze Consistency. Think of it like this:

The Real World: When two people talk, their eyes naturally lock onto each other. If Person A is looking at Person B, Person B's eyes are usually looking right back. Their heads and eyes are aligned in a way that makes geometric sense.
The AI World: Current AI is great at making a single face look real. But when it generates a picture of two people interacting, it often messes up the connection. One person might be looking at the other, but the other person's eyes might be staring slightly past them, or their pupils might be in the wrong spot relative to their head.

The AI is so focused on making each individual face look "photorealistic" that it forgets the social geometry of how two people actually look at each other. The paper argues that this "social disconnect" is a new, high-level clue that AI detectors have been missing.

How They Built the Detector (The "Training Gym")

To teach a computer to spot this, the researchers had to build a special training gym.

The "Controlled Surgery" Dataset:
Instead of just showing the computer random real and fake photos, they took real photos of people looking at each other and used AI to surgically "edit" just the eye region. They kept the rest of the photo (the face, the background, the lighting) exactly the same.
- Analogy: Imagine taking a real photo of a handshake and using AI to redraw just the fingers. If the fingers look weirdly connected to the hand, you know it's fake. By keeping everything else identical, they forced the computer to learn only about the eyes, ignoring other tricks the AI might use.
The "Scripted Reasoning" Teacher:
They didn't just ask the computer, "Is this real or fake?" (Yes/No). They forced it to write a short explanation using a specific 5-step template:
1. Decision: "This is a fake image."
2. Scene: "There are two people here."
3. Method: "I am checking their eye contact."
4. Evidence: "Person A is looking at Person B, but Person B's eyes are looking at the floor."
5. Conclusion: "Therefore, this is fake."
- Analogy: Instead of letting a student guess the answer, the teacher forces them to show their work using a specific formula. This stops the computer from just memorizing "patterns" and forces it to actually understand the logic of eye contact.

The Results: The "Magic" Works

They tested this new detector against the old ones on three different challenges:

The "Eye Surgery" Test: On the photos they created themselves, the new detector was nearly perfect (99.9% accuracy), while old detectors failed miserably.
The "Single Person" Test: On photos of just one person, it did slightly better than the old detectors, but not by a huge amount.
The "Group Chat" Test (The Big Win): On photos of people interacting (looking at each other), the new detector jumped significantly ahead.
- The Metaphor: The old detectors were like security guards looking for fake IDs (low-level clues). The new detector is like a bodyguard who notices that two people aren't making eye contact, even if their IDs look perfect.

Why It Matters

The paper claims that as AI gets better at hiding its "digital fingerprints," we need to start looking at social logic. Just because an image looks sharp and clear doesn't mean the people in it are behaving like real humans.

Key Takeaway:
The paper proves that AI is currently bad at simulating the subtle, geometric rules of how humans look at each other. By training computers to spot these "social awkwardness" errors in eye contact, we can catch fakes that the old methods miss.

What the paper does NOT claim:

It does not say this works for all types of AI images (like landscapes or animals). It is specifically for people interacting.
It does not claim this is a permanent fix forever; it just says it works right now against current AI models.
It does not suggest using this for medical diagnoses or legal court cases yet; it is a research tool for detecting image manipulation.

Technical Summary: When Eyes Betray AI

Problem Statement

Recent advancements in generative models (e.g., Stable Diffusion 3, FLUX.1, DALL·E 3, Midjourney v6) have effectively eliminated the low-level artifacts that previously served as the primary basis for AI-generated content (AIGC) detection. Signals such as frequency anomalies, upsampling traces, and pixel-level fingerprints are now largely mitigated, causing established detectors (NPR, UnivFD, AIDE) to collapse to near-chance performance on contemporary, person-centric datasets. Furthermore, the threat landscape has shifted toward "partial-edit" fakes, where inpainting models regenerate localized regions of authentic photographs. These fakes retain the global photometric statistics of the original image, evading detectors that rely on globally distributed cues.

The paper argues that detection requires a shift from low-level statistical analysis to a high-level semantic-consistency axis. Specifically, it posits that while generators can produce locally photorealistic faces, they struggle to maintain the geometric coherence of social gaze in multi-person interactions.

Methodology

1. Social Gaze Consistency as a Semantic Cue

The authors define Social Gaze Consistency as the mutual coherence of gaze direction, head–eye alignment, and pupil placement between interacting individuals. Unlike prior work that treats gaze as an action-recognition target or focuses on intra-face anatomical priors (e.g., pupil shape irregularities), this work treats gaze as a falsifiable forensic prediction. Authentic photographs satisfy these geometric relations by construction, whereas contemporary inpainting models systematically violate them in the eye region.

2. Custom Gaze Dataset Construction

To isolate the gaze cue from confounding generator fingerprints, the authors constructed the Custom Gaze dataset (46,830 paired images).

Pair-Level Identity Preservation: Each fake image is generated from a real source image (derived from the OI Mutual Gaze dataset) by regenerating only the eye region using FLUX.1-Fill. The base identity, scene, lighting, and global composition remain invariant.
Shortcut Blocking: This paired structure prevents models from learning generator-specific global fingerprints (e.g., FLUX-specific spectral residuals) as a shortcut, forcing the model to rely on the anatomically grounded differences in the eye region.
Masking: A specific mask ( $M_{eye}$ ) covering the vertical band $[0.25h, 0.55h]$ of the face bounding box is used for inpainting.

3. Block-Compositional Caption Supervision

The paper introduces a novel supervision schema to inject a reasoning skeleton into the training of Vision-Language Models (VLMs), specifically FakeVLM (LLaVA-v1.5 + Vicuna-7B).

Structure: Instead of free-form generation or binary classification, the model is trained to output a 5-block caption: Decision $\to$ Scene $\to$ Method (gaze-based reasoning) $\to$ Evidence $\to$ Conclusion.
Compositional Diversity: The reasoning skeleton (the logical flow) is invariant, but the surface lexicalization is varied. By sampling from macro-pools of 5 variants for 4 positions, the system generates 1,250 unique captions from only 20 macro-pool entries.
Rationale: This design decouples reasoning consistency from output diversity, preventing mode collapse while ensuring the model internalizes the specific "gaze-centric" reasoning path rather than defaulting to texture-based explanations.

4. Training Protocol

Mixed Fine-Tuning: The model is fine-tuned on a mixture of the Custom Gaze dataset and the existing FakeClue dataset (texture-centric supervision) to prevent distributional collapse on non-person categories.
Checkpoint Selection: The authors select the checkpoint based on Balanced Accuracy (BA) rather than validation loss, noting a significant decoupling where loss minimization does not correlate with class-balanced performance.
Architecture: Parameter-efficient fine-tuning (LoRA) is applied to the vision encoder and LLM, with full fine-tuning on the vision-language projector.

Key Contributions

A New Detection Axis (C1): The introduction of Social Gaze Consistency as a semantic detection axis orthogonal to low-level paradigms. The authors provide a four-step mechanistic account explaining its cross-generator transferability:
- Paired-edit shortcut blocking: Prevents memorization of generator fingerprints.
- Hard-to-easy difficulty transfer: Training on partial edits (hard) improves performance on full synthesis (easy).
- CLIP prior preservation: Leverages pre-trained CLIP representations shared across generators.
- Diffusion-family shared spectral weakness: Exploits consistent spectral weaknesses in periocular structures across diffusion-based generators.
Supervision Design (C2): The proposal of Block-Compositional Caption Supervision, which uses a fixed reasoning skeleton with surface variation to force the model to articulate why an image is fake based on gaze geometry, rather than just predicting a label.
Cross-Architecture Validation (C3): Demonstration that the gaze cue is architecture-agnostic. The supervision design improves a 7B parameter VLM (FakeVLM) and a vision-only backbone (Effort) across multiple benchmarks, proving the signal is not specific to a single model architecture.

Experimental Results

The method was evaluated on three benchmarks: Custom Gaze (partial-edit hard fakes), COCOAI Person (single-person AIGC), and COCOAI Interaction (multi-person interaction).

Performance Gains:
- On the COCOAI Interaction subset (the core diagnostic benchmark), the proposed method improved Balanced Accuracy (BA) from 67.8% to 71.5% (+3.7 pp) over the FakeVLM origin, with a massive gain in Macro-F1 (+7.5 pp) and MCC (+13.3 pp).
- On COCOAI Person, BA improved from 83.0% to 84.3% (+1.3 pp).
- On Custom Gaze, the method achieved near-perfect performance (99.9% BA), saturating the in-distribution ceiling.
Comparison to Baselines:
- Low-level detectors (AIDE, NPR, UnivFD) collapsed to near-random performance (BA $\le$ 51.5%) on the interaction subset.
- The proposed 7B model outperformed the 13B parameter competitor SIDA-13B by +14.4 pp in mean BA and +30.5 pp in MCC, demonstrating that supervision design outweighs parameter count.
Confusion Matrix Analysis: The gains were driven by a substantial recovery of "fake" classifications (+17 on the interaction subset) with near-zero degradation in "real" classification, ruling out a "predict-all-fake" artifact.
Cross-Architecture Transfer: Fine-tuning the vision-only Effort backbone on the Custom Gaze dataset yielded similar patterns of in-distribution saturation and cross-distribution transfer, confirming the cue's independence from the VLM architecture.

Significance and Claims

The paper claims that as low-level artifacts recede, social-semantic consistency becomes the critical frontier for AIGC detection. The significance of this work lies in:

Shifting the Paradigm: Moving detection from pixel statistics to high-level geometric relations (gaze consistency) that current diffusion models fail to enforce explicitly.
Effective Supervision: Demonstrating that how a model is asked to explain its prediction (structured reasoning vs. free-form or binary) is as consequential as the model architecture itself.
Robustness: Providing evidence that a cue derived from a single generator (FLUX.1-Fill) and a specific task (partial eye editing) can generalize to multi-generator, full-image synthesis scenarios across different model families.

The authors acknowledge limitations, including domain specialization (person-centric only), reliance on a single inpainter for dataset construction, and a tendency toward "card over-trust" (over-reliance on gaze cues) in single-person scenarios, which they suggest could be mitigated by principled gating mechanisms in future work.