💻 computer science

SGAP-Gaze: Scene Grid Attention Based Point-of-Gaze Estimation Network for Driver Gaze

This paper introduces SGAP-Gaze, a novel Point-of-Gaze estimation network that integrates driver facial features with traffic scene context via a Transformer-based attention mechanism, achieving state-of-the-art accuracy on a new benchmark dataset (UD-FSG) by significantly reducing mean pixel error compared to existing models.

Original authors: Pavan Kumar Sharma, Pranamesh Chakraborty

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pavan Kumar Sharma, Pranamesh Chakraborty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why Do We Need This?

Imagine you are driving a car through a chaotic, busy city. Cars are weaving in and out, rickshaws are cutting across lanes, and pedestrians are stepping off curbs. To stay safe, you don't just stare straight ahead; you constantly scan your surroundings.

The Problem: Current "Driver Monitoring Systems" (the cameras inside cars that check if you are sleepy or texting) are like a security guard who only looks at you. They know if your eyes are closed or if you are looking at your phone, but they don't know what you are actually looking at in the traffic outside. They don't know if you are watching a child about to run into the street or just staring at a billboard.

The Solution: This paper introduces a new system called SGAP-Gaze. It's like giving the security guard a pair of binoculars that look at the road and a magnifying glass that looks at your eyes, then combining those two views to guess exactly where your attention is focused on the road.

The Ingredients: A New "Recipe" for Data

To teach a computer to do this, you need a massive cookbook of examples. The authors created a new dataset called UD-FSG (Urban Driving-Face Scene Gaze).

The Old Way: Most previous datasets were like taking photos in a parking lot. The car was stopped, the traffic was simple, and the lighting was perfect.
The New Way (UD-FSG): The authors drove around a real, chaotic city (Kanpur, India) with a special car. They recorded:
1. Your Face: A camera inside looking at you.
2. The Road: A camera outside looking at the traffic.
3. The Truth: The drivers wore special glasses (eye-trackers) that told the computer exactly where their eyes were looking.

Think of this dataset as a synchronized movie where every frame shows the driver's face, the road ahead, and a "red dot" showing exactly where the driver is looking. This is the first time such a large, messy, real-world dataset has been made available.

How the System Works: The "Detective" Pipeline

The AI model (SGAP-Gaze) works like a detective solving a mystery. It doesn't just guess; it gathers clues from three different sources and combines them.

1. The Clue Gatherers (Feature Extraction)

The Face Clue: The system looks at the driver's face to see their general head position.
The Eye Clue (The Secret Weapon): It zooms in on the eyes and specifically the iris (the colored part). It uses a special math trick (a "Gaussian weight") to say, "Hey, the iris is the most important part! Let's pay extra attention to it and ignore the white part of the eye."
The Scene Clue: It looks at the road. It doesn't just see a blurry mess; it breaks the road image into a 7x7 grid (like a tic-tac-toe board). It analyzes what is happening in each square of that grid.

2. The Detective's Brain (The Transformer & Attention)

This is the coolest part. The system uses a mechanism called Attention, similar to how a teacher in a classroom focuses on a specific student.

The "Gaze Intent Vector" (a summary of what the driver's face and eyes are saying) acts as the Teacher.
The "Scene Grid" (the 49 squares of the road) acts as the Students.
The Teacher asks the grid: "Which of you is the driver looking at?"
The grid squares that match the driver's eye movement get a high "attention score." The system then calculates the average location of those high-scoring squares to pinpoint the exact spot on the road.

Analogy: Imagine you are in a crowded room (the scene grid). Your friend (the driver) points vaguely in a direction. Instead of guessing the whole room, the system looks at your friend's eyes and head, then asks the people in the room, "Who is he looking at?" The system then draws a circle around the person who got the most "attention."

The Results: Did It Work?

The authors tested their new detective against the current best detectives (state-of-the-art models).

The Scorecard: They measured "Pixel Error." Imagine the road is a giant screen. If the system guesses the wrong spot, how many pixels off is it?
- Old Models: Made mistakes about 137 pixels away from the truth.
- SGAP-Gaze: Made mistakes only 105 pixels away.
- The Win: This is a 23.5% improvement. That's a huge jump in accuracy.
The "Edge" Case: The real magic happened at the edges of the screen (the corners of the windshield). Most systems get confused when drivers look left or right. SGAP-Gaze was much better at these difficult, peripheral glances, which are actually the most critical for avoiding accidents.

Why This Matters

In the past, car safety systems were like a sleep monitor. They could tell you if you were falling asleep.
This new system is like a situational awareness coach. It can tell you, "You are looking at your phone, not the car merging in front of you," or "You are looking at the billboard, but a child just stepped off the curb."

By combining what the driver looks like with what the driver sees, this technology creates a much safer, smarter way to monitor drivers in the real, chaotic world.

1. Problem Statement

Driver gaze estimation is critical for assessing situational awareness and preventing accidents, particularly in complex, heterogeneous traffic environments (e.g., India). Existing driver monitoring systems (DMS) primarily focus on detecting in-cabin distractions (drowsiness, phone usage) or estimating gaze based solely on facial features.

Limitations of Current Approaches: Most state-of-the-art models rely exclusively on driver face/eye data to predict a 3D gaze vector or a Point-of-Gaze (PoG). They often ignore the scene context, despite evidence that driver attention is heavily influenced by surrounding traffic agents (vehicles, pedestrians).
Data Scarcity: Existing datasets are either zone-based (coarse classification), collected in parked vehicles, or lack synchronized scene images with ground-truth gaze coordinates. The only existing dataset with face and scene data (LBW) has limited traffic density and does not utilize scene images as direct input for gaze estimation.

2. Key Contributions

The authors propose three major contributions to address these gaps:

UD-FSG Dataset: A new large-scale benchmark dataset (Urban Driving-Face Scene Gaze) collected in real-world, heterogeneous traffic conditions. It contains 373,488 synchronized pairs of driver face images, forward-facing scene images, and 2D ground-truth gaze coordinates derived from eye-trackers.
Iris-Weighted Eye Feature Extraction: An improved method for extracting eye features using a Gaussian-weighted function centered on the iris. This emphasizes the iris region within the eye image to better capture movement patterns while suppressing peripheral noise.
SGAP-Gaze Architecture: A novel Scene Grid Attention-based Point-of-Gaze network. Unlike previous models that regress coordinates directly, this model treats gaze estimation as an attention problem over a spatial grid of the scene, fusing multi-modal facial cues with scene context using a Transformer-based mechanism.

3. Methodology

The proposed framework, SGAP-Gaze, follows a four-stage pipeline:

A. Facial Geometry Extraction (FEI Detector)

A custom YOLOv8 model detects the Face, Eyes, and Iris regions.
It calculates iris center coordinates relative to the eye bounding box.
Robustness Mechanism: If iris detection fails (due to blur/occlusion), a gating mechanism neutralizes the iris input, forcing the model to rely on face and eye appearance features to prevent noisy signals.

B. Multi-Stream Feature Extraction

Features are extracted using a pre-trained ResNet-18 backbone across three streams:

Face Features: Multi-level features (Layers 1–4) are extracted, projected to 256-D, and pooled to form a global face embedding.
Eye Features: Left and right eye crops are padded to 224×224, processed through ResNet-18, and then weighted by a 2D Gaussian function centered at the iris coordinates. This creates an "iris-weighted" eye embedding.
Scene Features: The forward-facing scene image is processed to produce a 7×7 spatial feature map (49 grid tokens). Each token represents a localized region of the driving scene.

C. Multi-Modal Feature Fusion

Gaze Intent Vector: Face, eye, and iris features are concatenated and compressed into a unified gaze intent vector ( $z_{gaze}$ ). This vector acts as the Query in the attention mechanism.
Scene Grid Representation: The 49 scene grid tokens act as Keys and Values.

D. Gaze Prediction Head (Transformer Attention)

Attention Mechanism: The model computes attention scores between the gaze intent vector (Query) and the 49 scene grid tokens (Keys) using dot-product similarity.
PoG Estimation: The final Point-of-Gaze is calculated as the weighted average of the normalized center coordinates of the scene grids, based on the computed attention scores.
Refinement: A learnable vertical residual correction is applied to compensate for grid discretization errors.
Loss Functions: The model is trained using a hybrid loss for 3D direction (Cosine + L2) and a Smooth L1 loss for PoG regression to handle outliers.

4. Experimental Results

The model was evaluated on the UD-FSG dataset and compared against the LBW dataset and the state-of-the-art GazePTR model.

Performance on UD-FSG:
- Achieved a Mean Pixel Error (MPE) of 104.73 pixels (7.13% normalized error).
- Compared to GazePTR (MPE 136.96), SGAP-Gaze achieved a 23.5% reduction in error.
- Modalities Analysis: Adding scene context (Face + Eye + Scene) reduced error and standard deviation compared to using Face or Face+Eye alone, proving the value of scene awareness.
Performance on LBW:
- Achieved an MPE of 63.48 pixels (5.89% normalized error), demonstrating strong generalization.
Spatial Analysis:
- SGAP-Gaze consistently outperformed GazePTR across all spatial regions.
- Critical Improvement: The model showed significant gains in outer regions (near windshield edges), where gaze targets are rare but critical for safety. GazePTR struggled significantly in these peripheral zones.
3D Gaze Direction:
- Using only facial features, the model achieved a Mean Angular Error (MAE) of 6.04°, outperforming GazePTR (6.18°) and other SOTA models like ETH-XGaze.

5. Significance and Conclusion

Scene Awareness: The study proves that incorporating scene information directly into the gaze estimation model (rather than just generating a saliency map) significantly improves accuracy, especially for peripheral gaze targets.
Robustness: The iris-weighted feature extraction and the gating mechanism make the model robust to real-world challenges like motion blur and occlusion.
Safety Impact: By accurately predicting where a driver is looking in complex traffic (including the "blind spots" or edges of the scene), this system can better detect inattentive driving and potential collision risks.
Future Work: The authors suggest integrating side-mirror views and camera calibration parameters to further enhance scalability across different vehicle types.

In summary, SGAP-Gaze sets a new benchmark for driver gaze estimation by moving beyond face-only analysis to a scene-aware, multi-modal approach, significantly reducing prediction errors in real-world driving scenarios.