💻 computer science

GazeOnce360: Fisheye-Based 360{\deg} Multi-Person Gaze Estimation with Global-Local Feature Fusion

This paper introduces GazeOnce360, a novel end-to-end model for estimating 3D gaze directions of multiple people from a single upward-facing fisheye camera, supported by the new MPSGaze360 synthetic dataset and a dual-resolution architecture that fuses global context with local eye features to handle severe fisheye distortion.

Original authors: Zhuojiang Cai, Zhenghui Sun, Feng Lu

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhuojiang Cai, Zhenghui Sun, Feng Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are sitting at a round table in a busy coffee shop. You want to know exactly what everyone around you is looking at—maybe to see who is interested in the menu, who is distracted, or who is making eye contact with you.

In the past, to do this, you would need a team of people with cameras standing around the table, or a complex system of mirrors and lenses. It would be expensive, clunky, and prone to mistakes.

Enter "GazeOnce360."

Think of this new technology as a super-powered, all-seeing eye placed right in the middle of your table, looking straight up at the ceiling. It's a special "fisheye" camera (like the ones used for security or 360-degree photos) that can see everyone in a full circle around it.

Here is how the paper explains this breakthrough, broken down into simple concepts:

1. The Problem: The "Funhouse Mirror" Effect

The main challenge is that fisheye cameras are like funhouse mirrors. They stretch and warp everything.

If you look at a person standing far to the side in a normal photo, their face looks normal.
In a fisheye photo, that same face looks stretched, squished, and rotated.
Previous methods tried to fix this by taking the warped image, cutting it into pieces, straightening them out, and then guessing where people are looking. This is like trying to solve a puzzle by taking it apart, fixing each piece, and then putting it back together. It's slow, and if you make a mistake on one piece, the whole picture is wrong.

2. The Solution: The "One-Step" Magic Trick

The authors built a new AI model called GazeOnce360. Instead of the slow, multi-step puzzle method, this model is like a magician who sees the whole picture at once.

It looks at the warped, stretched fisheye image directly.
It doesn't try to "fix" the image first; instead, it learns to understand the distortion as a natural part of the view.
It instantly tells you: "That person on the left is looking at the coffee cup," and "That person on the right is looking at you."

3. How It Works: The "Dual-Lens" Strategy

To be both fast and accurate, the model uses a clever dual-resolution trick. Imagine a security guard who has two ways of looking at a crowd:

The Wide-Angle View (Low-Res): The guard looks at the whole room from a distance. They can see where everyone is standing and how the group is arranged, but they can't see the details of their eyes.
The Zoom-In View (High-Res): The guard then zooms in on each person's face to see exactly where their pupils are pointing.

GazeOnce360 combines these two views. It uses the "Wide-Angle" to find the people and the "Zoom-In" to read their eyes. It then fuses this information together so it knows exactly who is looking at what, without getting confused by the warped edges of the fisheye lens.

4. The Secret Sauce: Rotational Convolution

Standard AI is used to seeing things upright. If you rotate a picture of a cat, a normal AI might get confused. But in a 360-degree fisheye view, people are rotated in every direction.

The authors gave their AI a special tool called Rotational Convolution.

Analogy: Imagine a standard flashlight that only shines straight ahead. If you turn your head, the light doesn't follow.
The New Tool: This is like a 360-degree spotlight that rotates with the object. No matter how the face is twisted or stretched by the fisheye lens, the AI's "spotlight" rotates to match it, ensuring it always sees the eyes clearly.

5. The Training Ground: The "Virtual Reality" Classroom

You can't easily teach a robot to do this with real people because it's hard to know exactly where a human is looking (you'd have to ask them, and they might lie or forget).

So, the authors built a massive virtual world (using a game engine called Unreal Engine).

They created thousands of digital people with different skin tones, ages, and clothes.
They placed them around a virtual table and programmed them to look in specific directions.
Because it's a computer simulation, the computer knows exactly where every pupil is looking, down to the pixel.
They trained the AI on this "fake" data, and surprisingly, it learned so well that it works perfectly on real-world photos too!

Why Does This Matter?

This isn't just a cool tech demo; it has real-world uses:

Smart Offices: A robot on a table could know if the whole team is bored or engaged during a meeting.
Service Counters: A bank or hotel desk could see if a customer is looking at a brochure or waiting for help.
Virtual Reality: It could make VR interactions feel more natural by understanding where you and others are looking in a 360-degree space.

In short: GazeOnce360 is a smart, fast, and flexible system that turns a distorted, 360-degree view into a clear understanding of human attention, all without needing a team of cameras or a complicated setup. It's like giving a computer the ability to read the room, no matter how the room is arranged.

1. Problem Definition

The paper addresses the underexplored challenge of multi-person 3D gaze estimation in a 360° scene captured by a single upward-facing fisheye camera (typically mounted on a tabletop).

Context: Unlike traditional gaze estimation which uses forward-facing cameras with constrained viewpoints, this setup captures a panoramic view of people sitting around a table (e.g., in collaborative workspaces or reception areas).
Challenges:
- Severe Geometric Distortion: Fisheye lenses introduce extreme radial distortion, particularly at the periphery, breaking standard translation-invariant assumptions of CNNs.
- Perspective Variation: Faces appear at extreme angles and varying scales.
- Pipeline Complexity: Existing solutions (e.g., GAM360) rely on multi-step pipelines (fisheye unwarping $\rightarrow$ face detection $\rightarrow$ perspective projection $\rightarrow$ gaze estimation), which are computationally expensive, prone to cumulative errors, and difficult to deploy.
- Data Scarcity: There is a lack of large-scale, annotated datasets for multi-person gaze estimation in upward-facing fisheye configurations.

2. Methodology: GazeOnce360

The authors propose GazeOnce360, a novel end-to-end, dual-resolution architecture designed to process raw fisheye images directly without intermediate projection steps.

A. Core Architecture Components

Rotational Convolution (RotConv):
- To handle the radial distortion and varying viewing angles inherent in fisheye images, the model integrates Rotational Convolution into the top levels of the Feature Pyramid Network (FPN).
- Instead of standard kernels, RotConv replicates kernels in four orthogonal orientations and aggregates their responses via weighted averaging. This enhances rotation invariance and geometric consistency for faces appearing at extreme angles.
Dual-Resolution Feature Fusion:
- Global Branch (Low-Resolution): Processes the full fisheye image to capture global spatial context, crowd arrangement, and coarse person-to-camera geometry.
- Local Branch (High-Resolution): Processes cropped high-resolution face regions to extract fine-grained ocular and periocular features crucial for precise gaze direction.
- Cross-Attention Fusion: A cross-attention mechanism fuses the global context (queries) with local facial features (keys/values). A spatial mask ensures attention is focused on relevant facial regions within the global feature map. This balances efficiency (low-res context) with precision (high-res details).
Multi-Task Supervision:
- The network predicts multiple outputs simultaneously: face confidence, bounding boxes, head pose, distance, gaze direction, and facial/eye landmarks.
- Eye Landmark Supervision: A key innovation is the use of precise eye landmark annotations (pupil centers, eye centers) as a supervisory signal. This guides the network to learn spatially meaningful ocular representations, significantly improving gaze accuracy.

B. Training Strategy

The model is trained end-to-end using a multi-task loss function combining classification, regression (Smooth L1), and landmark losses.
During training, the local branch uses ground-truth bounding boxes for cropping; at inference, it uses predicted boxes.

3. Key Contributions

A. MPSGaze360 Dataset

The authors introduce MPSGaze360, the first large-scale synthetic dataset for this specific task.

Generation: Created using Unreal Engine 5 and MetaHuman characters.
Scale: Contains 23,496 fisheye images with 1–7 people per scene.
Annotations: Includes pixel-perfect ground truth for 3D gaze vectors, 3D head poses, face bounding boxes, and precise eye landmarks (which are difficult to annotate manually in real data).
Diversity: Covers varied lighting, skin tones, ages, and extreme head poses/gaze directions.

B. Novel Model Architecture

End-to-End Efficiency: Eliminates the need for complex multi-stage pipelines (unwarping/projection), reducing cumulative errors and latency.
Distortion Robustness: Successfully adapts deep learning to fisheye imagery via Rotational Convolutions.
Global-Local Fusion: The dual-resolution design effectively manages the trade-off between processing the entire 360° scene and analyzing fine-grained eye details.

4. Experimental Results

Experiments were conducted on the MPSGaze360 dataset, with validation on real-world fisheye images to test generalization.

Ablation Studies:
- RotConv: Reduced gaze error from 12.14° (baseline) to 11.14°.
- Eye Landmark Supervision: Further reduced error to 8.89°, demonstrating that precise eye geometry cues are critical.
- Dual-Resolution: Achieved comparable accuracy to a full high-resolution model (8.97° vs 8.95°) but with 22% higher inference speed (16.23 FPS vs 13.30 FPS).
Comparison with State-of-the-Art (GAM360):
- GazeOnce360 outperformed the multi-stage GAM360 method significantly.
- Accuracy: Reduced gaze error from 18.96° (GAM360) to 10.39°.
- Speed: Achieved 16.23 FPS compared to GAM360's 4.23 FPS.
Generalization:
- The model maintained stable performance in cross-identity and cross-scene settings (gaze error ~10.4°), despite being trained purely on synthetic data.
- Qualitative results on real fisheye images confirmed the model's ability to generalize to unconstrained environments.

5. Significance and Impact

Practical Deployment: By enabling a single, compact, upward-facing camera to monitor gaze for multiple people in a 360° space, this work facilitates applications in collaborative robotics, smart meeting rooms, and service counters without the need for complex multi-camera arrays.
Methodological Advancement: It demonstrates that Rotational Convolutions are superior to standard or deformable convolutions for fisheye perception tasks, providing a new direction for handling omnidirectional distortions.
Data-Driven Insight: The success of eye landmark supervision highlights the importance of fine-grained geometric cues in gaze estimation, a finding enabled by the high-fidelity synthetic dataset.
Foundation for Future Work: The paper establishes a baseline for fisheye-based gaze estimation, identifying limitations (e.g., extreme poses, occlusions) and paving the way for future research in cross-domain robustness.