HiFiGaze: Improving Eye Tracking Accuracy Using Screen Content Knowledge

HiFiGaze improves gaze estimation accuracy on consumer devices by leveraging known screen content to segment and analyze the screen's reflection in the user's eyes, achieving an approximately 8% reduction in mean tracking error compared to baseline models.

The Gist

The Big Idea: Turning Your Eyes into a Mirror

Imagine you are looking at your smartphone. Your eyes are like tiny, curved mirrors. Just like a mirror in a hallway reflects the room behind you, your eyes reflect the screen you are staring at.

For a long time, eye-tracking technology on phones has been a bit like trying to guess where someone is looking by just looking at their face. It's okay, but it's often a little fuzzy—like trying to read a sign from 50 feet away. You might be off by a few inches, which is annoying if you want to tap a small button just by looking at it.

HiFiGaze is a new method that says: "Wait a minute! We don't just need to look at the eyes; we need to look at what the eyes are reflecting, and we need to know exactly what is on the screen."

The Problem: The "Glitchy Glint"

In the past, researchers tried to find a "glint" (a tiny spark of light) in the eye to figure out where you are looking.

• The Analogy: Imagine you are looking at a TV screen. If the screen is just a blank white wall, the reflection in your eye is a simple white dot. Easy to find.
• The Reality: But screens are full of movies, games, and text. The reflection in your eye is now a tiny, distorted, colorful mess. If you just look for a "dot," you might get confused. Is that reflection from a bright logo? Or a white letter? Or a glare? It's like trying to find a specific needle in a haystack that is constantly changing shape.

The Solution: The "Smart Detective"

HiFiGaze works like a super-smart detective who has two pieces of evidence:

1. The Crime Scene Photo: A high-quality picture of your eye (taken by your phone's 4K selfie camera).
2. The Suspect List: The phone knows exactly what is currently displayed on the screen.

How it works:
Instead of guessing, the phone takes the image of what's on the screen (like a tiny, blurry thumbnail) and tries to "match" it against the reflection in your eye.

• The Analogy: Think of it like a jigsaw puzzle. The phone has the picture of the puzzle piece (the screen content) and the box lid (the reflection in your eye). It slides the piece around until it fits perfectly. Once it finds the perfect fit, it knows exactly where your eye is looking.

Because the phone knows the screen content, it doesn't get confused by the messy colors. It can say, "Ah, that reflection matches the 'Play' button on the video player, so the user is looking at the bottom right."

Why This is a Big Deal

The researchers tested this on an iPhone 14 Pro Max. Here is what they found:

• Old Way (Just looking at the eye): About 2 cm of error. (That's like missing a coin on a table by the width of two fingers).
• New Way (HiFiGaze): About 1.6 cm of error.
• The Result: This is an 18% improvement. In the world of eye tracking, that's huge. It means you can tap small icons just by looking at them without needing to calibrate your eyes first.

The "Eyelash" Problem and the Bottom Camera

There was one snag. When you look down at the bottom of your phone, your eyelashes and upper eyelid sometimes block the reflection, like a curtain closing over a window. This made the system less accurate for the bottom of the screen.

The Fix: The researchers did a funny experiment where they turned the phone upside down so the camera was at the bottom.

• The Analogy: It's like moving a security camera from the ceiling to the floor. Suddenly, the "curtain" (eyelashes) isn't blocking the view anymore.
• The Result: Accuracy got even better! This suggests that future phones might put cameras at the bottom of the screen (or under the screen) to make eye tracking perfect.

Why Should You Care?

You don't need special glasses or expensive headsets for this. It uses the cameras and screens you already have in your pocket.

• Accessibility: It could help people with limited hand movement control their phones just by looking.
• Gaming: Imagine playing a game where you aim a weapon just by looking at the target.
• Privacy: The system can work with just a blurry "heat map" of the reflection, meaning it doesn't need to store a high-definition photo of your actual eye, which is safer for your privacy.

In a Nutshell

HiFiGaze is like giving your phone a pair of super-vision glasses. It realizes that your eyes are mirrors reflecting the screen, and by comparing that reflection to what the phone knows is on the screen, it can guess where you are looking with much higher precision than ever before. It turns your phone from a "dumb" screen into a device that truly understands your attention.

Technical Summary

1. Problem Statement

High-accuracy gaze tracking on consumer devices (smartphones, laptops) is traditionally limited by the reliance on appearance-based methods. These methods use standard RGB cameras to analyze eye features (pupil shape, iris texture) but often suffer from modest accuracy (typically ~2 cm error) without per-user calibration.

The authors identify a specific, underutilized signal: the reflection of the device's own screen in the user's cornea. While high-resolution (4K+) front-facing cameras on modern devices can capture this reflection, conventional approaches fail to utilize it effectively because:

1. Content Variability: Screen content is nearly infinite, making it difficult for a model to distinguish a "glint" from general screen noise.
2. Misalignment: Naive glint detection often fails to locate the correct reflection center when the screen contains complex patterns or dark areas.

The core challenge is how to leverage the device's knowledge of its own screen content to robustly segment and interpret the eye reflection for accurate gaze estimation.

2. Methodology: HiFiGaze

HiFiGaze is a gaze estimation framework that fuses high-resolution eye imagery with screen content knowledge. It operates without specialized hardware (IR illuminators or dedicated trackers).

A. Hardware & Data Acquisition

• Device: Tested on an iPhone 14 Pro Max (12MP front camera capable of 4K video).
• Dataset: Utilized the WebUI dataset (~300k mobile web pages) to generate diverse, realistic screen content (bright/dark, dense/sparse).
• Ground Truth: Users followed smooth pursuit targets moving across the screen at 100 pixels/s.

B. Preprocessing Pipeline

1. Iris Segmentation: Uses MediaPipe for initial face/eye detection but applies a custom correction step. It uses the GrabCut algorithm for iris contour extraction and RANSAC for outlier removal to achieve a stable iris center estimate (critical for vector calculation).
2. Screen Thumbnail Generation: The current screen content is downscaled to a 50×101 pixel thumbnail and blurred to match the low-resolution appearance of the reflection on the eye.

C. Input Encoding Strategies

The authors evaluated seven different input combinations for a neural network (based on MobileNetv4). The key encodings include:

1. Eye Crops: Standard RGB patches of the eyes (Baseline).
2. Screen Thumbnail: The blurred, downscaled screen image.
3. Reflection Heatmap: A correlation map generated by convolving the screen thumbnail over the segmented iris region to find the best matching reflection area.
4. Reflection Vector: A normalized 2D vector derived from the center of the iris to the center of the detected screen reflection. This vector encodes the gaze target relative to the screen.

D. Model Architecture

The model concatenates:

• Eye bounds (normalized coordinates).
• Eye crop features (via MobileNetv4).
• Screen content features (via lightweight CNNs).
• Reflection vectors (or heatmaps).
  The final Multi-Layer Perceptron (MLP) outputs the gaze coordinates in screen space.

3. Key Contributions

1. Screen Content Knowledge Integration: Demonstrated that knowing what is on the screen allows for robust segmentation of the corneal reflection, solving the "infinite variety" problem of screen content.
2. Novel Encoding: Introduced the Reflection Vector encoding, which mathematically translates the reflection's position relative to the pupil into a gaze target, significantly outperforming raw appearance models.
3. Privacy-Preserving Alternative: Showed that a Reflection Heatmap (grayscale correlation) alone can achieve baseline accuracy while obfuscating biometric iris details, offering a potential path for privacy-preserving cloud-based gaze tracking.
4. Camera Placement Optimization: Identified that upper eyelids/eyelashes occlude reflections during downward gaze. A supplemental study proved that placing the camera at the bottom of the device reduces this occlusion, further improving accuracy.

4. Results

The study involved 22 participants with a "leave-one-participant-out" cross-validation scheme (no per-user calibration).

• Baseline Performance: The standard "Eye Crops" model achieved a mean error of 2.00 cm.
• HiFiGaze Performance:
  • Best Model (Eye Crops + Reflection Vector): Achieved a mean error of 1.64 cm, representing an ~18% improvement over the baseline.
  • Alternative Model (Eye Crops + Screen Thumbnail): Achieved 1.77 cm error (~12% improvement) with minimal processing overhead.
• Dark Screen Robustness: Even on dark screen content (where reflections are faint), the best HiFiGaze model (1.73 cm error) outperformed the baseline on bright content (2.00 cm).
• Camera Location: In the supplemental study, moving the camera to the bottom of the phone improved the Reflection Vector model by an additional ~8% and the Screen Thumbnail model by ~18%, confirming that occlusion is a primary error source.
• Efficiency: The system runs at ~10 FPS (100ms latency) on an iPhone 14 Pro Max, with the inference step taking only 0.78 ms.

5. Significance and Implications

• Calibration-Free Precision: HiFiGaze achieves "app-icon-sized" accuracy (~1.64 cm) without requiring users to undergo tedious calibration routines, making gaze tracking viable for everyday consumer apps.
• Hardware Agnostic: It transforms standard 4K front-facing cameras into high-precision gaze trackers, eliminating the need for expensive IR illuminators or specialized headsets (like Apple Vision Pro or Tobii).
• Future Interfaces: This technology enables new interaction paradigms for smartphones and laptops, including accessibility tools, attention-aware interfaces, and hands-free control, leveraging hardware users already possess.
• Limitations & Future Work: The current system struggles with users wearing glasses (due to secondary reflections) and requires diverse data for different iris colors and lighting conditions. Future work aims to address these demographic and environmental gaps.

In summary, HiFiGaze represents a paradigm shift in consumer eye tracking by treating the screen not just as a display, but as a known light source that, when combined with high-resolution cameras, provides a geometrically robust signal for gaze estimation.