💻 computer science

Low Latency Gaze Tracking via Latent Optical Sensing

This paper presents a real-time, ultra-low-latency gaze tracking system that bypasses traditional image processing by using a passive optical encoder with a microlens array and binary mask to directly capture task-relevant latent features, achieving a 3.4 ms end-to-end latency and improved energy efficiency compared to conventional camera-based approaches.

Original authors: Yidan Zheng, Matheus Souza, Kaizhang Kang, Qiang Fu, Hadi Amata, Wolfgang Heidrich

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yidan Zheng, Matheus Souza, Kaizhang Kang, Qiang Fu, Hadi Amata, Wolfgang Heidrich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess where someone is looking. The traditional way to do this is like taking a high-definition, 4K photograph of their eye, sending that massive file to a supercomputer, and asking the computer to analyze every single pixel to figure out the direction of their gaze. This is accurate, but it's slow, uses a lot of battery, and requires a lot of data to move around.

This paper proposes a completely different, "shortcut" method. Instead of taking a full picture, they built a special optical filter that acts like a smart sieve.

Here is how their system works, broken down into simple concepts:

1. The "Smart Sieve" (The Optical Encoder)

Imagine you have a bucket of water (the light coming from the eye) and you want to know if it's raining heavily or lightly. Instead of catching every single drop in a giant net and counting them one by one (taking a full photo), you pour the water through a sieve with a very specific, custom-made pattern of holes.

The Hardware: The researchers built a device with a microlens array (a sheet of tiny magnifying glasses) and a binary mask (a sheet with a specific pattern of black and silver squares).
The Magic: When light from the eye passes through this "sieve," it gets scrambled into a specific pattern. The system doesn't care what the eye looks like (no iris color, no skin tone, no detailed features). It only cares about the pattern of light intensity that gets through.
The Result: Instead of a 256x256 pixel image (65,000+ data points), the system only captures 16 numbers. It's like compressing a whole novel into a single sentence that still tells you the main plot.

2. The "Lightweight Translator" (The AI Decoder)

Once the system has those 16 numbers, it doesn't try to reconstruct the eye image. That would be a waste of time. Instead, it feeds those 16 numbers into a tiny, super-fast computer brain (a lightweight neural network).

Training Trick: To teach this tiny brain, the researchers used a "teacher" AI that knows how to turn eye images into abstract codes. They taught the new system to mimic the teacher's "thought process" using only the 16 numbers.
Privacy Bonus: Because the system never actually sees or stores a real picture of the eye, it's naturally more private. You can't identify the person from the 16 numbers, but the system can still tell you exactly where they are looking.

3. The Speed Record (Latency)

The biggest win here is speed.

Traditional Cameras: Have to wait for the whole image to be taken, then read all the pixels, then process them. This usually takes 10 to 20 milliseconds (or more).
This System: Because it skips the "taking a photo" step and only reads 16 tiny sensors, the entire process—from the light hitting the sensor to the computer saying "they are looking left"—takes less than 3.4 milliseconds.

The Analogy:
Think of a traditional camera system like a photographer who takes a photo, prints it, walks it over to an art critic, and asks, "Where is the person looking?"
This new system is like a security guard who doesn't look at the person's face at all. Instead, they just look at the shadow the person casts on a specific wall. The shadow is distorted and unrecognizable as a face, but the guard knows exactly which way the person is facing based on the shadow's shape. It's instant, requires no photos, and uses very little energy.

What They Actually Achieved

Accuracy: In tests, the system guessed the gaze direction with an error of about 6 degrees (roughly the width of a thumb held at arm's length). This is good enough for many real-world uses.
Real-World Test: They built a physical prototype with real lenses, masks, and sensors. When they tested it on real people, they only needed to show the person 12 to 15 different spots on a screen to "calibrate" the system for that specific person, and it worked well.
Efficiency: The system uses a tiny fraction of the computing power required by standard cameras. It's so efficient it could run on very small, battery-powered devices like future AR glasses.

In Summary:
The paper demonstrates that you don't need a high-definition camera to track eye movement. By using a clever optical filter to scramble light into a tiny set of numbers, and a simple AI to read those numbers, you can track where someone is looking in under 4 milliseconds. This is fast enough to keep up with the human brain, making it perfect for next-generation virtual reality and augmented reality glasses where lag causes motion sickness.

Problem Statement

Conventional gaze tracking systems rely on high-resolution cameras to capture full eye images, followed by deep neural network inference. While accurate, this approach introduces significant sensing, bandwidth, and computational overhead due to the need to read out and process millions of pixels. This latency often exceeds the 5 ms threshold required for next-generation wearable AR/VR displays to support real-time interaction and foveated rendering without disrupting user immersion. Existing lightweight or event-based solutions often still require temporal aggregation or image-like reconstruction, failing to eliminate the fundamental bottleneck of full-frame image acquisition and processing.

Methodology

The authors propose a fully passive optical system that bypasses explicit image formation entirely. Instead of capturing a dense image, the system performs task-oriented optical encoding to directly acquire a compact, low-dimensional latent representation of the eye.

1. Optical Encoder and Hardware Architecture
The core of the system is a "latent optical encoder" consisting of three main components:

Microlens Array (MLA): A Fresnel lens array that demagnifies the eye image.
Binary Chromium Mask: A co-designed binary mask placed at the focal plane of the MLA.
Phototransistor Array: A 4×4 array (16 elements) that captures the spatially multiplexed optical measurements.
Light from the eye passes through the MLA and is modulated by the binary mask. The resulting encoded intensity is captured directly by the phototransistors, producing a 16-dimensional measurement vector. This hardware is integrated with an FPGA for rapid signal acquisition.

2. End-to-End Training Framework
The system is trained using a simulation-to-real pipeline that jointly optimizes the optical encoding masks and the digital inference network:

Digital Emulation: A differentiable digital encoder $\tilde{E}$ simulates the optical process using 16 learnable binary masks ( $256 \times 256$ resolution) applied to input eye images.
Latent Distillation: To ensure the 16 measurements contain sufficient information, the system employs a "teacher-student" distillation approach. A teacher network ( $P_T$ ) inverts a pretrained StyleGAN2 generator to map eye images to a 256-dimensional latent space ( $h_T$ ) and predicts gaze. A student decoder ( $P$ ) is trained to map the compressed 16-measurement vector directly to this same latent space ( $h$ ), as well as to predict gaze direction ( $\hat{g}$ ) and a validity flag ( $\hat{v}$ ).
Loss Function: The training objective combines supervised gaze regression, validity classification, gaze distillation (matching the teacher's gaze prediction), and feature distillation (matching the teacher's latent representation).
Data Augmentation: To address identity bias and limited dataset diversity, the authors use generative inpainting (Flux diffusion model with SAM3 and ControlNet) to vary periocular appearance (skin tone, lighting) while preserving the ground-truth gaze label.

3. Gaze Steering
The framework includes a control network that can steer gaze in the latent space. By predicting a residual displacement ( $\Delta h$ or $\Delta w^+$ ) based on source and target gaze vectors, the system can edit the latent representation to change gaze direction while preserving the subject's identity, demonstrating that the learned features effectively decouple gaze from appearance.

Key Contributions

End-to-End Latent-Space Framework: A novel framework that jointly optimizes passive optical encoding and neural decoding, enabling direct gaze inference from 16 scalar measurements without reconstructing full eye images.
Compact Hardware Prototype: A single-shot optical sensing system utilizing an MLA, optimized binary masks, and a 4×4 phototransistor array with FPGA acquisition. This design achieves an end-to-end latency of 3.4 ms (including sensing and computation).
Simulation-to-Real Pipeline: A two-stage evaluation strategy achieving a cross-subject simulation error of 6.47° and a calibrated per-subject real-world error below 6° using only 12–15 calibration points.

Results

Accuracy vs. Efficiency: On the ETH-XGaze dataset, the proposed method achieves an angular error of 6.47° using only 16 measurements and a lightweight MLP (0.244M parameters, 0.49M FLOPs).
- Compared to a 16×16 ResNet18 baseline, it reduces sensing dimensionality by 16×, parameters by ~48×, and computation by ~573×.
- Compared to a 256×256 ResNet18 baseline, it reduces sensing dimensionality by 4096× and computation by ~9300×.
- It outperforms compact backbones like MobileNetV3 and EfficientNet-B0 in terms of parameter and FLOP efficiency while maintaining competitive accuracy.
Latency: The system demonstrates a total end-to-end latency of 3.4 ms on a CPU (3.19 ms for optical sensing/acquisition + 0.22 ms for inference). This significantly outperforms conventional image-based mobile edge systems (typically tens of milliseconds) and approaches the performance of specialized high-speed commercial trackers (e.g., EyeLink 1000 Plus).
Real-World Performance: With subject-specific fine-tuning using 12 calibration points, the system achieves a mean angular error of 5.96°, with some subjects reaching 4.5°.
Privacy: The system preserves privacy by capturing only task-relevant latent features rather than biometric eye images, as demonstrated by the inability to reconstruct the subject's identity from the measurements.

Significance and Claims

The paper claims that shifting feature extraction into the optical domain via task-driven design offers a viable path to ultra-low-latency, energy-efficient gaze tracking. By eliminating the need for high-bandwidth image readout and heavy computational processing, the system meets the strict <5 ms latency requirements for immersive XR applications. The authors emphasize that while the current prototype is a proof-of-concept, the drastic reduction in data volume (from megapixels to 16 scalars) suggests a clear pathway for highly energy-efficient implementations in future battery-operated wearable devices. The work highlights the potential of "sensing-as-inference" to replace conventional image formation for specific computer vision tasks.