Technical Summary: GazePrior

Problem Statement

Eye tracking (ET) is a foundational technology for advanced Augmented Reality (AR) and Virtual Reality (VR) applications, enabling features like foveated rendering and gaze-based interaction. However, training deep learning models for ET is hindered by the high cost and time required to collect real-world data with high-quality ground truth for every new device generation. As hardware configurations (camera placement, resolution) evolve, new training sets comprising millions of frames are necessary. Existing synthetic data generation methods often lack realism, while crowdsourced real data lacks accuracy. Furthermore, current semi-synthetic approaches that attempt to synthesize eye images often rely on explicit eyeball masks and rigid rotations, ignoring the intricate coupling between the eyeball, eyelids, and surrounding facial features, which degrades physical plausibility and accuracy.

Methodology

The authors propose GazePrior, a zero-shot approach that leverages a learned 3D gaze prior to synthesize high-fidelity training data for new ET devices using only sparse input data from existing devices. The method consists of two primary stages:

1. Learning a Disentangled 3D Gaze Prior (Pretraining)

Inspired by 3D avatar methods, the authors train a conditional Neural Radiance Field (NeRF) that models the distribution of human eyes across diverse identities, gaze directions, and lighting conditions.

• Conditional Radiance Field: The network $\Psi$ takes a 3D point $x$, view direction $v$, and latent embeddings for subject identity ($\hat{s}$), gaze direction ($\hat{g}$), and lighting ($\hat{l}$). It outputs density $\sigma$ and color $c$.
• Disentanglement: The architecture explicitly disentangles these factors. The density network depends on identity and gaze but is independent of lighting ($\hat{l}$), while the color network depends on all three. This ensures geometric consistency is not confounded by illumination changes.
• Latent Representations:
  • Subject ($\hat{s}$): A learnable codebook vector for each identity, encouraging a shared, low-frequency representation that acts as a regularizer against artifacts (floaters).
  • Gaze ($\hat{g}$): Unlike rigid eyeball rotation models, gaze is encoded via sinusoidal encoding of the ground-truth gaze vector. This allows the network to learn the non-rigid movements of eyelids and eyebrows coupled with gaze changes.
  • Lighting ($\hat{l}$): A learnable codebook vector that affects only the color output, ensuring the reconstructed geometry remains invariant to lighting.
• Training: The model is pretrained on a large dataset (Lin et al. [11]) containing over 400 subjects with multi-view images, varying gazes, and lighting, using a loss function combining RGB reconstruction, perceptual loss, and KL divergence to regularize the latent space.

2. Sparse-Input Retargeting (Finetuning)

To adapt the prior to a new target device (e.g., a headset with a different camera configuration), the method performs a two-step optimization on sparse source images (e.g., 4 views) from an existing headset:

1. Latent Optimization: With the pretrained network weights frozen, the system optimizes the specific latent codes ($\hat{s}_I, \hat{l}_I$) for the new subject and lighting to fit the sparse source views. The ground-truth gaze embedding $\hat{g}_I$ is computed from the known annotations and kept fixed.
2. Finetuning: The network weights are unfrozen and jointly optimized with the latent codes to capture person-specific details and refine the reconstruction.
3. Synthesis: The resulting high-fidelity radiance field is rendered from the perspective of the target device's cameras. This generates synthetic training images that maintain the original ground-truth gaze annotations but match the new device's optical characteristics.

Key Contributions

• First Learned Gaze Prior: The paper introduces the first 3D gaze prior that disentangles subject, gaze, and light, featuring biologically accurate eyelid movements without relying on explicit eyeball masks or rigid rotations.
• Sparse-View 3D Reconstruction: The method enables high-quality novel view synthesis from extremely sparse inputs (as few as 2-4 views), a significant improvement over previous methods requiring dense multi-view captures.
• Zero-Shot Data Synthesis: It provides a pipeline to generate training data for new AR/VR devices without collecting any new real-world data, effectively bridging the domain gap between legacy and new hardware.
• Superior Performance: The synthesized data allows ET models to achieve higher accuracy and robustness compared to models trained on data from competing synthesis baselines.

Experimental Results

The authors evaluated GazePrior using the Aria headset as the target device and compared it against state-of-the-art baselines (ControllableGaze [10], Lin et al. [11]).

• Novel View Synthesis (NVS): In quantitative evaluations (PSNR, SSIM, MSE), GazePrior significantly outperformed baselines as the number of input views decreased. With only 2 input views, GazePrior achieved a performance gap of nearly 10 dB over the best baseline. Qualitative results showed GazePrior produced high-fidelity details (eyelashes, eyelids) with minimal domain gap compared to real Aria captures, whereas baselines exhibited floaters and artifacts.
• Eye Tracking Performance: An ET model trained on GazePrior-synthesized data achieved a mean gaze angle error of 5.85° on real Aria test data. This is a substantial improvement over baselines (12.53° and 12.69°) and approaches the performance of a model trained on real, device-specific data (4.35°).
• Ablation Studies: The study confirmed that disentangled latent spaces and the separation of lighting from density are critical. Using a single latent vector or making density light-dependent resulted in significant performance drops. Sinusoidal gaze encoding proved more effective than learned codebooks for gaze.

Significance and Claims

The paper claims that GazePrior addresses the prohibitive costs of data collection for new AR/VR devices by enabling a zero-shot approach to eye tracking. By leveraging existing annotated datasets from older headsets and retargeting them via a learned 3D prior, the method allows developers to:

1. Train robust eye trackers for new devices before they are physically built.
2. Evaluate how camera placement impacts performance without physical prototyping.
3. Achieve high accuracy without the need for expensive, large-scale data collection campaigns.

The authors position this work as a foundational step toward unlocking advanced features like foveated rendering and gaze-pinch interactions in XR, removing the data-collection bottleneck that currently hinders rapid iteration in the field. They acknowledge limitations, such as the difficulty of simulating high-frequency corneal glints with radiance fields, but emphasize the method's success in generating diverse, realistic training data that outperforms existing synthesis techniques.