Technical Summary: EgoRelight

Problem Statement

The paper addresses the challenge of creating photorealistic, relightable, and full-body digital avatars solely from the constrained viewpoint of a head-mounted display (HMD). While Mixed Reality (MR) telepresence promises immersive experiences where virtual humans blend indistinguishably into real or virtual surroundings, existing solutions fail to holistically solve three core requirements simultaneously:

1. Accurate Motion and Geometry Capture: Faithfully capturing a user's full-body motion and clothing dynamics from a single HMD, which suffers from severe self-occlusion and limited field of view.
2. Relightable Appearance: Synthesizing realistic shading (both diffuse and specular) under novel, arbitrary lighting conditions, rather than relying on "baked-in" lighting.
3. Environment Lighting Estimation: Recovering High Dynamic Range (HDR) environment maps from the HMD to seamlessly integrate the avatar into the physical world.

Current approaches typically treat these as isolated problems, either focusing on motion capture without relighting capabilities or requiring complex studio setups (e.g., lightstages) for relighting that are not portable.

Methodology

EgoRelight is a holistic framework that operates in three main stages: Egocentric Perception, Relightable Avatar Modeling, and Test-Time Environment Map Capture.

1. Egocentric Perception and Geometry Recovery

To drive the avatar, the system extracts dense geometric signals from stereo down-facing cameras on the HMD:

• Pose Estimation: A modified version of the FRAME network is used to predict sparse 3D keypoints from stereo images and head pose. This is followed by an Inverse Kinematics (IK) solver that optimizes for temporally plausible skeleton motion, utilizing a PCA model to constrain hand gestures and ensure temporal consistency.
• Depth Estimation: A person-specific depth estimator (based on DepthAnythingV2) is fine-tuned to predict metric depth maps from the stereo input. These maps are unprojected into 3D point clouds.
• Depth-Conditioned Animation: Instead of test-time optimization, the system employs a feed-forward "AnimationNet." This network takes the skeleton motion and the unprojected depth point cloud as conditioning signals. It hierarchically predicts embedded graph deformation parameters and per-vertex offsets to deform a personalized parametric mesh template, ensuring precise recovery of the visible frontal body geometry.

2. Relightable Avatar Appearance Modeling

The avatar's appearance is modeled using a two-layer representation: a deformed mesh for geometry and a layer of 3D Gaussian primitives parameterized in the mesh's UV space for high-fidelity texture. The rendering pipeline separates diffuse and specular shading to improve learning efficiency and physical accuracy:

• Geometry Lifting: A GeoLiftingNet processes temporal mesh sequences to generate high-frequency normal maps and flat-lit albedo textures, resolving ambiguities between geometry and material.
• Diffuse and Specular Separation:
  • Diffuse: A DiffuseNet (CNN) predicts view-independent diffuse shading and Gaussian parameters (color, scale, rotation) based on the albedo, shadow maps, and normal maps.
  • Specular: A SpecularNet utilizes a cross-attention mechanism to handle the permutation-invariant nature of incoming light directions. It samples a subset of rays (top-k based on a Blinn-Phong importance score) to encode the 6D ray information (light direction, view direction, half-vector, and visibility). This allows the model to learn complex specular light transport without restrictive analytical BRDF priors.
• Training: The model is trained on data captured in a lightstage with 331 programmable lights, using photometric supervision against ground-truth HDR images.

3. Affordable HDR Environment Map Capture

To enable Mixed Reality telepresence, the system estimates the lighting of the user's current environment using the HMD's front-facing cameras:

• Scan and Inpainting: A 360-degree LDR panorama is reconstructed from 20 sampled frames and inpainted using a diffusion model to fill unobserved regions.
• Inverse Rendering Calibration: Since the LDR scan does not match the HDR color space of the training data, the system performs a test-time inverse rendering process. It renders the pre-trained relightable avatar into the HMD's down-facing view and optimizes color correction (gain) and gamma parameters to match the observed avatar appearance. This effectively aligns the LDR environment map to the HDR space required for realistic relighting.

Key Contributions

1. EgoRelight Framework: The first system to holistically support precise egocentric full-body motion capture, photorealistic virtual human relighting, and environment lighting recovery using only a single HMD.
2. Depth-Conditioned Mesh Tracking: A novel method leveraging stereo depth maps as conditioning signals for a feed-forward animation network, achieving high-fidelity geometry recovery without expensive test-time optimization.
3. Motion-Driven Relightable Avatar: An end-to-end learned appearance model that separates diffuse and specular shading, utilizing a specialized ray-sampling and cross-attention strategy to generalize to unseen illumination without relying on analytical BRDFs.
4. In-the-Wild HDR Capture: An affordable technique for capturing and color-calibrating HDR environment maps from LDR egocentric cameras, enabling seamless avatar integration into physical environments.

Experimental Results

The authors evaluated EgoRelight on four subjects using a custom lightstage dataset and in-the-wild testing sequences.

• Geometry Reconstruction: The depth-conditioned avatar outperformed state-of-the-art baselines (including optimization-based methods like EgoAvatar in terms of efficiency) in point-to-surface distance, particularly on the visible frontal surfaces.
• Relighting Quality: Quantitative metrics (PSNR, SSIM, LPIPS, FID) showed EgoRelight significantly outperforming competing relightable avatar methods (e.g., Relighting4D, MeshAvatar) and non-relightable baselines. Qualitative results demonstrated superior preservation of high-frequency details (wrinkles, hair) and accurate color tone matching under novel lighting.
• Ablation Studies: Removing components like the geometry lifting network, diffuse/specular separation, or the cross-attention mechanism resulted in noticeable degradation, such as loss of wrinkle details, failure to converge, or unrealistic metallic highlights.
• Illumination Estimation: The proposed HDR capture pipeline produced more natural shading and color consistency compared to image-based harmonization tools (e.g., IC-Light, Photoshop) which often failed to capture correct shading directions or destroyed facial details.

Significance and Claims

The paper claims that EgoRelight represents a significant step toward the vision of immersive Mixed Reality telepresence. By unifying performance capture, relighting, and environment estimation into a single HMD-driven pipeline, it overcomes the limitations of studio-bound systems and the lack of photorealism in existing egocentric avatars. The authors position this work as a foundational step for next-generation video conferencing and social telepresence, enabling users to seamlessly insert their digital twins into any physical or virtual environment with coherent, physically plausible lighting. The work acknowledges limitations regarding real-time performance (currently ~200ms per frame excluding IK) and the reliance on person-specific lightstage data for training, suggesting future work in universal priors and real-time optimization.