Learning a Delighting Prior for Facial Appearance Capture in the Wild

Problem Statement

High-quality facial appearance capture, essential for creating photorealistic digital humans, has traditionally relied on costly, controlled studio environments (e.g., Light Stages). While recent efforts have attempted to capture faces "in the wild" using unconstrained smartphone videos, these approaches face significant challenges. Specifically, model-based inverse rendering paradigms struggle to disentangle complex reflectance (albedo) from unknown, real-world illumination. This often results in unstable optimization and "baked-in" shadows or lighting artifacts in the reconstructed textures. Existing data-driven solutions either remain proprietary (limiting reproducibility) or suffer from spatial blurring and physically incorrect diffuse albedo when trained on available open-source datasets like FaceOLAT. Furthermore, hybrid approaches like WildCap, which combine data-driven delighting with inverse rendering, require manual intervention to correct artifacts, hindering scalability.

Methodology

The authors propose OpenDelight, a fully automatic system that shifts the focus of facial appearance capture toward learning a robust delighting prior. The methodology consists of three main components:

1. Dataset Latent Modulation (DLM) for Heterogeneous Training
To train a powerful delighting network, the authors combine two complementary data sources:

• OLAT Dataset (FaceOLAT): Provides real-world image distributions but suffers from spatial blurring and specular baking artifacts due to imperfect alignment and lack of polarization.
• Rendered Light Stage Scan Dataset: A custom dataset of 120 subjects captured in a Light Stage (material mode) and rendered. It offers sharp details and physically correct diffuse albedo but has a larger domain gap from real-world images.

To seamlessly integrate these sources without the network settling for a compromised solution, the authors introduce Dataset Latent Modulation (DLM).

• Mechanism: The core network (a ViT-based encoder with a convolutional decoder) is conditioned on learnable source-aware tokens.
• Disentanglement: During end-to-end training, these tokens absorb dataset-specific styles (e.g., color distribution, sharpness), while the core network distills the universal principles of facial delighting.
• Inference: By conditioning the network on tokens associated with the high-quality rendered scan dataset, the model produces sharp, physically correct predictions while retaining the generalization ability learned from the real-world OLAT data.

2. Detail Enhancement Network
Recognizing that the ViT-based architecture lacks skip connections and may lose fine-grained details, the authors employ a secondary UNet-based detail enhancement network. This network takes the original image and the predicted diffuse albedo as input to recover fine details, trained specifically on the rendered dataset with stochastic degradations to simulate noise.

3. Prior-Grounded Reconstruction Pipeline
The delighting prior is integrated into a facial appearance capture pipeline:

• Input: A casual smartphone video sequence.
• Process: OpenDelight estimates the diffuse albedo for each view. These clean albedos are fused into a unified texture map.
• Optimization: Unlike previous methods requiring complex, region-specific inverse rendering to fix artifacts, the authors find their texture maps contain negligible baking artifacts. Consequently, the optimization is simplified to estimating global Spherical Harmonics (SH) lighting and sampling a patch-level diffusion prior.
• Output: High-resolution (4K) relightable facial assets (base mesh, diffuse, specular, and normal maps).

Key Contributions

• OpenDelight: A powerful, open-source delighting prior that outperforms leading proprietary models (e.g., SwitchLight) in removing complex lighting and shadows from single-view in-the-wild face images.
• Dataset Latent Modulation (DLM): A novel training technique that enables effective joint training on heterogeneous data sources (synthetic scans and real OLAT data) by disentangling dataset-specific styles from core physical principles using source-aware tokens.
• NeRSemble-Scan Dataset: The authors transformed the existing multi-view NeRSemble dataset into a large-scale collection of 4K-resolution, relightable Light Stage scans using their capture method.
• Fully Automatic Pipeline: A system that eliminates the need for manual artifact correction or specialized region-based inverse rendering, achieving high-quality results comparable to methods requiring human intervention.

Results

• Delighting Performance: On the FaceOLAT and 3DRFE test sets, OpenDelight achieves superior quantitative metrics (PSNR, SSIM, LPIPS) compared to SwitchLight, IC-Light, and DreamLight. Qualitative results show significantly cleaner diffuse albedo with fewer baking artifacts, particularly in hard shadow regions.
• Ablation Studies:
  • Training solely on FaceOLAT results in blurred, physically incorrect albedos.
  • Training solely on rendered scans leads to domain gaps (wrong skin tones).
  • Removing DLM causes the network to unpredictably switch between dataset distributions at inference, leading to view inconsistency.
  • The detail enhancement network is crucial for restoring fine-grained facial features lost by the ViT encoder.
• Appearance Capture: The method produces 4K relightable assets that, when re-rendered under novel lighting, closely match real-captured images (average LPIPS of 0.0576 against real images). It outperforms model-based methods (FLARE, DeFace) and matches or exceeds the quality of the concurrent WildCap method, despite being fully automatic.

Significance and Claims

The paper claims that learning a robust delighting prior fundamentally simplifies the facial appearance capture problem. By internalizing accurate light transport into the prior, the authors demonstrate that complex, intervention-heavy inverse rendering pipelines can be replaced by a simpler, automatic optimization process.

The authors emphasize the democratization of high-end facial capture:

1. Reproducibility: By open-sourcing the model, code, and the NeRSemble-Scan dataset, they provide the research community with the tools to build upon high-quality facial capture without relying on proprietary black boxes.
2. Quality Gap Reduction: The system significantly narrows the quality gap between unconstrained smartphone recordings and controlled studio captures.
3. Foundation for Future Research: The release of NeRSemble-Scan provides a new foundation for building photorealistic digital humans and advancing research in portrait delighting and relighting.

The authors acknowledge limitations, including potential identity loss regarding colored nevi (which may be mistaken for lighting) and a bias toward lighter skin tones due to dataset imbalance, suggesting these as areas for future improvement.