Technical Summary: DiFaReli++: Diffusion Face Relighting with Consistent Cast Shadows

Problem Statement

Single-view face relighting in the wild is a challenging task requiring the adjustment of facial shading to new lighting conditions while accounting for complex physical phenomena such as global illumination, subsurface scattering, and self-occlusion (cast shadows). Existing state-of-the-art methods typically rely on intrinsic image decomposition, estimating 3D shape, albedo, and lighting parameters to re-render the image. However, this approach is error-prone in real-world scenarios where accurate geometry estimation (especially for hair, ears, and occluded areas) is difficult. Furthermore, these methods often struggle to realistically remove or add strong highlights and cast shadows, leading to residual artifacts and physically implausible results. Additionally, many high-performing approaches require extensive training data, such as multi-view images, light stage data, or lighting ground truth, which are not readily available for in-the-wild applications.

Methodology

The authors propose DiFaReli++, a framework that bypasses the need for accurate intrinsic decomposition and can be trained solely on 2D images without 3D scans, relit pairs, or lighting ground truth. The core of the method leverages a Conditional Denoising Diffusion Implicit Model (DDIM).

1. Feature Encoding and Disentanglement

Instead of decomposing the image into physical components, the method encodes the input image into a feature vector that disentangles lighting from other attributes. This vector includes:

• Lighting: Spherical Harmonic (SH) coefficients estimated via the off-the-shelf estimator DECA.
• Shape & Camera: 3D face shape parameters (FLAME model) and camera pose, also from DECA.
• Identity: Deep feature embeddings from ArcFace.
• Shadow & Background: A scalar for shadow intensity (in the base version) or a shadow map (in DiFaReli++), and segmentation masks or background images.

2. Novel Conditioning Technique

A key innovation is the conditioning mechanism for the DDIM, which separates inputs into "spatial" and "non-spatial" conditions:

• Spatial Conditioning: The target lighting and shape parameters are used to render a shading reference (a gray-shaded face mesh). This reference, along with a shadow map and segmentation masks, is fed into a Modulator network. The Modulator computes layer-wise spatial modulation weights that are multiplied with the DDIM's residual block outputs. This allows the network to learn complex light-geometry interactions implicitly while preserving spatial structure.
• Non-Spatial Conditioning: Identity, shape, and camera parameters are processed via Multi-Layer Perceptrons (MLPs) and injected into the DDIM using Adaptive Group Normalization (AdaGN) to preserve identity and structural consistency.

3. Consistent Cast Shadows (DiFaReli++ Specifics)

To address the limitation of scalar shadow intensity control, DiFaReli++ introduces a shadow map conditioning strategy:

• Training: A shadow map is inferred by using a pre-trained DiFaReli model to generate "strong" and "weak" shadow versions of an input image. The pixel difference between these versions is thresholded to create a binary shadow mask, which is then scaled by the estimated shadow intensity.
• Inference: At test time, a ray-traced shadow map is generated from the estimated depth and target lighting. This map guides the DDIM to synthesize new, physically plausible cast shadows that align with the target lighting direction, ensuring temporal consistency in dynamic lighting scenarios.

4. Expanded Relightable Area

The framework expands the relightable region beyond the face by replacing raw background pixel conditioning with segmentation masks (e.g., clothing, hats, neck). This allows the network to relight non-facial parts while preserving their structural integrity, rather than treating them as static background.

5. Single-Shot Distillation

To overcome the slow inference speed of diffusion models (which require multiple denoising steps), the authors propose a single-shot relighting framework. A supervised network with the same architecture as DiFaReli++ is trained on synthetic data generated by the teacher model (DiFaReli++) using a simple L2 loss. This distilled network performs relighting in a single pass, achieving a 1,000× speedup while surprisingly outperforming the teacher model in visual metrics.

Key Contributions

1. A State-of-the-Art Framework: A conditional DDIM-based face relighting system that produces photorealistic results without requiring accurate intrinsic decomposition or 3D/lighting ground truth.
2. Novel Conditioning Technique: A method that converts a rendered shading reference into spatial modulation weights for the DDIM, effectively modeling complex light-geometry interactions.
3. Consistent Cast Shadows: An extension (DiFaReli++) that utilizes inferred and ray-traced shadow maps to generate temporally consistent and physically plausible cast shadows under dynamic lighting.
4. Expanded Scope: The ability to relight non-facial parts (clothing, hats) using segmentation masks, moving beyond the face-only limitation of previous works.
5. Single-Shot Efficiency: A distilled single-shot network that achieves superior performance metrics and a massive speedup compared to the diffusion-based teacher model.

Results

The method was evaluated on the Multi-PIE benchmark and the FFHQ dataset, comparing against recent state-of-the-art methods (e.g., Total Relighting, DiffusionRig, IC-Light, HoloRelighting).

• Quantitative Performance: DiFaReli++ achieved the best scores across all metrics (DSSIM, MSE, LPIPS) on the Multi-PIE dataset for both same-subject and cross-subject target lighting scenarios.
• User Studies: In three user studies (relighting quality on facial/non-facial parts, moving lights, and rotating HDR maps), the proposed method was preferred over all baselines in the majority of comparisons (e.g., 70-72% preference rate).
• Qualitative Improvements: The method effectively removes existing hard shadows and synthesizes new ones that align with target lighting. It preserves high-frequency details (hair, skin texture) and handles accessories and clothing better than competitors.
• Speed: The single-shot distilled model runs in approximately 0.07 seconds per image (given pre-processed data), significantly outperforming diffusion acceleration techniques (DPM-Solver, UniPC) which require multiple steps and take over 1 second.

Significance and Claims

The paper claims that DiFaReli++ represents a significant advancement by shifting the paradigm from explicit physical decomposition to implicit learning via diffusion models. By leveraging a novel conditioning technique and shadow map inference, the method achieves photorealistic relighting with consistent cast shadows in a single network pass.

The authors emphasize that their approach is robust in "in-the-wild" scenarios where geometry estimation is unreliable. They note that the single-shot distilled model not only accelerates inference but also surpasses the performance of the diffusion-based teacher model, suggesting that the distillation process allows the student network to learn the relighting task more directly and effectively. The work opens the possibility for extending relighting to full human bodies and objects, where intrinsic decomposition is even more challenging.

The authors acknowledge limitations, such as the potential for generated shadows to lack perfect physical accuracy and the dependency on the quality of the light estimator (DECA) for target lighting, which can struggle with extreme or unusual lighting conditions (e.g., strong orange or green tints) not well-represented in the training data. However, they assert that the framework is flexible and can be adapted to new lighting models or estimators.