💻 computer science

WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Rendering

WildCap is a novel hybrid inverse rendering framework that enables high-quality facial albedo capture from unconstrained smartphone videos by combining a data-driven lighting conversion method with a model-based approach featuring a texel grid lighting model and diffusion priors to effectively disentangle complex lighting from material properties.

Original authors: Yuxuan Han, Xin Ming, Tianxiao Li, Zhuofan Shen, Qixuan Zhang, Lan Xu, Feng Xu

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxuan Han, Xin Ming, Tianxiao Li, Zhuofan Shen, Qixuan Zhang, Lan Xu, Feng Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a perfect, lifelike digital twin of a friend so they can star in a video game or a movie. To do this, you need to capture their "skin texture" (called albedo in computer graphics)—the pure color of their skin without any shadows, wrinkles caused by lighting, or shiny spots from sweat.

Usually, to get this perfect texture, you need a Hollywood-style studio with a giant ring of lights (a "Light Stage") or a very specific setup where you control every single beam of light. It's expensive, time-consuming, and requires a professional crew.

WildCap is a new method that says: "Why do we need a studio? Just use your smartphone and walk around your friend in the park, at a cafe, or in their living room."

Here is how it works, broken down into simple concepts:

1. The Problem: The "Shadow Baking" Mess

When you take a photo of someone in the real world, the lighting is messy. There are shadows from trees, harsh sunlight, or dim lamps.

Old AI methods try to guess the skin color by looking at the photo. But because they aren't perfect, they often get confused. They think a shadow on the cheek is actually a dark spot on the skin. This is called "shadow baking." It's like trying to paint a wall, but the painter accidentally paints the shadow of a tree onto the wall itself. You can't wash the shadow off because it's now part of the paint.

2. The Solution: A Two-Step "Hybrid" Team

WildCap uses a clever team of two different types of "detectives" to solve the puzzle.

Detective A: The "Fast AI" (Data-Driven)

First, the system uses a powerful AI (called SwitchLight) to look at the messy smartphone video.

What it does: It quickly guesses what the skin should look like and tries to remove the obvious lighting effects.
The Flaw: It's fast but not perfect. It leaves behind some "ghost shadows" (the baked artifacts). It's like a rough draft of a painting.

Detective B: The "Physics Expert" (Model-Based)

This is where WildCap gets clever. Instead of trying to fix the original messy photo, it takes the rough draft from Detective A and treats it like a new photo.

The Twist: The system realizes, "Hey, this rough draft has weird shadows that don't make sense physically."
The Fix: Instead of trying to delete those shadows, it invents a new kind of "lighting" to explain them.

3. The Secret Sauce: The "Pixel Grid" of Lights

This is the paper's biggest innovation.

Normal Lighting: Usually, computers assume light comes from one big source (like the sun) or a few big bulbs.
WildCap's Lighting: Imagine the face is a giant grid of tiny tiles (like a pixelated video game). WildCap puts a tiny, invisible "light bulb" under every single tile of the grid.
How it works: If there is a weird shadow baked into the AI's rough draft, WildCap says, "Okay, I will turn on a tiny, dark light bulb right under that specific tile to explain why it's dark."
The Result: By turning on these tiny, local "dark lights," the system can mathematically cancel out the fake shadows. What's left behind is the pure, clean skin color underneath.

4. The "Art Teacher" (Diffusion Prior)

There's a risk here: If you have a tiny light bulb under every tile, you could accidentally turn the whole face into a weird, noisy mess. How do we stop the computer from going crazy?

WildCap uses a Diffusion Prior, which is like an "Art Teacher" trained on thousands of perfect, high-quality scans of real human faces.
As the computer tries to clean up the face, the "Art Teacher" whispers, "Hey, that looks like a weird noise pattern, not a real pore. Make it look more like a real human face."
This guides the computer to find the most realistic solution, ensuring the final result looks like a real person, not a glitchy video game character.

5. The Final Product

After all this math magic:

You get a 4K resolution map of the person's skin that is completely free of shadows and lighting tricks.
You can export this to a video game engine.
You can now light your digital character with any light you want (a sunset, a neon sign, a candle) and it will look perfectly realistic, because the "shadow" isn't baked into the skin anymore.

Summary Analogy

Think of it like restoring an old, dirty painting:

Old Way: You need a special room with perfect lights to see the painting clearly.
WildCap Way:
- Step 1: Use a fast robot to wipe off the biggest smudges (the AI guess).
- Step 2: The robot leaves some weird smears. Instead of scrubbing harder, you use a special "magic eraser" that acts like a tiny flashlight under every square inch of the canvas.
- Step 3: You adjust these tiny flashlights to cancel out the smears, revealing the true colors underneath.
- Step 4: An expert art restorer checks your work to make sure you didn't accidentally erase the actual paint.

The Bottom Line: WildCap turns your humble smartphone into a portable, high-end "Light Stage," allowing anyone to capture movie-quality digital faces just by walking around in the wild.

1. Problem Statement

The paper addresses the challenge of reconstructing high-quality facial diffuse albedo maps from smartphone videos captured in uncontrolled, "in-the-wild" environments.

Current Limitations: Existing high-quality methods rely on controllable lighting (e.g., Light Stages or smartphone flashlights), which increases capture cost and limits usability.
The Gap: Purely data-driven methods (neural networks) are robust to wild lighting but often produce "baking artifacts" (e.g., shadows baked into the texture) because they cannot fully understand physical light transport. Conversely, purely model-based inverse rendering methods are unstable and ill-posed when faced with complex, high-frequency lighting and shadows in the wild.
Goal: To achieve production-level facial albedo capture using only a standard smartphone video without any assumptions about scene lighting.

2. Methodology: Hybrid Inverse Rendering Framework

The authors propose WildCap, a novel hybrid framework that combines data-driven prediction with model-based optimization to disentangle albedo from complex lighting.

A. Data Capture & Preprocessing

Input: A ~30-second smartphone video of a subject.
Processing: 300 frames are sampled, camera parameters are calibrated (COLMAP), and a detailed mesh is reconstructed (2DGS). A coarse normal map is derived from the geometry.
Skin Tone: The subject's skin tone is provided (manually or automatically) to resolve color ambiguity between albedo and lighting.

B. The Hybrid Pipeline

The core innovation is a two-stage process:

Data-Driven Stage (SwitchLight): The raw captured images are first processed by SwitchLight, a state-of-the-art data-driven method, to predict a diffuse albedo image for each view.
- Role: This removes most global lighting effects and provides a robust initialization.
- Problem: SwitchLight's predictions still contain "shadow-baking" artifacts (non-physical shadows baked into the texture) because the network cannot perfectly simulate light transport.
Model-Based Stage (Inverse Rendering): The predicted images are treated as "real" images captured under simpler conditions. A model-based optimization is performed in UV space to explain the remaining baking artifacts as local physical lighting effects, thereby recovering a clean albedo map.

C. Key Technical Components

1. Texel Grid Lighting Model (TGL)

Challenge: Standard physics-based lighting models (like global Spherical Harmonics) cannot explain non-physical baking artifacts in neural network predictions.
Solution: The authors propose a Texel Grid Lighting Model.
- It models lighting as a 2D grid in UV space, where each grid cell contains Spherical Harmonics (SH) parameters.
- A binary mask ( $M$ ) identifies regions with baking artifacts.
- For regions inside the mask, the lighting is modulated by the local grid parameters ( $\gamma_V$ ); outside the mask, it uses global SH lighting ( $\gamma_g$ ).
- Effect: This allows the system to interpret shadow-baking artifacts as a clean albedo illuminated by "local dark lights," effectively removing the artifacts.

2. Diffusion Prior for Optimization

Challenge: The increased expressiveness of the Texel Grid Lighting Model makes the optimization problem highly ill-posed (scale ambiguity between lighting and albedo).
Solution: The authors introduce a Patch-level Diffusion Prior.
- A diffusion model is trained on high-quality Light Stage scans to learn the distribution of valid facial reflectance patches.
- During optimization, Diffusion Posterior Sampling is used. The system jointly optimizes the lighting parameters and samples the albedo map from the diffusion prior.
- This steers the optimization toward valid, high-quality albedo maps that match the observations while adhering to the statistical distribution of real human skin textures.

3. Upsampling

The method solves for reflectance maps at 1K resolution (1024x1024) using the diffusion prior, then uses a Super-Resolution (SR) network to upsample the final result to 4K. This is significantly more efficient than directly sampling 4K from a diffusion model.

3. Key Contributions

WildCap Framework: A hybrid inverse rendering method that bridges the quality gap between controllable lighting setups and in-the-wild captures, enabling high-fidelity albedo reconstruction from smartphone videos.
Texel Grid Lighting Model: A novel, non-physical lighting representation that effectively models and removes network-induced baking artifacts by treating them as local lighting effects.
Joint Optimization Strategy: A scheme that combines a patch-level diffusion prior with model-based optimization to resolve scale ambiguity and ensure the reconstruction of physically valid, high-quality textures.
Performance: The method achieves 4K resolution results in ~8 minutes on a single GPU, significantly outperforming prior art in the wild.

4. Results & Evaluation

Qualitative Results: WildCap produces clean, shadow-free albedo maps with high-frequency details (e.g., moles, skin texture) that are preserved from the input. It successfully removes complex shadows and specularities that confuse other methods.
Quantitative Comparison:
- Compared to DeFace and FLARE (in-the-wild methods), WildCap achieves superior PSNR, SSIM, and LPIPS scores.
- Compared to DoRA (a state-of-the-art method requiring controlled lighting), WildCap achieves comparable quality but with significantly lower capture cost (smartphone video vs. co-located flash setup).
Ablation Studies:
- Removing the hybrid approach (using raw images directly) fails to disentangle complex lighting.
- Removing the Texel Grid (using global SH only) fails to remove baking artifacts.
- Removing the diffusion prior leads to severe artifacts due to optimization instability.

5. Significance

Democratization of Digital Humans: By removing the need for expensive Light Stages or controlled lighting setups, WildCap makes high-fidelity facial capture accessible to everyday users using standard smartphones.
Technical Advancement: It demonstrates a successful paradigm of combining data-driven robustness with model-based physical constraints, specifically addressing the "baking artifact" problem in neural rendering.
Application: The output (diffuse albedo, specular albedo, and detailed normal maps) is directly exportable to graphics engines for photo-realistic rendering and relighting in virtual environments.

6. Limitations

Dependency: The method relies on SwitchLight for preprocessing, which is a proprietary model with limited API access.
Shadow Detection: The automatic shadow detection method (using DiFaReli) is slow and can miss subtle ambient occlusion effects; manual masking is currently preferred for highest quality.
Sharp Shadows: In cases with extremely sharp shadow boundaries (e.g., direct noon sunlight), the continuous lighting representation may struggle to completely eliminate the boundary artifacts.