💻 computer science

High-Quality Facial Geometry and Appearance Capture at Home

This paper presents a novel, easy-to-use method for capturing high-quality, complete 3D facial geometry and appearance—including skin, hair, eyes, and mouth interior—from a single smartphone flashlight sequence in dim lighting, utilizing a hybrid representation and advanced lighting models to achieve relightable results suitable for daily home use.

Original authors: Yuxuan Han, Junfeng Lyu, Feng Xu

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuxuan Han, Junfeng Lyu, Feng Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a digital twin of yourself—a 3D character that looks exactly like you, down to the pores on your skin and the sparkle in your eyes. Usually, to do this, you'd need to visit a high-tech movie studio with a room full of expensive cameras and lights, costing thousands of dollars.

This paper introduces a new method that lets you do this right in your living room, using just your smartphone and a dimly lit room.

Here is the breakdown of how they did it, using some simple analogies:

1. The Setup: The "Flashlight Detective"

Instead of a fancy studio, you just need a dark room (like at night with the curtains drawn) and your phone.

The Trick: You turn on your phone's flashlight and hold it right next to the camera lens.
The Action: You walk in a circle around the person (or yourself) for about 25 seconds while recording a video.
The Analogy: Think of this like a detective shining a single, bright flashlight on a suspect in a dark room. By moving around, the detective sees how the light bounces off the suspect's face from every angle. The computer uses these shadows and highlights to figure out the 3D shape and the texture of the skin.

2. The Big Problem: The "Glass Marble" Eyes

The hardest part of scanning a face is the eyes.

The Issue: Skin is like a matte piece of paper; it scatters light softly. But eyes are like glass marbles or shiny marbles; they reflect light sharply and clearly. If you try to scan a shiny marble with a standard 3D scanner, it often looks like a blurry blob or a flat circle.
The Solution: The authors realized that trying to "guess" the shape of the eye from a video is too hard. So, they built a hybrid system.
- For the Skin, Hair, and Mouth: They use a super-smart, flexible digital clay (called a "Neural Field") that can mold itself to fit any shape perfectly.
- For the Eyes: They don't guess. They place two perfect, pre-made digital spheres (like glass marbles) where the eyes should be. They tell the computer, "These are the eyes; just figure out the color of the iris, but keep the shape round and shiny."
The Result: This prevents the "blobby eye" problem and gives you realistic, sparkling eyes that reflect light correctly.

3. The Lighting Puzzle: Untangling the "Soup"

When you take a photo in a dim room with a flashlight, the light hitting your face is a mix of two things:

The Flashlight: A bright, sharp beam (High Frequency).
The Room Glow: The faint light bouncing off the walls (Low Frequency).

It's like trying to taste the difference between salt and sugar when they are both dissolved in a bowl of soup.

The Innovation: The researchers created a special mathematical "recipe" (a Combined Lighting Model) that acts like a sieve. It separates the "salt" (the sharp flashlight reflection) from the "sugar" (the soft room glow). This allows them to figure out exactly how your skin reflects light, which is crucial for making the 3D model look real when you move it to a new environment.

4. The "Digital Passport"

Once the computer finishes its work, it doesn't just give you a weird, unrecognizable 3D blob. It exports the result as a standard 3D file (like a mesh with texture maps).

Why this matters: You can take this file and plug it directly into common software like Blender (used by animators and game developers).
The Magic: Because they separated the "diffuse" (matte skin color) from the "specular" (shiny highlights), you can now take your digital twin and put it in a sunny park, a dark cave, or a neon-lit city, and it will react to the new light realistically. It's like giving your digital self a universal passport that works in any lighting condition.

Summary

In short, this paper says: "You don't need a million-dollar studio to get a Hollywood-quality 3D scan of your face."

By combining a simple smartphone video with a clever mix of "pre-made eye shapes" and "smart math to separate light," anyone can create a high-quality, relightable digital version of themselves that looks great in any video game or movie.

1. Problem Statement

High-quality 3D facial geometry and appearance capture (including skin, hair, mouth interior, and eyes) is traditionally restricted to expensive studio environments using specialized equipment like Light Stages or polarized spherical gradient illumination. While recent efforts have attempted to democratize this process, existing home-capture methods suffer from significant limitations:

Inconvenience: They often require specific conditions like direct sunlight (SunStage) or darkrooms with polarization filters (PolFace).
Incomplete Modeling: Most methods focus only on facial skin, failing to reconstruct complex regions like the highly reflective eyes, hair, or mouth interior.
Reflectance Disentanglement: Separating diffuse albedo from specular highlights is difficult with a single light source, especially for highly reflective surfaces like eyeballs.

The authors aim to develop a low-cost, easy-to-use system that captures a complete, relightable 3D face asset using only a standard smartphone in a dimly lit room.

2. Methodology

The proposed method reconstructs a 3D face from a single co-located smartphone flashlight sequence (video) captured in a dim room. The core technical components are:

A. Hybrid Face Representation

To address the challenge of modeling different facial regions with varying reflectance properties, the authors propose a hybrid representation splitting the face ( $F$ ) into two regions:

Eyeballs ( $E$ ): Modeled as two sphere meshes with predefined specular reflectance (albedo and roughness). The spatially varying diffuse albedo is learned from data. This leverages the strong prior that eyeballs are spherical and highly specular, bypassing the difficulty of reconstructing high-frequency specular geometry purely from images.
Other Regions ( $S$ ): Includes skin, mouth interior, and hair. These are modeled using Neural Fields:
- Geometry: A Neural Signed Distance Field (SDF).
- Reflectance: A Neural field modeling the parameters of the Disney BRDF (diffuse albedo, specular albedo, roughness).

B. Mesh-Aware Volume Rendering

To train this hybrid representation, a novel mesh-aware volume rendering technique is introduced.

The rendering process seamlessly integrates the explicit sphere meshes (eyeballs) into the volumetric rendering of the neural SDF.
A differentiable select operator determines whether a sampled point belongs to the eyeball region or the skin region based on the minimum SDF value, allowing the system to apply the correct geometry and reflectance model (mesh-based vs. neural) for shading calculations.

C. Combined Lighting Model

The lighting environment is modeled as a combination of:

High-Frequency Source: The smartphone flashlight, treated as a point light co-located with the camera.
Low-Frequency Ambient: Dim room lighting, represented compactly using 2nd-order Spherical Harmonics (SH) applied to the diffuse term.
This compact representation allows the system to disentangle ambient light from the flashlight using only a single "flashlight-on" sequence, unlike previous methods requiring "flashlight-off" sequences.

D. Training Strategy & Priors

To regularize the optimization and prevent artifacts (e.g., eyeballs merging with skin or incorrect albedo), the authors employ specific loss functions:

Composition Loss ( $L_{comp}$ ): Uses a face parsing network to generate masks for eyes and skin, enforcing the hybrid components to occupy their respective regions.
Reflectance Regularization ( $L_{ref}$ ): Utilizes AlbedoMM (a 3D morphable face albedo model trained on Light Stage data) as a prior. It fits the specular albedo to the AlbedoMM model to guide the neural field, ensuring plausible diffuse/specular separation.
Two-Stage Training:
1. Geometry & Reflectance Joint Optimization: Optimizes SDF, BRDF, and lighting coefficients.
2. Surface Refinement: Extracts the mesh and fixes the geometry, optimizing only the reflectance and lighting to recover person-specific details.

3. Key Contributions

Novel Capture Setup: A user-friendly system requiring only a smartphone and a dim room (curtained or night), eliminating the need for polarization filters, darkrooms, or outdoor sunlight.
Complete Face Modeling: The first method to successfully reconstruct a complete face including skin, mouth interior, hair, and eyes with plausible specularities from a single sequence.
Hybrid Representation: A novel architecture combining explicit mesh priors for eyes with neural fields for the rest of the face, solved via mesh-aware volume rendering.
Compact Lighting & Priors: A combined lighting model and the use of AlbedoMM priors to effectively disentangle diffuse and specular components in a low-cost setup.

4. Results and Evaluation

Qualitative Results: The method produces high-quality, relightable 3D assets compatible with standard CG software (e.g., Blender). It successfully generates realistic eye specularities and handles challenging facial expressions.
Quantitative Performance:
- Compared to NextFace++ (a state-of-the-art in-the-wild method), the proposed method achieves significantly higher PSNR (26.12 vs. 17.62) and SSIM, and lower LPIPS, particularly in eye regions.
- Compared to WildLight (generic object inverse rendering), the proposed method outperforms it in both geometry and reflectance reconstruction due to the integration of face-specific priors and the hybrid representation.
Comparison to Studio Methods: While slightly behind high-end Light Stage solutions in resolution and pore-level detail (due to smartphone camera limits), the method achieves a remarkably high quality-to-cost ratio.
Application: The authors demonstrate relightable facial performance capture by combining their method with Reflectance Transfer, allowing a captured performance sequence to be re-lit in new environments.

5. Significance

This work represents a significant step toward democratizing high-fidelity digital human creation. By removing the need for expensive hardware and complex capture environments, it enables broad daily users to create professional-grade, relightable 3D avatars. The ability to model the entire face (including the difficult eye region) makes the resulting assets suitable for high-end applications in VR, AR, and film production, bridging the gap between "in-the-wild" convenience and studio-quality results.

Limitations

Manual Eyeball Setup: The position and radius of the eyeball spheres are manually set (though automatic estimation is proposed as future work).
Capture Duration: The 25-second capture time can lead to motion blur or texture baking if the subject moves significantly.
Hair Modeling: Hair is currently modeled as a diffuse surface; strand-based hair modeling remains a challenge for low-cost setups.
Resolution: Limited by smartphone video quality compared to DSLR-based studio scans.