💻 computer science

GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers

GeoRelight introduces a unified Multi-Modal Diffusion Transformer that jointly performs 3D geometry reconstruction and image relighting from a single photo by leveraging a novel distortion-free depth representation and mixed-data training to overcome the limitations of sequential pipelines and ensure physical consistency.

Original authors: Yuxuan Xue, Ruofan Liang, Egor Zakharov, Timur Bagautdinov, Chen Cao, Giljoo Nam, Shunsuke Saito, Gerard Pons-Moll, Javier Romero

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxuan Xue, Ruofan Liang, Egor Zakharov, Timur Bagautdinov, Chen Cao, Giljoo Nam, Shunsuke Saito, Gerard Pons-Moll, Javier Romero

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a single, flat photograph of a person. Now, imagine you want to take that photo and move the person into a completely different world—maybe a sunny beach, a dark cave, or a neon-lit city—and have the lighting look real.

The problem is that a 2D photo is a magic trick. It hides the truth. It mixes together three things that are actually separate:

The Shape: The 3D bumps, wrinkles, and curves of the person.
The Paint: The actual color of their skin and clothes (ignoring shadows).
The Light: The sun or lamps that created the shadows in the original photo.

Trying to separate these three things from just one photo is like trying to un-bake a cake to get the flour, eggs, and sugar back out. It's nearly impossible, and most computers get confused, making shadows that look fake or shapes that look flat.

GeoRelight is a new AI system that solves this puzzle. Here is how it works, using some simple analogies:

1. The "Swiss Army Knife" Brain (The Multi-Modal DiT)

Old methods were like a factory assembly line: Step 1, guess the shape. Step 2, guess the color. Step 3, paint the new light. If you made a mistake in Step 1, the whole cake was ruined.

GeoRelight is different. It's like a Swiss Army Knife that does everything at once. It uses a special "brain" (a Diffusion Transformer) that looks at the photo and simultaneously guesses the shape, the color, and the new lighting. Because it does them all together, they help each other.

Analogy: Imagine trying to guess the shape of a hidden object by feeling it. If you also know what the object is made of (clay vs. metal), you can guess the shape better. GeoRelight uses the "clay" (color) to guess the "shape," and the "shape" to guess the "light," all in one go.

2. The "Perfect Map" (iNOD)

To teach the AI about 3D shapes, you usually need a "depth map" (a picture where white is close and black is far). But standard maps are messy when you try to compress them for the AI to learn. They get blurry or distorted, like a map of the world that stretches the poles and squishes the equator.

GeoRelight invented a new way to draw maps called iNOD.

Analogy: Think of a standard depth map like a rubber sheet that stretches unevenly. GeoRelight's iNOD is like a perfectly inflated balloon. No matter how you look at it, the shape stays true. It wraps the 3D shape into a flat picture without squishing or stretching it, so the AI can read it perfectly without getting confused.

3. The "Teacher and the Student" (Mixed-Data Training)

To learn this skill, the AI needs to practice.

Synthetic Data: Imagine a video game world where the computer knows the exact shape and light of everything. The AI learns the rules of physics here perfectly, but the pictures look a bit like a video game (too clean).
Real Data: Imagine taking photos of real people in the wild. These look amazing and realistic, but the computer has no idea what the "true" shape or light is. It's like a student trying to learn math without an answer key.

GeoRelight uses a clever trick:

It trains on the Video Game data first to learn the rules of physics.
Then, it acts as a Teacher to label the Real World photos. It says, "I think this real photo has this shape and this light."
Finally, it studies the Real World photos using those "teacher labels" to learn how to make the final result look photorealistic, not like a video game.

What Can It Do?

When you give GeoRelight a single photo, it doesn't just change the background. It:

Rebuilds the 3D person: It creates a tiny, detailed 3D point-cloud of the person (like a digital sculpture).
Peels off the paint: It separates the person's actual skin/clothing color from the shadows.
Re-lights the scene: It can put that person into a new environment, and the shadows will fall correctly on their nose, under their chin, and on their clothes, just like a real camera would capture it.

Why Does This Matter?

This is a huge leap for things like:

Movies & Games: You can take a photo of an actor and instantly put them into a sci-fi scene without expensive 3D scanning studios.
Virtual Reality: You can create realistic avatars from just a selfie.
Editing: You can change the time of day in a photo (from noon to sunset) and have the shadows move realistically.

In short, GeoRelight is like a digital time-traveler for light. It takes a frozen moment in time, figures out the hidden 3D secrets of the scene, and lets you shine a new light on it, making the impossible look perfectly real.

1. Problem Statement

Relighting a person from a single monocular image is a fundamental but ill-posed problem in computer vision and graphics. A single 2D image is an ambiguous projection of three entangled factors:

3D Geometry: The subject's shape.
Intrinsic Appearance: Base color (albedo) and material properties.
Illumination: The scene's lighting conditions.

Limitations of Existing Methods:

End-to-End "Translators": Models that map pixels directly to pixels often lack physical plausibility because they do not explicitly model geometry, failing to generate realistic shadows or highlights consistent with 3D shape.
Sequential Pipelines: Methods that first estimate intrinsics (albedo, normals) and then render the image suffer from error accumulation. Inaccuracies in the initial geometry estimation are "baked" into the intermediate buffers and cannot be corrected by the subsequent rendering stage.
Data Scarcity: High-quality real-world data with accurate 3D geometry and intrinsic labels is extremely difficult to acquire, while synthetic data often lacks photorealism.

2. Methodology: GeoRelight

The authors propose GeoRelight, a unified generative framework that solves relighting and 3D reconstruction jointly rather than sequentially. The core philosophy is that accurate geometry aids relighting (for shadows), and shading cues aid geometry estimation (shape-from-shading).

A. Architecture: Flexible Multi-Modal Diffusion Transformer (DiT)

The framework is built upon a Video Latent Diffusion Transformer (DiT).

Modality-as-Time: The authors repurpose the temporal dimension ( $T$ ) of a video DiT as a modality dimension ( $M$ ). This allows the model to stack different target modalities (Albedo, Normals, Geometry, Segmentation, Relit Image) and process them in parallel as if they were video frames.
Conditioning:
- Global Condition: The input image latent is concatenated to all modalities to preserve identity and texture.
- Illumination Condition: High Dynamic Range (HDR) environment maps are decomposed into three Low Dynamic Range (LDR) representations (Reinhard tonemapped, log-intensity, directional) and encoded into a lighting latent, concatenated only to the relit image target.
- Modality Switch ( $c_{switch}$ ): A binary mask signals whether a specific modality is a "clear" input condition (to be used as guidance) or a "noisy" target (to be generated). This enables flexible inference (e.g., generating relit images given known normals, or vice versa).

B. Key Technical Contribution: Isotropic NDC-Orthographic Depth (iNOD)

A major challenge is representing 3D geometry in a way compatible with pretrained 2D Variational Autoencoders (VAEs) used in latent diffusion.

Problem: Standard point maps become noisy after VAE compression. Standard depth maps require anisotropic normalization (scaling only the Z-axis), which distorts 3D shapes and requires camera intrinsics for recovery.
Solution (iNOD):
1. Unproject: Convert metric depth to a 3D point cloud.
2. Isotropic Normalization: Scale the entire 3D geometry (X, Y, and Z) to fit within a $[-1, 1]$ bounding box based on its longest edge. This preserves the aspect ratio and relative geometry without distortion.
3. Orthographic Projection: Project the normalized 3D points onto the XY-plane, using the Z-value as the pixel value.
4. Dilation: A morphological dilation is applied to the foreground to prevent boundary artifacts caused by VAE compression.
Benefit: iNOD is a VAE-friendly, distortion-free 2D representation that requires no camera intrinsics at inference time.

C. Strategic Mixed-Data Training

To bridge the gap between synthetic accuracy and real-world photorealism, the authors employ a two-stage training strategy:

Stage 1 (Synthetic Pre-training): Train on a large synthetic dataset (8,000 identities) to learn the fundamental disentanglement of geometry, albedo, and lighting.
Stage 2 (Mixed Post-Training):
- Auto-Labeling: The Stage 1 model is used to generate pseudo-ground-truth intrinsics (albedo, normals, iNOD) for real-world datasets (Light Stage "Dome" and "In-the-Wild" data).
- Task-Specific Training: The model is trained on a hybrid dataset using different modes:
  - Synth/Dome: Full joint generation (Default/Rendering modes) to learn physical relationships.
  - In-the-Wild (ITW): "Intrinsic $\to$ Relit" mode. The model is given the auto-labeled intrinsics as a condition and tasked with generating the photorealistic relit image. This teaches the model to synthesize realistic appearance from real-world data without needing ground-truth relit pairs.

3. Key Contributions

Unified Framework: The first framework to jointly generate relit images, intrinsic color, surface normals, and 3D geometry in a single multi-modal DiT pass.
iNOD Representation: A novel, distortion-free depth representation that is robust to VAE compression and eliminates the need for camera intrinsics during inference.
Strategic Training: A methodology combining synthetic data (for geometric accuracy) and auto-labeled real-world data (for photorealism) to overcome data scarcity.

4. Results

The authors evaluated GeoRelight on synthetic data, Light Stage captures, and the HumanOLAT dataset, comparing against state-of-the-art (SoTA) baselines like NeuralGaffer, IC-Light, DiffusionRenderer, and VGGT.

Relighting Performance: GeoRelight achieves State-of-the-Art (SoTA) results across all metrics (PSNR, SSIM, LPIPS, RMSE). It produces physically plausible shadows and highlights consistent with the subject's 3D shape, outperforming baselines that often suffer from artifacts or incorrect exposure.
3D Reconstruction: The model outperforms specialized geometry estimators (VGGT, MoGe2) in Chamfer Distance and F-Score. It captures fine-grained details (wrinkles, hair, clothing folds) that other methods miss or distort.
Intrinsic Estimation: Achieves SoTA performance in albedo and normal estimation, with significantly lower angular error and RMSE compared to existing methods.
Ablation Studies:
- Joint Modeling: Removing geometry generation leads to a significant drop in relighting quality (loss of shadows/wrinkles).
- Shape-from-Shading: Providing relit images as conditions significantly improves the accuracy of the generated normal maps, proving the synergy between the tasks.
- In-the-Wild Data: Including ITW data corrects "dark bias" observed in models trained only on synthetic/Light Stage data, resulting in balanced, natural illumination.

5. Significance and Impact

Synergy of Tasks: The paper demonstrates that relighting and geometry reconstruction are mutually beneficial tasks. Solving them jointly prevents error accumulation and leverages cross-modal cues (shading $\to$ shape, shape $\to$ shading).
Practical Applicability: By removing the need for camera intrinsics and enabling high-fidelity generation from a single image, GeoRelight opens doors for applications in virtual reality, film production, and creative editing.
Future Directions: The authors note that while currently a single-image model, the architecture is built on a video DiT, making it naturally extensible to video relighting with temporal consistency. The disentangled outputs also serve as a foundation for 3D-consistent avatar generation.

In summary, GeoRelight represents a paradigm shift from sequential pipelines to joint generative modeling, achieving superior physical consistency and visual fidelity by unifying geometry and appearance estimation within a flexible diffusion framework.