💻 computer science

LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting

LightCrafter is a hybrid video relighting framework that achieves physically grounded, temporally consistent, and controllable illumination by translating PBR-rendered proxy videos rather than raw footage, leveraging a post-trained CogVideoX to capture complex effects like global illumination.

Original authors: Zixin Guo, Yehonathan Litman, Yifeng He, John Miller, Chuhan Chen, Deva Ramanan

Published 2026-07-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zixin Guo, Yehonathan Litman, Yifeng He, John Miller, Chuhan Chen, Deva Ramanan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a home video of a sunny day, but you want to magically turn it into a spooky, moonlit scene without changing the actors, the trees, or the camera's movement. That's the dream of "video relighting." But here's the catch: making light behave realistically over time is incredibly hard. If you just ask a smart AI to "make it dark," it might get confused, making the shadows flicker or the colors drift as the video plays.

The authors of this paper, LightCrafter, found a clever shortcut. Instead of asking the AI to invent the new lighting from scratch, they decided to bake the lighting into a "rough draft" first, and then just ask the AI to clean up the mess.

The Two Paths That Didn't Work (According to the Paper)

Before finding their solution, the researchers looked at two other ways people tried to do this, and they found them wanting:

The "Perfect Reconstruction" Attempt: Some methods try to reverse-engineer the video to figure out exactly what the 3D shapes, materials, and lights were, and then re-render it. The paper argues this is like trying to rebuild a shattered vase perfectly just by looking at the pieces; it's often too noisy and ambiguous. If the reconstruction is slightly wrong, the new lighting looks broken.
The "Magic AI" Attempt: Other methods just ask a generative AI to translate the video from "sunny" to "moonlit" using text or a map. The paper suggests this is risky because the AI might forget the rules of physics over a long video. It's like asking a storyteller to rewrite a whole novel in a new style; they might change the characters' names or forget who is standing where by the end.

The LightCrafter Solution: The "Rough Draft" Trick

The team's main finding is that you don't need the AI to understand how light works; you just need it to understand how to fix a bad drawing.

Here is their three-step recipe:

Step 1: The "Rough Draft" (The PBR Proxy)
First, they use a standard computer graphics engine (called a Physically-Based Renderer, or PBR) to create a "rough draft" of the video under the new lighting. Think of this like a sketch artist quickly drawing the scene with the new moonlight.

The Good News: The paper found that this "rough draft" actually does a surprisingly good job on its own! It gets the shadows and the general mood right because it follows the laws of physics.
The Bad News: Because the computer had to guess the 3D shapes from a flat video, the sketch has glitches. The shadows might look jagged, the textures might look plastic, or there might be weird holes where objects disappear.

Step 2: The "Polisher" (The Diffusion Refiner)
This is where the magic happens. They take a powerful video AI (based on a model called CogVideoX) and teach it one specific job: fix the glitches in the rough draft.

Instead of asking the AI to "make it dark," they show it the "rough draft" and the "perfect final video" side-by-side.
The AI learns to spot the jagged shadows and the plastic textures and smooth them out. It's like a photo editor who only knows how to remove blemishes, not how to repaint the whole picture.
Because the "rough draft" already has the correct lighting baked in, the AI doesn't have to guess where the light should go. It just makes it look real.

Step 3: The Long-Video Trick
Videos in the real world are long, but AI models usually get tired after a short clip. If you try to edit a long video in chunks, the lighting might drift (the moon might suddenly turn blue in the middle of the video).

The authors solved this by using a technique called overlap-fused temporal tiling. Imagine editing a long movie by overlapping the scenes slightly, like a puzzle, so the edges blend perfectly. They fuse the AI's predictions together so the lighting stays consistent from the very first second to the very last, no matter how long the video is.

How They Taught the AI

To make sure the AI could handle real-world messiness, they didn't just use perfect computer graphics. They built a special training pipeline:

Synthetic Data: They created fake videos where they knew the "perfect" answer and the "rough draft" answer.
Real-World Data: They took real videos, ran them through their "rough draft" process, and used the original video as the "perfect" answer.
The Result: By training on both, the AI learned to fix the specific kinds of errors that happen when you try to guess 3D shapes from a 2D video. The paper suggests that if you only trained on perfect computer graphics, the AI would fail on real videos, and if you only trained on real videos, it wouldn't learn the physics well enough.

What Can You Do With This?

Because the system relies on a "rough draft" that is built from 3D rules, it's surprisingly flexible. The paper shows you can:

Insert new lights: Add a virtual lamp to the scene, and the AI will make the shadows fall correctly.
Change materials: Turn a shiny car into a matte one, and the AI will adjust the reflections.
Move objects: The system can handle adding new objects into the scene without needing to be retrained.

The Bottom Line

The paper concludes that this "rough draft then polish" approach works better than trying to generate the whole thing from scratch or trying to perfectly reconstruct the scene first. In their tests on synthetic and real-world videos, LightCrafter produced more stable, consistent, and realistic results than previous methods.

However, the authors are honest about the limits: if the original video has very shiny, transparent, or metallic surfaces, or if the camera moves in a way that confuses the 3D guess (like motion blur), the "rough draft" might be too messy for the AI to fix perfectly. But for most scenes, this method suggests a new, more reliable way to change the mood of a video without breaking the physics.

Technical Summary: LightCrafter

Problem Statement

Video relighting requires editing scene appearance while preserving content, geometry, materials, and motion, all while maintaining temporal coherence and physical consistency. Existing approaches generally fall into two paradigms, both of which face significant limitations:

Inverse Rendering + Forward Rendering: These methods explicitly reconstruct intrinsic scene properties (geometry, materials, illumination) and re-render them under new lighting. While they offer precise control and temporal consistency, they suffer from noisy reconstructions due to the ill-posed nature of monocular inverse rendering. Errors in geometry or material estimation propagate directly to the output, and they struggle to model complex effects like global illumination.
Generative Video-to-Video Translation: These methods frame relighting as a generative task conditioned on target illumination (e.g., environment maps or text). While they leverage strong diffusion priors for photorealism, they often lack precise control over shadows and shading, struggle with long-form temporal consistency due to limited context windows, and are hindered by the scarcity of paired real-world training data.

Methodology

The authors propose LightCrafter, a hybrid pipeline that reformulates video relighting as a PBR-conditioned video refinement task. Instead of directly translating an input video to a relit target, the method translates a Physically-Based Rendering (PBR) proxy video (rendered under the target illumination) into a photorealistic output.

The pipeline operates in three main stages:

1. Inverse Rendering (Scene Recovery)

Given a monocular input video, the system recovers scene intrinsics using off-the-shelf methods:

Geometry & Camera: Depth maps and camera parameters are extracted (e.g., via MegaSAM) to construct per-pixel aligned meshes in world coordinates.
Materials: Albedo, normal maps, and BRDF parameters (roughness, metalness) are extracted (e.g., via DiffusionRenderer).
Source Illumination: An environment map is estimated and refined via differentiable rendering to match the input video.
This results in a recovered scene state $S$ and a source illumination estimate.

2. Forward Rendering (PBR Proxy Generation)

Using the recovered intrinsics $S$ and a user-specified target illumination $L_{target}$ , a physically-based renderer generates a proxy video $P$ .

The renderer solves the rendering equation using a Lambertian diffuse lobe and an isotropic specular lobe (Cook-Torrance microfacet model).
Because the rendering is deterministic and grounded in recovered geometry, the resulting proxy $P$ is frame-aligned and temporally consistent by construction.
Crucially, the proxy captures most structured lighting changes (shadows, shading, highlights) but retains artifacts from the imperfect inverse rendering (e.g., flickering, jagged shadows, disocclusion holes).

3. PBR-Conditioned Diffusion Refinement

A video diffusion model (initialized from CogVideoX-5B) is fine-tuned to refine the noisy PBR proxy $P$ into a photorealistic relit video.

Training Objective: The model learns to predict noise added to the ground-truth relit video, conditioned on the latent representation of the PBR proxy.
Artifact Matching: To address the train-test mismatch, the training data is curated such that the PBR proxies contain the same types of artifacts (noise, geometry errors) that the model will encounter at inference.
Long-Form Consistency: To handle videos longer than the model's training window, the authors employ overlap-fused temporal tiling. The video is split into overlapping windows, denoised, and fused in the noise-prediction space using temporal tent weights. This ensures a single coherent denoising trajectory, preventing lighting drift across chunk boundaries.

Data Curation Strategy

The authors introduce a novel data curation pipeline to bridge the gap between synthetic and real-world data:

Synthetic Pairs: Ground-truth videos are rendered, then inverse-rendered and re-rendered to create pairs of (Noisy PBR, Ground Truth). This teaches the model to correct reconstruction artifacts.
Real-World Pseudo-Pairs: Real videos are inverse-rendered and re-rendered under their own estimated lighting. The pair (Re-rendered Input, Original Input) teaches the model to correct artifacts on real-world appearances without requiring ground-truth relit videos.

Key Contributions

Hybrid Pipeline: A reformulation of video relighting as a refinement task where a PBR proxy provides explicit lighting control and temporal consistency, while a diffusion model handles photorealism and artifact correction.
PBR Proxy Superiority: The authors demonstrate that the PBR rendering alone often outperforms prior state-of-the-art end-to-end methods in terms of lighting control and consistency, validating the quality of modern inverse renderers.
Artifact-Matched Training: A data curation strategy that exposes the diffusion model to realistic rendering artifacts during training, significantly improving generalization to inference-time conditions.
Long-Form Consistency: A temporal tiling and fusion mechanism that enables consistent relighting of arbitrarily long videos without the drift common in chunked processing.
Downstream Editing: The framework naturally supports scene editing tasks (material changes, object insertion, light insertion) by simply re-rendering the PBR proxy with modified inputs, requiring no retraining of the diffusion model.

Experimental Results

The method was evaluated on synthetic and real-world benchmarks (MIT Multi-Illumination, DL3DV).

Quantitative Performance: LightCrafter achieves state-of-the-art performance in PSNR, SSIM, and LPIPS compared to baselines like LightX, UniRelight, and DiffusionRenderer. It significantly outperforms baselines in temporal consistency metrics (T-CLIP, Warp-SSIM).
Qualitative Results: The method produces stable shadows, highlights, and global appearance over long sequences, whereas baselines exhibit visible degradation, inconsistent shadows, and color drift.
Ablation Studies: Experiments show that training on both synthetic and real-world data is critical. Removing real-world data leads to hallucinated structures, while removing synthetic data results in poor relighting accuracy. The artifact-matched curation is essential for correcting in-the-wild artifacts.

Significance

The paper claims that LightCrafter achieves a new balance between controllability and photorealism in video relighting. By leveraging the explicit physical grounding of PBR rendering for structure and consistency, and the generative power of diffusion models for refinement, the method overcomes the limitations of purely inverse rendering (noise/artifacts) and purely generative approaches (lack of control/temporal drift). The authors assert that this approach enables consistent, long-form relighting and extends naturally to broader scene editing tasks, all while releasing their dataset, benchmarks, and code to the community.