🤖 machine learning

PIXLRelight: Controllable Relighting via Intrinsic Conditioning

PIXLRelight is a fast, feed-forward neural renderer that achieves state-of-the-art, physically controllable single-image relighting by bridging physically based rendering and learned synthesis through shared intrinsic conditioning derived from albedo, diffuse shading, and residuals.

Original authors: Miguel Farinha, Ronald Clark

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Miguel Farinha, Ronald Clark

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful photograph of a room, but the lighting is a bit dull. You wish you could move a lamp, change the color of the sun, or add a spotlight, just like a director does on a movie set. Usually, doing this with a single photo is incredibly hard because the computer doesn't know where the walls are, what the objects are made of, or where the light is coming from.

PIXLRelight is a new tool that solves this problem. Think of it as a "Lighting Translator" that bridges the gap between the messy, real world and the perfect, controllable world of 3D computer graphics.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Reverse Engineering" Trap

Previous methods tried to solve this by doing two difficult things at once:

Guessing the 3D world: Trying to figure out the shape and material of everything in the photo.
Re-rendering: Trying to paint a new picture based on those guesses.

The paper compares this to trying to rebuild a car engine by looking at a photo of the car, then trying to paint a new car based on that sketch. If your sketch of the engine is slightly wrong, the new car looks broken. Errors pile up, and the result often looks fake or blurry.

2. The Solution: The "Intrinsic Recipe"

PIXLRelight takes a different approach. Instead of asking the computer to guess the whole 3D world, it asks for a "Lighting Recipe" (which the paper calls an intrinsic decomposition).

Think of a photograph as a cake made of three layers:

The Flavor (Albedo): The actual color of the object (e.g., the red of an apple), regardless of the light.
The Frosting (Diffuse Shading): How the light hits the surface to create soft shadows and gradients.
The Sprinkles (Non-Diffuse Residual): The shiny spots, reflections, and transparency that make things look realistic.

The Magic Trick:

Training Time: The AI learns by looking at pairs of real photos taken of the same scene under different lights. It learns how to swap the "Frosting" and "Sprinkles" while keeping the "Flavor" (the object's true color) exactly the same.
Inference Time (When you use it): You don't give the AI a photo of the new light. Instead, you give it a 3D sketch of your scene (a rough guess of the shape) and tell a 3D computer program (Blender) to light it up exactly how you want. The 3D program spits out just the "Frosting" and "Sprinkles" layers.
The Result: The AI takes your original photo and the new "Frosting/Sprinkles" recipe and instantly paints the new light onto your photo.

3. Why It's Special

It's a "Feed-Forward" Chef: Most other AI lighting tools are like chefs who taste the soup, add salt, taste again, and repeat for a long time. PIXLRelight is like a chef who looks at the recipe and the ingredients, mixes them once, and serves the dish instantly. It takes less than a tenth of a second.
It Preserves the Original: Instead of trying to redraw the whole picture from scratch (which often loses details like the texture of a shirt or the grain of wood), PIXLRelight acts like a smart filter. It keeps your original photo exactly as it is and only changes the lighting math. It's like putting a new pair of colored glasses on the photo rather than repainting the whole thing.
It Handles Mistakes: Since the 3D sketch used to create the "recipe" is often rough and imperfect, the AI was trained to ignore those imperfections. It focuses only on the lighting instructions, not the messy 3D shape.

4. The Result

The paper shows that PIXLRelight can take a photo of a room and let you move a light source, change its color, or add a spotlight, and the result looks photorealistic. It is significantly faster and higher quality than previous methods, which often struggled to keep the image sharp or failed to handle complex indoor lighting.

In short: PIXLRelight separates the "what" (the object's color) from the "how" (the lighting), allowing you to change the lighting of a real photo with the same control a 3D artist has, but in a fraction of a second.

Technical Summary: PIXLRelight

Problem Definition

Single-image relighting aims to modify the illumination of a photograph to match a user-specified target lighting condition. While physically based rendering (PBR) engines (e.g., Blender, Unreal Engine) offer precise control over light type, position, color, and intensity, applying this control to "in-the-wild" photographs is fundamentally difficult due to the challenge of recovering geometry, materials, and light transport from a single image.

Existing approaches suffer from three primary limitations:

Limited Control: Methods conditioned on HDR environment maps, text, or reference images struggle to express spatially localized, multi-source illumination common in indoor scenes.
Error Accumulation: Inverse-then-forward pipelines that estimate G-buffers and re-render them often fail to encode necessary cues (e.g., transparency, subsurface scattering), causing errors to compound across stages.
Computational Cost: Approaches combining PBR with neural rendering (e.g., Careaga and Aksoy [5]) often require costly per-image differentiable rendering optimizations to fit 3D lighting environments, limiting training data diversity and inference speed.

Methodology

The authors propose PIXLRelight, a feed-forward transformer-based approach that bridges PBR controllability and learned image synthesis through a shared intrinsic conditioning interface. The core insight is to decouple what the target lighting should be from how it is applied to a real photograph.

1. Intrinsic Conditioning Interface

The method utilizes an intrinsic image formation model where an image $I$ is decomposed into:
$I = A \odot S + R$
Where $A$ is the diffuse albedo, $S$ is the diffuse shading, and $R$ is the non-diffuse residual (specular highlights, transparency, etc.).

The target lighting condition is represented not by an RGB image, but by the channel-wise concatenation of the target intrinsic maps: $C_T = [A_T; S_T; R_T]$ . This interface is consistent across both training and inference:

Training: $C_T$ is extracted from real, paired multi-illumination photographs using a frozen intrinsic decomposition model (Marigold-IID-Lighting [20]).
Inference: $C_T$ is generated by path-tracing a coarse 3D reconstruction of the input scene under user-specified PBR lights. Crucially, the model never sees the rendered RGB image; it only sees the intrinsic buffers, which carry precise cues for the desired lighting while absorbing the imperfections of the underlying 3D reconstruction.

2. Architecture

PIXLRelight is a feed-forward transformer designed for dense prediction:

Asymmetric Encoders: The source image ( $I_S$ ), dominated by high-frequency details, is tokenized by a Vision Transformer (ViT). The intrinsic stack ( $C_T$ ), dominated by smooth, lower-frequency structures, is tokenized by a ConvNeXt.
Fusion: Source and conditioning tokens are fused per spatial location via a small MLP before entering a shared transformer trunk.
Modulation Head: Instead of regressing the target RGB image directly, the network predicts a per-pixel affine modulation of the source image. The output is computed as $\hat{I}_T = \text{clip}((1 + g) \odot I_S + b, 0, 1)$ , where $g$ (gain) and $b$ (bias) are learned maps. The network is initialized to output the identity ( $g=0, b=0$ ), learning only the residual lighting transformation. This preserves the source's photorealistic details by construction.

3. Training and Inference Strategy

Training: The model is trained end-to-end on paired multi-illumination datasets (MIIW, BigTime, VIDIT) using photometric and perceptual losses. No inverse-then-forward rendering pipeline or per-image optimization is required. To simulate inference-time artifacts, the training pipeline applies stochastic corruption augmentations to the intrinsic conditioning $C_T$ .
Inference:
1. Estimate metric depth (Depth Anything 3) and materials (Marigold-IID-Appearance) from the source image.
2. Construct a textured mesh and import it into Blender.
3. Author the desired lighting using arbitrary PBR lights.
4. Render the scene in Cycles to extract the target intrinsic buffers ( $A_T, S_T, R_T$ ).
5. Feed the source image and intrinsic buffers into PIXLRelight to generate the relit image in a single forward pass.

Key Contributions

Intrinsic Conditioning Interface: A unified conditioning mechanism using target appearance decomposition (albedo, shading, residual) that serves as a bridge between real photographs and PBR renders.
Direct Training Strategy: An end-to-end training approach on paired multi-illumination photographs that eliminates the need for inverse-then-forward rendering pipelines or per-image optimization.
PIXLRelight Model: A feed-forward transformer achieving state-of-the-art relighting quality with sub-0.1s inference time, enabling interactive lighting authoring.

Results

Quantitative Performance

Evaluated on the MIT Multi-Illumination Images in the Wild (MIIW) test split, PIXLRelight significantly outperforms existing baselines (DiffusionRenderer, UniRelight, RGBX, Ouroboros, V-RGBX):

PSNR: 29.18 dB (vs. 19.38 dB for the next best, Ouroboros).
SSIM: 0.893 (vs. 0.763).
LPIPS: 0.195 (vs. 0.438).
Speed: 0.09 seconds per image, which is at least an order of magnitude faster than all baselines (the next fastest being 0.62s).

Qualitative Performance

Controllability: In experiments using path-traced, user-authored lighting (Blender), PIXLRelight successfully transfers complex lighting effects (e.g., side-lit shadows, warm interior glows, focused spotlights) while preserving the source image's photographic details.
Robustness: Unlike baselines that drift from the source or fail to represent near-field sources, PIXLRelight maintains identity fidelity. It is robust to the coarse geometry and material errors inherent in single-image 3D reconstruction because the source RGB and target intrinsics are processed through separate branches.

Significance and Claims

The paper claims that PIXLRelight successfully brings the physical lighting control of computer graphics to in-the-wild photographs. By separating the specification of target lighting from its application, the method enables arbitrary PBR-style lighting control without the computational overhead of per-image optimization. The authors emphasize that this approach achieves state-of-the-art quality while running in under a tenth of a second, making interactive lighting authoring on real photographs feasible. The method relies on the assumption that intrinsic decomposition provides a sufficient bridge between real and synthetic domains, allowing the model to absorb reconstruction artifacts while accurately transferring lighting cues.