💻 computer science

Consistent Feature Transport for Image Relighting

This paper introduces Consistent Feature Transport (CFT), a training principle built on rectified flow that reformulates image relighting as an illumination feature transport problem to explicitly enforce consistent feature transformation between source and target images, thereby achieving superior illumination control and content preservation compared to existing diffusion-based methods.

Original authors: Bohan Zhang, Huanwei Liang, Yuhan He, Hongteng Xu, Quxiao Chao, Luoqi Liu, Dixin Luo, Ting Liu

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bohan Zhang, Huanwei Liang, Yuhan He, Hongteng Xu, Quxiao Chao, Luoqi Liu, Dixin Luo, Ting Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a digital artist with a magic wand that can change the lighting in any photo. You want to turn a gloomy, rainy street scene into a sunny, golden-hour masterpiece, or shift a portrait from a harsh office fluorescent to a soft, romantic sunset glow. But here's the catch: you don't want to change the person's face, their clothes, or the shape of the buildings. You only want to move the light. This is the challenge of "image relighting." In the world of computer science, specifically in a field called generative AI, researchers use "diffusion models"—think of them as incredibly smart artists who learn to paint by starting with a messy cloud of static noise and slowly refining it into a clear picture. While these AI artists are amazing, they sometimes struggle when asked to just change the light. They might accidentally change the person's hair color, warp their face, or make the shadows look like they belong to a different planet. The big question is: how do we teach the AI to move the light around without messing up everything else?

This paper introduces a new trick called Consistent Feature Transport (CFT) to solve that exact problem. The authors, a team from Beijing Institute of Technology, Meitu Inc., and Renmin University of China, realized that previous methods were trying to guess how to change the light, often leading to inconsistent or messy results. Instead, they decided to teach the AI a specific "dance" between the original photo and the new, relit photo. They formulated relighting not just as painting over an image, but as a precise "feature transport" problem. Imagine you have two identical clay sculptures, but one is lit by a red lamp and the other by a blue lamp. The goal is to teach the AI how to slide the "red light" features from the first sculpture to the second without squishing the clay itself.

To do this, the researchers built a massive new dataset of 34,695 portrait image pairs, featuring everything from neon city lights to soft morning sun, specifically designed to be tricky and diverse. They then trained their AI using a special three-part rulebook. First, they taught it how to generate images from noise (the standard way). Second, they made sure it could perfectly reconstruct the original image (so it doesn't forget what the person looks like). But the secret sauce is the third rule: they forced the AI to learn the "transport" between two different photos that share the same lighting change but feature different people. It's like showing the AI a photo of a person in a red light, then a photo of a different person in the same red light, and saying, "Figure out how the light moved from the first person to the second, and apply that exact same move to this new photo." By doing this, the AI learns that "moving the light" is a specific, consistent action, separate from "changing the face."

The results are quite promising. When they tested their new method against other top-tier AI tools, their approach consistently produced better results. In the numbers, their best model achieved a Structural Similarity Index (SSIM) of 0.9202 and a Peak Signal-to-Noise Ratio (PSNR) of 23.5105, which are higher scores than most competitors, meaning the faces stayed truer to the original and the lighting looked more realistic. They also found that this trick wasn't just for lighting; when they tried it on "style transfer" (changing a photo to look like a painting), it worked there too, suggesting this "feature transport" idea is a powerful tool for many types of image editing. The authors suggest that by explicitly teaching the AI how to move specific features (like light) while holding others (like identity) steady, we can make digital editing much more reliable and less prone to weird glitches.

Technical Summary: Consistent Feature Transport for Image Relighting

Problem Definition

Image relighting aims to modify the illumination conditions of an image (e.g., direction, intensity, color, and light-material interaction) while strictly preserving non-lighting content such as identity, geometry, and material appearance. The paper identifies three primary limitations in existing diffusion-based approaches:

Control-based methods: While they offer controllability via signals (e.g., environment maps), they rely heavily on the accuracy and completeness of these signals, which is difficult to specify for complex lighting.
Decomposition-based methods: These attempt to isolate lighting factors via intrinsic decomposition but often suffer from inaccuracies in decomposition, leading to errors like inconsistent shading or color shifts.
Direct Image-to-Image Translation: These treat relighting as an instance-level translation problem. While they preserve structure, they lack an explicit mechanism to model illumination-specific feature displacement, often resulting in limited precision and controllability.

Furthermore, the field is constrained by a lack of large-scale, diverse portrait relighting datasets, particularly those featuring complex, real-world illumination effects.

Methodology: Consistent Feature Transport (CFT)

The authors reformulate image relighting as an illumination feature transport problem. They introduce Consistent Feature Transport (CFT), a training principle built upon Rectified Flow that explicitly enforces illumination-consistent transport between source and target image distributions.

Core Framework

The method models the relationships between noise ( $z_0$ ), the source image ( $x_{src}$ ), and the target image ( $x_{tgt}$ ). The training objective consists of three components:

Noise-to-Target Generation ( $\mathcal{L}_1$ ): The model learns to transport samples from a prior distribution to the target relit image distribution, conditioned on the source image and a lighting description ( $c_{tgt}$ ).
Noise-to-Source Reconstruction ( $\mathcal{L}_2$ ): The model learns to transport samples from the prior to the source image distribution under a neutral condition ( $c_{src}$ ). This strengthens content preservation.
Illumination-Consistent Source-to-Target Transport ( $\mathcal{L}_3$ ): This is the novel contribution. Instead of directly supervising the transport from $x_{src}$ $x_{sr c}$ to $x_{tgt}$ $x_{t g t}$ (which risks learning instance-specific correlations), the model is supervised using an alternative image pair $(x'_{src}, x'_{tgt})$ $(x_{sr c}^{'}, x_{t g t}^{'})$ that shares the same illumination transformation but differs in identity and scene content.
- The model predicts a direct velocity field ( $v_t^{direct}$ ) between the source and target latent spaces using the parallelogram law.
- The loss $\mathcal{L}_3$ minimizes the difference between this predicted velocity and the ground truth transport derived from the alternative pair.
- This forces the model to learn a generalizable illumination feature transformation rather than fitting specific content differences.

The final objective is a weighted sum: $\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2 + \alpha \cdot \mathcal{L}_3$ .

Dataset Construction

To support complex lighting scenarios, the authors constructed a large-scale Portrait Relighting Dataset containing 34,695 image pairs (34,249 for training, 446 for testing).

Source: Collected from existing open-source portrait datasets.
Generation: Candidate relit images were generated using pre-trained models (Nano Banana, Seedream, RoboNeo) guided by diverse text prompts covering natural and artificial lighting (e.g., Rembrandt, Tyndall effects, neon lights).
Filtering: An ensemble of three models (QwenVL, EditScore, GPT-4o) and manual selection were used to filter for illumination quality, content consistency, and physical plausibility.

Key Results

The method was evaluated against control-based, decomposition-based, and direct translation baselines (including IC-light, LBM, FlowEdit, and Intrinsic Edit) using metrics like SSIM, PSNR, LPIPS, FID, and Estimated Irradiance (EI).

Quantitative Performance: CFT consistently outperformed state-of-the-art methods. For instance, when applied to the Flux-Kontext backbone, CFT achieved an SSIM of 0.9202 and PSNR of 23.5105, significantly surpassing the original pretrained model (SSIM 0.6386) and other baselines.
Qualitative Improvements: Visual results demonstrated more physically coherent lighting, better preservation of identity and geometry, and fewer artifacts (e.g., color shifts) compared to decomposition-based methods.
Transfer Learning: A model trained with CFT on the constructed dataset generalized well to the unseen "Multi-Illumination Dataset," outperforming models trained without CFT, indicating the learned lighting features are robust.
User Study: In a study with 30 volunteers, CFT-based methods received the highest scores across Illumination Quality, Content Consistency, Physical Plausibility, and Image Aesthetics.
Generalization: The authors applied CFT to a Style Transfer task. Results showed that CFT improved structural preservation and perceptual fidelity (higher SSIM/PSNR, lower LPIPS), though it slightly reduced global distribution diversity (higher FID) due to the deterministic nature of consistent transport.

Significance and Claims

The paper claims that the primary significance of this work lies in:

Reformulating Relighting: Shifting the paradigm from generic image translation or conditional generation to explicit illumination feature transport. This allows the model to isolate lighting variations from content.
Training Principle: The introduction of the consistent feature transport principle (using alternative pairs with shared illumination for supervision) effectively decouples illumination learning from instance-specific content, leading to more stable and consistent relighting.
Data Resource: The construction of a large-scale, diverse portrait relighting dataset addresses a critical gap in available resources for training models on complex lighting effects.
Unified Strategy: The demonstration that CFT generalizes to style transfer suggests it is a unified strategy for image editing tasks that require modifying specific attributes while preserving underlying structure.

The authors conclude that CFT provides a robust framework for achieving faithful content preservation while enabling targeted, realistic, and controllable illumination modifications.