🤖 machine learning

HorizonRelight: Relighting Long-horizon Videos Consistently via Diffusion Transformers

Original authors: Jing Yang, Mayoore Jaiswal, Zian Wang, Steven Zeng, Rochelle Pereira, Yajie Zhao, Jianyuan Min

Published 2026-06-30

📖 4 min read☕ Coffee break read

Original authors: Jing Yang, Mayoore Jaiswal, Zian Wang, Steven Zeng, Rochelle Pereira, Yajie Zhao, Jianyuan Min

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a home video of a family dinner, and you want to change the lighting from a warm, cozy sunset to a cool, moody blue moonlight. You want the whole video to look like it was filmed under that new light, without the characters flickering or the room suddenly changing shape.

This is the problem HorizonRelight solves.

Here is the simple breakdown of how it works, using everyday analogies:

The Problem: The "Jigsaw Puzzle" Glitch

Current video AI tools are like artists who can only paint on small, 5-second sticky notes. If you give them a 2-minute video, they chop it up into 5-second chunks, paint each one, and tape them back together.

The Issue: Because the artist paints each sticky note independently, the end of one note might look slightly different from the start of the next. When you tape them together, you see a "seam" or a glitch where the lighting jumps or the object flickers.
The Paper's Term: This is called "temporal discontinuity at chunk boundaries."

The Solution: The "Conveyor Belt" Approach

HorizonRelight changes the workflow. Instead of painting 5-second notes in isolation, it treats the video like a conveyor belt.

The "Warm-Start" (The First Step):
Before the conveyor belt starts moving, the system takes a single "starter image" (generated by a smart AI like ChatGPT or a specialized tool) that shows exactly what the scene should look like under the new light. Think of this as setting the first domino. It tells the AI, "Start here, and keep this look."
The "Passing the Torch" (Chained Propagation):
This is the magic trick. When the AI finishes painting the first 5-second chunk, it doesn't just stop. It takes the very last few frames of that chunk and hands them to the AI working on the next chunk.
- Analogy: Imagine a relay race. The runner (the video chunk) doesn't just run their own race; they pass the baton (the visual state) to the next runner. The next runner starts exactly where the previous one left off, ensuring the lighting and shadows flow smoothly without any jumps.
The "Training Gym" (Masked Self-Conditioning):
To teach the AI how to do this relay race perfectly, the researchers trained it in a special way. During training, they would hide (mask) parts of the video and ask the AI to guess what comes next based only on the frames it just saw.
- Analogy: It's like a teacher showing a student the first half of a sentence and asking them to finish it. By practicing this over and over, the AI learns to "continue" the story naturally rather than restarting it every time.

What This Achieves

No More Flickering: Because the AI is constantly "remembering" the end of the previous chunk, the lighting stays consistent. You won't see the gray ball in the video suddenly change color when the video cuts to the next 5-second segment.
Long Videos: This allows the system to handle long videos (like a whole movie scene or a long travel vlog) without the quality degrading or the lighting getting weird halfway through.
Style Transfer: The paper notes this same "starter image" trick can be used to change the style of the video (e.g., turning a real video into a cartoon) while keeping the lighting consistent, not just changing the light source.

The Catch (Limitations)

The paper is honest about one limitation: If you make the video extremely long (much longer than a typical movie scene, say over 300 frames), the fine details (like the texture of a shirt or the sharpness of a face) might start to get a little blurry. The lighting stays perfect, but the "crispness" fades slightly the further you go down the conveyor belt.

Summary

HorizonRelight is a new way to change the lighting in long videos. Instead of treating the video as a stack of separate, disconnected clips, it treats it as one continuous stream where the end of one clip helps paint the beginning of the next. This ensures the new lighting looks natural and steady from start to finish.

Technical Summary: HorizonRelight

Problem Statement

Modern video relighting, which enables controllable lighting edits from a single input video, relies heavily on video diffusion models. However, current state-of-the-art backbones (e.g., DiffusionRenderer, UniRelight) are trained on short, fixed-length clips due to the high memory cost of spatiotemporal self-attention in Diffusion Transformers (DiT). To process long-horizon videos, these systems employ a "chunked sliding-window" inference strategy.

The paper identifies a critical failure mode in this regime: temporal discontinuities at chunk boundaries. Because each chunk is processed largely independently, the model must re-infer latent causes (such as intrinsic scene attributes and appearance cues) for every window shift. This creates a train-test mismatch where the model is trained on short clips but deployed on long sequences, leading to visible flickering, appearance drift, and inconsistent lighting across chunk boundaries. As demonstrated in the paper, even under identical target illumination, independent chunk runs produce visibly different results, breaking long-horizon consistency.

Methodology

HorizonRelight reframes long-horizon relighting not as a series of independent short-clip generations, but as a temporally conditioned latent domain translation problem. The framework introduces two core mechanisms to enforce cross-chunk continuity:

1. Propagated Context via Masked Target-Domain Self-Conditioning

The authors propose a training and inference strategy where the model learns to continue a relit sequence from partial evidence rather than starting from scratch.

Training: The model is trained with masked target-domain self-conditioning. During training, a subset of the target-domain latent frames (the "relit" output) is masked out. The model is conditioned on the source video (input) and the unmasked portion of the target domain, learning to predict the missing frames. This teaches the model to continue a sequence based on partial temporal context.
Inference (Chained Generation): At inference, the model processes video chunks sequentially. For chunk $k$ , the ending latent frames from the prediction of chunk $k-1$ are propagated and used as the "observed" anchor for the beginning of chunk $k$ . This effectively chains the generation, ensuring the relit state is continued rather than re-inferred, thereby minimizing boundary seams.

2. Warm-Start Prompting

To initialize the chain for the very first chunk (where no previous propagated context exists), the authors introduce warm-start prompting.

Instead of a "cold start" (random noise or purely source-conditioned generation), the first chunk is initialized with a relit prompt anchor.
This anchor can be a single image or a short video generated by an external controllable generative model (e.g., Nano Banana Pro, ChatGPT) that depicts the input scene under the target illumination.
This anchor establishes the initial target-domain state (appearance and lighting) for the entire sequence. The framework supports flexible initialization modes, including image-based, video-based, or cold-start, though image-based prompting with a well-aligned anchor yields the most stable results.

Architecture

The system follows a two-stage pipeline adapted from DiffusionRenderer:

Inverse Stage: Predicts editable scene attributes (G-buffers: basecolor, normal, depth, roughness, metallic) from the input video.
Forward Stage: Synthesizes the relit RGB video conditioned on the predicted G-buffers and the target lighting condition (HDRI).
Both stages utilize the proposed masked self-conditioning and context propagation. The forward stage specifically uses the propagated relit latent from the previous chunk to condition the current chunk's generation.

Key Contributions

Reframing Long-Horizon Relighting: The paper shifts the paradigm from independent chunk processing to a chained, temporally conditioned latent translation process.
Masked Target-Domain Self-Conditioning: A novel training objective that enables the model to learn to continue relighting from partial propagated context, directly addressing the train-test mismatch in sliding-window inference.
Warm-Start Prompting: A principled initialization method using external generative models to set the target-domain state, providing a general interface for prompt-based relighting and style transfer.
Unified Framework: The approach supports both inverse decomposition (G-buffer prediction) and forward relighting with consistent temporal propagation across both stages.

Results

The authors evaluate HorizonRelight on both synthetic datasets (procedurally rendered from Objaverse) and in-the-wild YouTube Creative Commons videos.

Temporal Consistency: Qualitative and quantitative results show a marked improvement in temporal consistency.
- Boundary Artifacts: The method largely reduces chunk-boundary artifacts. Difference maps between adjacent chunks show that changes are concentrated only around true scene dynamics (motion), whereas baselines (DiffusionRenderer, UniRelight) exhibit broad, unwanted appearance changes in static regions.
- Quantitative Metrics: On a synthetic dataset with a controlled "duplicate-reverse" setup, HorizonRelight achieves significantly lower Mean Squared Error (MSE) at chunk boundaries compared to baselines. For instance, in the relighting task, the boundary MSE dropped from ~0.1048 (DiffusionRenderer) to ~0.0729 (Ours). On 100 in-the-wild YouTube clips, the method reduced boundary MSE by over 94% compared to UniRelight and DiffusionRenderer.
Generality: The framework supports various initialization modes (cold start, video prompting, image prompting). While cold starts are feasible, warm-start prompting with a high-quality anchor significantly improves the preservation of specular details and lighting cues.
Extended Applications: The warm-start mechanism is shown to support long-horizon style transfer and content editing, not just relighting, by propagating a prompted appearance throughout the sequence.

Limitations and Claims

The paper maintains a modest stance regarding its limitations:

Fine-Detail Degradation: While relighting consistency is maintained over long horizons, the paper notes that fine scene details gradually degrade after extremely long chains (e.g., beyond ~300 frames or five chunks). This is attributed to the learned appearance prior and data coverage limitations rather than a failure of the propagation mechanism itself.
Scope: The authors clarify that the method is designed for practical long-take relighting (typical film shots of 5–25 seconds) rather than unrestricted future generation. The chain is intended to be reset at shot changes in practical workflows.
Not Autoregressive Generation: The paper distinguishes its approach from autoregressive video generation. HorizonRelight does not roll out generated content; it re-controls each chunk using the source video and G-buffers, using the previous relit output only as a boundary color cue to stabilize appearance.

Significance

HorizonRelight addresses a fundamental bottleneck in applying video diffusion models to long-horizon tasks: the trade-off between computational constraints (fixed windows) and temporal consistency. By treating long-horizon relighting as a domain translation problem with learned continuation, the work enables the generation of temporally consistent, high-quality relit videos from single inputs without requiring retraining of the backbone on long sequences. The introduction of warm-start prompting further bridges the gap between controllable generative models and video editing pipelines, offering a flexible interface for prompt-based visual effects.