LightMover: Generative Light Movement with Color and Intensity Controls

1. Problem Statement

The paper addresses the challenge of controllable light manipulation in single images. While users often wish to edit lighting in a photo (e.g., moving a lamp, changing its color temperature, or dimming its brightness) to alter the mood and shadows, existing methods fall short:

• Inverse Rendering: Physically grounded but computationally expensive, ill-posed from a single image, and lacks intuitive interactive controls.
• Diffusion-based Editors: Can adjust tone or ambient light but fail to model spatial light movement (shifting a light source and updating shadows/reflections accordingly).
• Object-Centric Models: Can move objects but treat illumination implicitly, failing to propagate correct shadows, falloff, and color interactions when a light source moves.
• General Generative Models: Lack explicit, parametric light-centric control, leading to physically implausible results.

The core difficulty lies in the global nature of illumination: moving a light source requires simultaneous, physically consistent updates to shadows, reflections, specular highlights, and ambient falloff across the entire scene.

2. Methodology

LightMover proposes a video-diffusion framework that treats light editing as a sequence-to-sequence prediction problem in visual token space. It leverages video diffusion priors to reason about light transport without requiring explicit 3D reconstruction.

A. Sequence-to-Sequence Formulation

The model repurposes a pre-trained image-to-video diffusion transformer (similar to Sora). Inputs are encoded as a sequence of "pseudo-frames":

1. Reference Image ($I_{ref}$): The original input.
2. Object Frame ($I_{obj}$): The cropped light source to be manipulated.
3. Movement Map ($I_{move}$): A spatial control signal encoding source and target bounding boxes (using RGB channels to distinguish start/end regions).
4. Color Control ($I_{color}$): A global frame specifying desired hue or color temperature.
5. Intensity Control ($I_{intensity}$): A frame encoding exposure changes in stops (photographic units), where $Gain = 2^{EV}$.
6. Output Frame ($X_t$): The noisy latent to be denoised.

B. Multi-Signal Positional Encoding (MSPE)

To ensure the transformer correctly interprets the role of each token, the authors introduce a specialized positional encoding with four orthogonal subspaces:

• Spatial (W, H): Preserves 2D structure within frames.
• Temporal (T): Maintains sequence order.
• Condition-Type (C): Distinguishes between reference, object, movement, color, and intensity inputs.
• Frame-Role (R): Differentiates conditioning inputs from the prediction target.
  This allows the model to jointly reason over spatial alignment and conditional dependencies.

C. Adaptive Token Pruning

Adding control frames increases sequence length, hurting efficiency. LightMover introduces an adaptive pruning mechanism:

• Spatially-Aware Pruning: For movement maps, if the target area is small, the full resolution is kept; otherwise, tokens are downsampled based on the area ratio to reduce background redundancy.
• Learnable Downsampling: For non-spatial attributes (color, intensity), the model learns optimal compression ratios.
• Result: Reduces control sequence length by 41% while maintaining editing fidelity.

D. Data Generation Pipeline

To train the model, the authors constructed a scalable pipeline combining real and synthetic data:

• Synthetic Data: Uses Blender to render 25 artist-designed indoor scenes with 100+ light assets. It employs physically disentangled rendering, separating ambient base ($I_{amb}$) and direct light ($I_{light}$) components. This allows for parametric control of light position, spectrum, and intensity during post-processing.
• Real Data: Captured using mobile devices with synchronized triggering to create pairs of images where only the light source location changes.
• Multi-Task Training: The model is trained jointly on light movement, color change, intensity change, insertion, and removal. This multi-task approach acts as a regularizer, improving stability and accuracy across all tasks.

3. Key Contributions

1. Unified Framework: A single diffusion-based model that handles both object movement and explicit, parametric light control (position, color, intensity) within a 2D generative framework.
2. Adaptive Token Pruning: A novel mechanism that significantly improves computational efficiency by compactly encoding non-spatial attributes while preserving spatial precision for localized edits.
3. Scalable Data Pipeline: A physically accurate rendering pipeline that generates massive paired datasets with disentangled lighting components, enabling the learning of causal illumination effects (shadows, reflections) that pure video priors cannot provide.
4. 2.5D Paradigm: Demonstrates that "2.5D" learning (using video priors on single images) can approximate complex global light transport more efficiently than full inverse rendering.

4. Results

The model was evaluated on LightMove-A (real-world data) and LightMove-B (synthetic ground-truth data).

• Quantitative Performance:

  • Light Movement: Outperforms state-of-the-art (SOTA) baselines including Qwen-Image, Gemini-2.5-Flash-Image, and ObjectMover.
    • PSNR: 20.38 (vs. 19.59 for Gemini and 19.49 for ObjectMover).
    • DINO: 81.27 (vs. 72.46 for Gemini).
    • CLIP: 91.85 (vs. 89.72 for Gemini).
  • Color/Intensity Control: Consistently achieves higher PSNR, DINO, and CLIP scores across single-attribute and compositional (movement + color/intensity) tasks.
  • Object Movement: Maintains strong performance on general object movement tasks, outperforming baselines like MagicFixup and 3DiT, proving the model's generalization beyond just light sources.

• Qualitative Performance:

  • Successfully generates physically plausible shadows, reflections, and specular highlights that shift correctly with the light source.
  • Handles complex material interactions (e.g., metallic vs. matte surfaces) and occlusion reasoning.
  • The adaptive pruning mechanism achieves comparable accuracy to the full sequence while being significantly more efficient.

5. Significance

LightMover represents a significant step forward in physically aware image editing.

• Bridging the Gap: It bridges the gap between computationally heavy 3D inverse rendering and purely 2D generative models, achieving physical coherence without explicit 3D reconstruction.
• User Control: It provides intuitive, parametric controls (drag-and-drop movement, color wheels, exposure sliders) that were previously impossible in single-image editing.
• Efficiency: The adaptive token pruning and multi-task training strategy offer a blueprint for efficient, controllable generative models.
• Future Impact: This work opens new directions for scalable, physically consistent editing in photography, virtual reality, and content creation, moving beyond semantic edits to precise physical manipulation of the scene's illumination.