💻 computer science

LumiCtrl : Learning Illuminant Prompts for Lighting Control in Personalized Text-to-Image Models

LumiCtrl is a novel method for personalizing text-to-image models that enables precise lighting control by learning illuminant prompts from a single object image through physics-based augmentation, edge-guided prompt disentanglement, and masked reconstruction loss, resulting in superior illuminant fidelity and aesthetic quality compared to existing baselines.

Original authors: Muhammad Atif Butt, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammad Atif Butt, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical art teacher (an AI) who is incredibly good at drawing whatever you describe. You can say, "Draw a golden retriever," and it will do so. But there's a problem: this teacher is a bit stubborn about the lighting.

If you show the teacher a photo of a dog in a warm, cozy living room and ask it to draw that same dog in "cold, blue moonlight," the teacher often ignores your request. It keeps drawing the dog in that same cozy, warm living room light, no matter what you say. It's like the teacher has forgotten how to turn off the living room lamp and turn on the moon.

This paper introduces LumiCtrl, a new method that teaches this magical art teacher how to actually listen to your instructions about lighting.

Here is how LumiCtrl works, explained through simple analogies:

1. The Problem: The Teacher's "Language Gap"

The authors discovered that the AI's "brain" (specifically its text encoder) doesn't really understand words like "Tungsten," "6500K," or "Shade."

The Analogy: Imagine the AI thinks the word "Tungsten" is just a random number, like "42," rather than a type of warm light. It doesn't connect the word to the feeling of warm light. So, when you ask for "Tungsten light," the AI just draws its default "Daylight" picture because it doesn't know what else to do.

2. The Solution: LumiCtrl's Three-Step Magic Trick

To fix this, LumiCtrl uses three clever tricks to retrain the AI without breaking its ability to draw the object correctly.

Step A: The "Physics Lab" (Temperature Mapping)

Before teaching the AI new words, they need to show it what those words look like.

The Analogy: Imagine you want to teach someone what "Spicy" tastes like. You can't just say the word; you have to give them a pepper. LumiCtrl takes your image of the object and uses physics to mathematically "paint" it with different lights (like turning a photo into a sunset scene or a fluorescent office scene). This creates a library of examples so the AI can see what "Tungsten" actually looks like on that specific object.

Step B: The "Structural Glasses" (Edge-Guided Disentanglement)

This is the most important trick. When the AI learns a new word, it often gets confused and mixes up the object with the light.

The Analogy: Imagine you are teaching a student to recognize the color "Red." If you show them a red fire truck, they might learn that "Red" means "has wheels and a ladder." That's bad!
LumiCtrl puts on a pair of special glasses called ControlNet. These glasses only look at the edges and shapes of the object (the outline of the dog), ignoring the colors. This forces the AI to say: "Okay, I see the shape of the dog is the same. The only thing changing is the color of the light. I will only learn the word for the light, not the shape of the dog."

Step C: The "Spotlight Mask" (Masked Reconstruction Loss)

When you change the light on an object, the background usually changes too, but not in a simple, flat way. Shadows and reflections are complex.

The Analogy: Imagine you are painting a portrait. If you just paint the whole canvas with a new color, the background looks fake and flat. LumiCtrl puts a mask (a stencil) over the object.
- It tells the AI: "Focus 100% of your energy on getting the light right on the dog."
- It tells the AI: "For the background, just use your own imagination to make it look natural."
  This allows the dog to look perfectly lit while the background adapts naturally, creating a realistic scene rather than a flat, painted-over mess.

3. The Result

By using these three steps, LumiCtrl teaches the AI to understand that "Tungsten" means "warm yellow light" and "Shade" means "cool blue light."

Before LumiCtrl: You ask for a dog in moonlight, and you get a dog in sunlight.
With LumiCtrl: You ask for a dog in moonlight, and you get a dog in moonlight, with the background looking like a cool, dark night, while the dog still looks exactly like the dog you showed it.

Why This Matters

Currently, if you want to change the lighting in a photo, you need to use complex software like Photoshop or specialized 3D tools. LumiCtrl allows you to do this just by typing a sentence. It bridges the gap between what we say ("Make it look like sunset") and what the computer actually draws, giving artists and designers total control over the mood and atmosphere of their images without needing to be lighting experts.

1. Problem Statement

While Text-to-Image (T2I) diffusion models have achieved remarkable success in generating realistic images, they lack precise control over scene illuminants (lighting conditions).

The Semantic Gap: The authors identify a fundamental limitation in current T2I models: standard illuminant terms (e.g., "tungsten," "6500K," "cloudy") are not semantically grounded in the text encoder's embedding space.
- Embeddings for Kelvin temperatures cluster with generic numerals rather than lighting concepts.
- Embeddings for named presets (e.g., "tungsten") do not cluster with general lighting terms (e.g., "warm light").
Consequence: When users attempt to prompt for specific lighting, the model fails to interpret the instruction, defaulting to a daylight prior or entangling the lighting with the object's identity learned during personalization.
Limitations of Existing Solutions:
- Image-Space Illumination Control (ISIC): Post-hoc methods (e.g., IC-Light) require separate relighting models and often fail to preserve spatial background or introduce artifacts.
- Existing Personalization: Methods like DreamBooth or Textual Inversion entangle lighting information from training images, making it impossible to generate the same object under different text-guided illuminants.

2. Methodology: LumiCtrl

LumiCtrl is the first Prompt-Space Illumination Modeling (PSIM) method. It learns specific text tokens (prompts) for illuminants, allowing users to control lighting directly within the generation prompt while preserving object identity and background context. The method consists of three core components:

A. Physics-Based Illuminant Augmentation (Temperature Mapping)

To create training data for illuminant prompts, the authors apply Flat Light Adaptation to input images.

Process: They multiply image pixels by a diagonal matrix representing the desired illuminant color.
Illuminants: They utilize 7 illuminants based on the Planckian locus: 4 traditional presets (Tungsten, Fluorescent, Cloudy, Shade) and 3 intermediate temperatures.
Goal: This creates fine-tuning variants of the object under standard illuminants to serve as the ground truth for learning.

B. Edge-Guided Prompt Disentanglement

A major challenge in prompt learning is "spurious detail leakage," where the model learns structural features of the training image instead of just the color/lighting.

Solution: The authors employ a frozen ControlNet conditioned on Canny edge maps during training.
Mechanism: The ControlNet provides structural guidance (edges), forcing the prompt optimization to focus only on color and illumination changes. The structural information is already encoded by the ControlNet, preventing the learned illuminant tokens from altering the object's shape or introducing artifacts.
Note: ControlNet is discarded during inference.

C. Masked Reconstruction Loss (Contextual Light Adaptation)

Flat light adaptation applies a uniform color shift, which is unrealistic because real-world lighting interacts differently with foreground objects and backgrounds (shadows, reflections).

Solution: A Masked Reconstruction Loss ( $L_{mrl}$ ) is introduced.
Mechanism: The loss function is weighted based on a user-provided mask:
- Foreground: High weight ( $\lambda$ ) to ensure the object learns the specific illuminant color accurately.
- Background: Lower weight ( $1-\lambda$ ) to allow the background to adapt contextually based on the pretrained model's prior knowledge.
Result: This enables Contextual Light Adaptation, where the background lighting shifts naturally to complement the foreground, creating a coherent scene.

3. Key Contributions

Discovery of Semantic Gap: The paper empirically demonstrates that T2I text encoders treat Kelvin temperatures as generic numbers and fail to associate named illuminants with their physical lighting properties, explaining why naive prompting fails.
Illuminant Prompt Learning: LumiCtrl is the first method to learn specific text tokens for illuminants, enabling precise lighting control within the generative prompt itself.
Technical Innovations:
- Edge-Guided Disentanglement: Uses frozen ControlNet to isolate lighting learning from structural learning.
- Masked Reconstruction Loss: Enables realistic, context-aware background adaptation while focusing learning on the foreground object.
Comprehensive Evaluation: The method is validated against both T2I personalization baselines and image editing methods using quantitative metrics and human preference studies.

4. Results

The authors evaluated LumiCtrl on 20 concepts (humans, pets, objects) across 7 illuminants.

Quantitative Metrics:
- Illuminant Accuracy: LumiCtrl achieved the lowest Mean Angular Error (4.51°) and Mean Squared Error (16.80) compared to all baselines (e.g., DreamBooth: 12.76°, IC-Light: 10.39°), indicating superior chromaticity and intensity rendering.
- Image Fidelity: It achieved the highest SSIM (0.77), demonstrating better preservation of structural details.
- Ablation: Removing ControlNet increased MSE significantly, confirming its role in structural integrity. Removing the Masked Loss caused divergent lighting sources.
Qualitative Analysis:
- LumiCtrl successfully generated objects under specific text-guided illuminants while maintaining identity and texture.
- Unlike personalization methods (which retained training lighting) or editing methods (which distorted backgrounds), LumiCtrl produced photo-realistic scenes with natural background adaptation.
Human Preference Study:
- In a 2-Alternative Forced Choice (2AFC) study with 15 participants, LumiCtrl significantly outperformed all baselines, with higher z-scores indicating a strong user preference for its realistic lighting and coherence.

5. Significance

Paradigm Shift: LumiCtrl moves lighting control from the "Image Space" (post-processing) to the "Prompt Space" (generative control), allowing for more flexible and integrated lighting manipulation.
Design Utility: It addresses a critical need for content designers who require precise control over mood and atmosphere (e.g., switching from "daylight" to "tungsten") without losing the specific object they are designing.
Foundation for Future Work: By identifying the semantic gap in text encoders, the paper opens avenues for improving the grounding of photometric concepts in large language models and diffusion pipelines.

In conclusion, LumiCtrl provides a robust framework for personalized lighting control, solving the entanglement of object identity and scene illumination through physics-based augmentation, structural disentanglement, and context-aware loss functions.