UniLight: A Unified Representation for Lighting

The paper introduces UniLight, a unified latent space representation that aligns diverse lighting modalities—including text, images, irradiance, and environment maps—through contrastive learning and spherical harmonics prediction, enabling consistent cross-modal transfer and flexible lighting control in image synthesis tasks.

The Gist

Imagine you are a director trying to set the mood for a movie scene. You have four different ways to tell your lighting crew what you want:

1. A Text Script: "Make it look like a sunny afternoon with the sun high in the sky."
2. A Reference Photo: A picture of a room with a specific lamp.
3. A 360-Degree Panorama: A full wrap-around photo of the sky and surroundings.
4. A Scientific Map: A complex chart showing exactly how much light hits every square inch of the floor.

The Problem:
In the world of computer graphics, these four "languages" don't speak to each other. If you give the computer a text description, it can't easily use a reference photo to help. If you have a 360-degree panorama, the computer can't easily turn that into a text description. They are like four people speaking different languages in a room; they can't collaborate, which makes creating or changing lighting in digital images very rigid and difficult.

The Solution: UniLight (The Universal Translator)
The researchers behind this paper, UniLight, built a "Universal Translator" for light.

Think of UniLight as a giant, shared filing cabinet (a "joint latent space") where every type of lighting information gets translated into the same secret code.

• The Process: They built special "translators" (encoders) for text, photos, maps, and charts. When you feed any of these into the system, it converts them into a single, standardized "light fingerprint."
• The Magic: Because everything is now in the same fingerprint format, the computer can instantly understand that "sunny afternoon text" and "a photo of a sunny room" are actually the same thing.

How They Trained It:
To make sure this fingerprint was accurate, they used two clever tricks:

1. The "Match the Pair" Game (Contrastive Learning): They showed the computer thousands of examples where a text description, a photo, and a map all described the same lighting. The computer learned to push these different inputs closer together in the filing cabinet so they look identical in the secret code.
2. The "Compass Check" (Spherical Harmonics): To make sure the computer didn't just memorize the color of the light but also the direction (e.g., knowing the sun is on the right, not the left), they added a math test. The system had to predict a simple mathematical compass reading (Spherical Harmonics) from the fingerprint. If it got the direction wrong, it failed the test. This forced the system to really understand where the light is coming from.

What Can You Do With It?
Once the system is trained, it unlocks three superpowers:

1. Lighting Search Engine: You can upload a photo of a cozy living room, and the system can instantly find a text description that matches it, or a 360-degree sky map that creates the exact same vibe. It's like Shazam, but for lighting.
2. Lighting Generator: You can type "scary dark cave with a flickering torch," and the system can generate a perfect 360-degree lighting map for a 3D scene, or even generate a whole new image with that lighting.
3. The "Magic Wand" for Relighting: This is the coolest part. Imagine you have a photo of a person in a dark room. With UniLight, you can tell the computer, "Move the light source to the left and make it warmer." The computer understands the direction and quality of the light, so it can realistically move the shadows and highlights on the person's face without breaking the image. It's like changing the time of day on a photo with a single click.

In Summary:
Before this, lighting was like having four different remote controls for a TV, and none of them worked with the same buttons. UniLight replaced them with one universal remote that understands every command, whether you speak in words, pictures, or maps. It makes digital lighting flexible, searchable, and easy to control, bridging the gap between human imagination and computer graphics.

Technical Summary

1. Problem Statement

Lighting is a fundamental determinant of visual appearance in images, yet representing and controlling it remains a significant challenge in computer vision and graphics. Existing lighting representations are fragmented and mutually incompatible:

• Environment Maps: Capture full 360° illumination but are high-dimensional and difficult to edit semantically.
• Irradiance Maps: Useful for shading but lack high-frequency directional details.
• Spherical Harmonics (SH): Compact and directional but often too low-frequency for complex scenes.
• Text Descriptions: Semantically rich but lack precise geometric or physical grounding.

Current methods are typically designed around a single modality, making cross-modal transfer (e.g., using a text prompt to relight an image based on an environment map) impossible. There is a lack of a unified latent space that can bridge these disparate modalities to enable flexible lighting control, retrieval, and generation.

2. Methodology

The authors propose UniLight, a joint latent space that unifies multiple lighting modalities into a shared embedding. The framework consists of three main components:

A. Multi-Modal Data Pipeline

To train the model, the authors constructed a dataset of 72,180 multi-modal samples derived from 8,020 high-dynamic-range (HDR) environment maps (indoor and outdoor). For each environment map, they generated:

1. Environment Map: The original 360° HDR map.
2. Image: Perspective crops (512×512, 90° FOV) extracted from the map.
3. Irradiance Map: Estimated using Prism, an intrinsic decomposition method.
4. Text Description: Generated by a Vision-Language Model (VLM, InternVL3-38B). The VLM is prompted with the image and the location of dominant light sources (detected via connected components in the HDR map) to produce precise directional descriptions.
5. Estimated Environment Map: Generated from the image using DiffusionLight-Turbo to ensure robustness.

B. Architecture

The model employs modality-specific encoders aligned into a shared latent space:

• Image-based Encoders: Uses DINOv2-B backbones for environment maps, images, and irradiance maps.
  • Environment Map Input: Processed as Low-Dynamic-Range (LDR) via Reinhard tone-mapping, Logarithmic encoding, and explicit directional coordinates ($x,y,z$).
  • Robustness: Random dropout is applied to log channels to handle LDR-only inputs.
• Text Encoder: Uses Qwen3 Embedding (0.6B parameters), fine-tuned to encode lighting semantics, dominant light positions, and color temperature.
• Fusion Module: A lightweight transformer-based module uses learnable query tokens (default $T=8$) to attend to backbone features. These tokens are projected into a shared latent space ($D=512$).
• Spherical Harmonics (SH) Head: An auxiliary prediction head estimates SH coefficients (up to degree 3) directly from the latent embedding. This provides explicit supervision on light directionality.

C. Training Objective

The model is trained using a combination of two losses:

1. Contrastive Loss ($L_C$): A cross-entropy loss that maximizes the cosine similarity between embeddings of different modalities describing the same lighting condition (e.g., matching an image to its corresponding text description) while pushing apart non-matching pairs.
2. SH Supervision Loss ($L_{SH}$): A Mean Squared Error (MSE) loss between the predicted SH coefficients from the latent space and the ground-truth SH coefficients extracted from the environment map. This ensures the latent space captures accurate directional lighting structure.

3. Key Contributions

1. Unified Lighting Representation: The first joint latent space that aligns text, images, irradiance maps, and environment maps, enabling cross-modal transfer.
2. Multi-Modal Data Pipeline: A novel pipeline for generating aligned training data involving VLMs and intrinsic decomposition, supporting large-scale training.
3. Directional Encoding via SH: The integration of an auxiliary SH prediction task significantly improves the model's ability to encode light direction, which is often lost in standard contrastive learning.
4. Versatile Downstream Applications: The framework supports three distinct tasks:
   • Cross-modal lighting retrieval.
   • Environment map generation from any supported modality.
   • Diffusion-based image relighting with flexible lighting control.

4. Results and Evaluation

The authors evaluated UniLight on three primary tasks:

• Cross-Modal Retrieval:

  • UniLight significantly outperforms baselines like CLIP and Qwen3-VL in retrieving lighting matches across modalities.
  • Metrics: Achieved Recall@1 of 24.9% and Mean Reciprocal Rank (MRR) of 0.367 for Image↔Text retrieval, compared to CLIP's 2.6% and 0.077 respectively.
  • Ablation: Removing the SH supervision (NOSH) dropped Recall@1 to 10.2%, proving the critical role of directional supervision. Increasing token count beyond 8 yielded diminishing returns.

• Environment Map Generation:

  • Fine-tuned Stable Diffusion 3.5 to generate 360° LDR environment maps conditioned on UniLight embeddings.
  • Quality: Outperformed DiffusionLight-Turbo in quantitative metrics (PSNR: 28.85 vs. 27.77; SSIM: 0.915 vs. 0.902) and visual coherence.

• Image Relighting:

  • Integrated into an X→RGB framework (using intrinsics like depth, normals, albedo) to relight images.
  • Performance: UniLight allowed for precise control of shadows and highlights when rotating environment maps or changing text prompts. In contrast, baselines like Qwen3-VL often failed to update shadows dynamically, leaving them static despite lighting changes.

5. Significance

UniLight represents a major step forward in physically grounded generative AI. By unifying disparate lighting representations, it breaks the silos between text-to-image, image-to-image, and 3D rendering workflows.

• Flexibility: Users can now control lighting using their preferred modality (e.g., a rough sketch, a text prompt, or a reference photo) and apply it to any other modality.
• Precision: The inclusion of SH supervision ensures that the generated lighting is not just semantically correct but also geometrically and directionally accurate.
• Future Impact: This work lays the groundwork for more controllable generative models, enabling applications in virtual production, digital avatars, and automated scene editing where consistent and manipulable lighting is crucial.

Limitations: The current representation captures global light direction well but does not yet model complex spatial variations of light (local lighting changes) within a scene, which remains a challenge for complex indoor environments.