🤖 machine learning

Lighting-Aware Representation Learning under Controllable Lighting Variation

This paper proposes a lighting-aware representation learning framework that treats illumination variation as an explicit training signal rather than a nuisance factor, demonstrating improved robustness and downstream performance in image classification and object detection tasks compared to standard contrastive learning baselines.

Original authors: Lizhen Zhu, Charantej Reddy Pochimireddy, James Z Wang, Brad Wyble

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lizhen Zhu, Charantej Reddy Pochimireddy, James Z Wang, Brad Wyble

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Why Computers Get Confused by Light

Imagine you are looking at a red apple. If you take a photo of it in bright noon sunlight, it looks bright and vibrant. If you take the same photo at sunset, it looks orange and warm. If you take it under a blue streetlamp, it looks purple.

To a human, we instantly know it's the same apple in all three photos. Our brains are amazing at figuring out the "real" object despite the changing light.

But for a computer vision system (the "brain" inside a robot or a self-driving car), this is a nightmare. To the computer, those three photos look like three completely different objects because the colors and shadows are so different. Most current AI tries to solve this by pretending the light doesn't exist. It tries to "wash out" the lighting differences so the computer only sees the object. The authors of this paper argue that this is the wrong approach. They say, "Why ignore the light? Let's teach the computer to understand it."

The Solution: A Two-Headed Brain

The researchers built a new way to train AI called a "Lighting-Aware Representation Learning Framework."

Think of a standard AI model as a student who is told to study a book but is constantly distracted by a flickering lamp. The student tries to ignore the lamp and just memorize the words.

The new method gives the student two heads (or two ways of thinking) to learn from the same book:

The "What" Head (Content): This head focuses on identifying the object. Is it a chair? A dog? A car? It learns to ignore the light and focus on the shape and identity.
The "How" Head (Lighting): This head focuses on the environment. Is it sunny? Is it dark? Is the light coming from the left or right? It learns to pay attention to the shadows and colors caused by the light.

By training both heads at the same time, the AI learns a much richer understanding of the picture. It doesn't just know "that's a dog"; it knows "that's a dog in the rain."

How They Did It: The Virtual Photo Studio

To teach the AI this, they couldn't just use regular photos because it's hard to know exactly what the light was doing in a real photo.

Instead, they used a virtual reality simulator (a digital 3D world). Imagine a digital camera taking pictures of a virtual room.

They kept the furniture and objects exactly the same.
They changed the "sky" and the "sun" in the simulation to create 10 different lighting conditions (bright day, cloudy, sunset, etc.) for every single scene.

This gave them perfect pairs of photos: identical objects, but with totally different lighting. They used these pairs to train the AI's two heads.

The Results: Smarter and Stronger

The researchers tested this new "two-headed" AI on standard tasks like identifying objects in photos (ImageNet) and finding objects in low-light conditions (ExDark).

Better than the old way: The new AI consistently performed better than the standard AI that tries to ignore the light.
The "Joint" Secret: They found that the best results came when the two heads were allowed to share a little bit of information. Sometimes, the light and the object interact (like a shadow cast by a specific shape). The AI learned that some information belongs to both the object and the light, and it kept a special "shared" section in its brain for that.
Low Light: The new AI was particularly good at recognizing things in the dark or in weird lighting, proving that understanding the light helps you see the object better.

What They Didn't Do (Important Limits)

The paper is very specific about what this research does and doesn't do:

It's not about fixing dark photos: They didn't build a tool to make a dark photo look bright (like a photo editor app). They built a brain that understands the darkness.
Simple filters aren't enough: They tried to trick the AI by just making photos brighter or darker using simple computer filters (like adjusting brightness on your phone). This didn't work as well as the virtual 3D simulation. Real light bounces off 3D objects in complex ways that simple filters can't copy.
No medical or clinical claims: The paper does not claim this will help doctors diagnose diseases or improve medical imaging. It is strictly about general computer vision and object recognition.

The Takeaway

The main idea is simple: Don't treat light as a nuisance to be ignored; treat it as a clue to be understood.

By teaching AI to explicitly learn about lighting conditions alongside the objects themselves, the computer becomes more robust and adaptable, much like a human who can recognize a friend whether they are standing in the sun, the shade, or under a neon sign. The authors suggest this "factor-aware" approach could eventually be used to teach AI about other changing factors, like different weather or viewing angles, not just light.

Technical Summary: Lighting-Aware Representation Learning under Controllable Lighting Variation

Problem Statement

Variations in illumination constitute a fundamental challenge in computer vision, inducing substantial appearance changes that compromise the robustness and generalization of visual representation learning. While human vision achieves perceptual constancy by estimating illuminants and constructing internal models of lighting conditions, standard artificial vision systems typically treat illumination as a nuisance factor. Existing approaches primarily rely on data augmentation (e.g., brightness, contrast, hue adjustments) to encourage models to become invariant to lighting changes. However, these strategies suppress lighting information rather than explicitly modeling it, often failing to match the adaptability of human vision or to capture the complex interactions between geometry, materials, and light that define real-world visual signals.

Methodology

The authors propose a Lighting-Aware Representation Learning Framework that incorporates illumination variation as an explicit training signal rather than noise. The core methodology extends contrastive learning through a Dual-Head Training architecture.

1. Theoretical Foundation

The approach is grounded in the premise that an observed image $I$ is generated from two underlying factors: intrinsic scene content $C$ (geometry, layout, semantics) and illumination condition $L$ . The goal is to learn representations that preserve semantic consistency across lighting changes while remaining sensitive to lighting-dependent visual structure.

2. Architecture and Training Objective

The framework utilizes a shared ResNet-50 backbone that branches after global average pooling into two distinct heads:

Content Head (Contrastive): Enforces consistency for images sharing the same semantic content but differing in illumination. This utilizes standard contrastive learning objectives (based on MoCo V2 and ESS-MB) to group semantically similar images.
Lighting Head (Contrastive): Enforces consistency for images sharing the same illumination configuration (e.g., the same "skybox" in a virtual environment) but differing in content. This head learns to cluster representations based on lighting conditions.

The total loss function is a weighted sum of the content contrastive loss ( $L_{Con.}$ ) and the lighting contrastive loss ( $L_{Light}$ ):
$L = L_{Con.} + \alpha L_{Light}$
where $\alpha$ balances the contributions of the two objectives.

3. Representation Factorization

To address the statistical dependencies between lighting and content (e.g., shadows and reflections), the authors introduce a flexible factorization of the representation space into three components:

Lighting Component: Captures features primarily determined by illumination.
Content Component: Encodes scene-intrinsic information invariant to lighting.
Joint Component: Captures interactions between lighting and content that cannot be cleanly separated.

The allocation of representation dimensions is denoted by a ratio $(L:J:C)$ . For instance, $(1:0:7)$ allocates 1/8th of the dimension to lighting, none to joint, and 7/8ths to content.

4. Data and Experimental Setup

The method was evaluated using the House100KLighting dataset, collected from a virtual ray-traced environment (ThreeDWorld), which allows for precise manipulation of lighting conditions while preserving scene content. Baselines included standard contrastive learning (MoCo V2/ESS-MB) and single-representation models trained with both objectives. Downstream tasks included image classification (ImageNet), object and lighting classification (ExDark), and object detection (PASCAL VOC).

Key Results

Superior Downstream Performance: Models trained with the dual-head lighting-aware framework (specifically configuration $1:0:7$ ) consistently outperformed standard contrastive learning baselines on ImageNet classification, ExDark object/lighting classification, and PASCAL VOC detection. The combined representation (concatenating content and lighting features) yielded the best results.
Complementary Representations: The content head dominated performance on standard object classification (ImageNet), while the lighting head contributed significantly to performance on low-light datasets (ExDark), sometimes surpassing the content head. This confirms that the two heads capture complementary information.
Importance of Joint Components: Configurations including a joint component (e.g., $1:1:6$ ) showed even greater performance gains, suggesting that retaining limited interaction between semantic and illumination features benefits downstream recognition.
Limitations of Simple Augmentation: Experiments using simple image-level transformations (brightness, gamma, etc.) to approximate lighting variation (House100K-AugLighting) resulted in only marginal improvements over standard contrastive learning. This indicates that physically based ray-traced lighting provides complex interactions that simple transformations cannot replicate.
Supervised Learning Extension: The framework was successfully adapted to a supervised setting on ImageNet100K-AugLighting, where the dual-head approach achieved improved test accuracy despite using reduced representation dimensions for each specific task, demonstrating enhanced generalization through multi-factor supervision.
Layer-wise Analysis: Probing tasks revealed that illumination-aware representations are distributed throughout the network but become less sensitive in the final layers under standard contrastive training, which tends to conflate content and lighting information. The dual-head approach better preserves illumination-related structure across layers.

Significance and Claims

The paper claims that illumination should not be treated solely as a source of noise to be suppressed via augmentation. Instead, explicitly modeling lighting as a controllable factor during representation learning yields more robust and adaptable visual representations.

The authors position this work as a case study for a broader "factor-aware representation learning" paradigm. By leveraging controlled environmental variations (such as lighting, viewpoint, or weather) as explicit supervision signals, models can learn to disentangle semantic content from environmental conditions while capturing their interactions. The results suggest that this approach enhances model robustness in complex visual environments and improves performance in conventional image processing tasks, offering a pathway toward more human-like perceptual constancy in artificial systems.

The paper modestly acknowledges limitations, noting that the internal propagation of signals requires further investigation, that experiments were conducted on relatively small datasets for tractability, and that simple image transformations are insufficient proxies for realistic physical lighting. Future work is suggested to extend this framework to other controllable factors and to explore efficient methods for generating realistic training datasets using generative models.