💻 computer science

Lighting-aware Unified Model for Instance Segmentation

This paper introduces Lighting Convolutional-Attention (LCA), a lightweight adapter module that enhances the illumination robustness of foundation models like SAM for instance segmentation by processing RGB features alongside contrast maps and optimizing via a pairwise training strategy, validated through extensive experiments on existing benchmarks and a novel Unity-based synthetic dataset.

Original authors: Qisai Liu, Alloy Das, Zhanhong Jiang, Joshua R. Waite, Aditya Balu, Adarsh Krishnamurthy, Soumik Sarkar

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qisai Liu, Alloy Das, Zhanhong Jiang, Joshua R. Waite, Aditya Balu, Adarsh Krishnamurthy, Soumik Sarkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant named SAM (Segment Anything Model). SAM is incredible at looking at a photo and drawing a perfect outline around any object it sees—like a cat, a car, or a tree. It learned this skill by studying billions of photos in a perfectly lit, sunny studio.

But here's the problem: SAM gets confused in the real world.

If you take SAM out at night, into a dark warehouse, or under a harsh, overexposed sun, it starts to hallucinate. It might draw a giant, fuzzy blob over the whole background because the shadows look like objects, or it might miss a car entirely because the lighting is too dim. It's like a person who has only ever seen the world in a bright, white room suddenly being asked to navigate a dark, stormy forest; they lose their sense of direction.

This paper introduces a solution called PLAP-LCA. Think of it not as rebuilding the robot, but as giving it a pair of special "lighting-proof" glasses and a new training regimen.

Here is how it works, broken down into simple parts:

1. The "Lighting Glasses" (The LCA Module)

The researchers didn't want to re-teach the whole robot (which would take forever and cost a lot of money). Instead, they added a small, lightweight attachment called LCA (Lighting Convolutional-Attention) to the robot's brain.

Imagine the robot's brain is looking at a picture and seeing three things at once:

The Object: The actual shape of the car or person.
The Light: The brightness and color of the sun or lamp.
The Edges: The sharp lines where one thing ends and another begins.

In bad lighting, the "Light" part gets too loud and drowns out the "Object" part. The LCA module acts like a noise-canceling headphone for vision. It has three specific filters:

The Channel Filter: It asks, "Is this color just a trick of the light?" and turns down the volume on those confusing colors.
The Spatial Filter: It asks, "Is this shadow just a patch of darkness?" and tells the robot to ignore that specific spot.
The Contrast Filter (The Secret Sauce): This is the most important one. It uses a mathematical trick (called a Laplacian filter) to look specifically for edges. Even if a car is in total darkness, its outline still exists as a change in texture. This filter tells the robot: "Ignore the dark background; focus only on the sharp lines where the object actually is."

2. The "Twin Training" (Pairwise Training)

How do you teach a robot to ignore the light? You don't just show it one picture. You show it twins.

The researchers created a system where the robot sees the same scene twice:

Version A: A bright, clear, sunny photo.
Version B: The exact same photo, but digitally tweaked to look like it's at night, in a fog, or under a weird colored light.

The robot is then given a strict rule: "Your answer must be the same for both pictures."

If the robot draws a perfect circle around a car in the sunny photo but draws a giant mess in the dark photo, it gets a "penalty." It has to learn to ignore the lighting differences and focus only on the shape of the car. This forces the robot to learn that the object stays the same, even if the light changes.

3. The "Virtual Reality Gym" (Unity Dataset)

To train this, the researchers needed millions of these "twin" photos. They couldn't just go out and take photos at night because that's slow and expensive.

Instead, they built a virtual reality world using a game engine called Unity. They built 3D rooms and placed furniture in them. Then, they programmed the virtual sun to move, the lights to flicker, and the weather to change.

They could take a photo of a chair in "Bright Daylight."
Then, with one click, they could take a photo of the exact same chair in "Severe Nighttime Darkness."
Because it's a computer simulation, the computer knows exactly where the chair is in both photos. This gave them perfect "twin" training data that is physically accurate but impossible to get easily in the real world.

The Result

When they put these "glasses" and this "training" onto the robot:

Before: In bad lighting, the robot was confused, drawing messy outlines or missing objects entirely.
After: The robot became lighting-robust. It could look at a dark, shadowy image and still draw a crisp, accurate outline around the object, just as well as it did in the sunny studio.

In short: The paper didn't invent a new robot; it invented a way to teach an existing super-robot to ignore the tricks of the light and focus on the true shapes of the world, using a special set of filters and a virtual reality gym to practice.

Technical Summary: Lighting-aware Unified Model for Instance Segmentation

Problem Statement

Foundation models like the Segment Anything Model (SAM) have demonstrated impressive zero-shot generalization across diverse visual domains. However, their performance frequently degrades in real-world environments characterized by adverse illumination, such as nighttime driving, low-light indoor scenes, overexposed outdoor settings, and mixed artificial/natural lighting. Existing research often treats lighting variation as a secondary concern, addressing it primarily through generic data augmentation or domain adaptation techniques that require full-model retraining or access to target-domain data. This creates a critical reliability gap for safety-critical applications (e.g., autonomous driving, medical imaging, agriculture) where lighting conditions are variable and unpredictable.

Methodology

The authors propose PLAP-LCA (Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention), a framework designed to equip SAM with robust lighting invariance without computationally prohibitive fine-tuning of its heavy backbone. The approach operates on three levels: data, architecture, and training.

1. Data Generation: Pairwise Lighting Augmentation

To train for lighting invariance, the model requires pairs of images depicting the same scene under different lighting conditions but sharing identical ground-truth masks. The authors employ two complementary strategies:

PLAP (Image-Space): A pipeline that applies controlled, physically motivated lighting operations (e.g., exposure, directional shadows, color temperature, vignetting, haze) to existing benchmark images (COCO, VOC, Cityscapes). It utilizes conflict-aware sampling to avoid contradictory operations (e.g., warm vs. cool) and a three-level severity protocol (Mild, Moderate, Severe) to simulate a spectrum of lighting degradation.
Unity-Lighting (Synthetic): A physically rendered synthetic dataset generated within the Unity engine. Unlike 2D image-space operations, this approach simulates actual 3D light transport, including reflections, inter-reflections, and complex interactions between natural and artificial light sources. It provides pixel-perfect instance segmentation masks across four lighting conditions: Clean, Mild, Moderate, and Severe.

2. Architecture: Lighting Convolutional Attention (LCA) Module

Instead of fine-tuning the frozen SAM ViT-B encoder, the authors inject a lightweight LCA adapter into the final two transformer blocks. The LCA module is designed to process structural edge information alongside RGB features to maintain sensitivity to object boundaries independent of illumination. It employs a dual-branch architecture with three parallel attention gates:

Channel Attention Gate: Recalibrates feature channels to counteract global illumination bias (e.g., brightness shifts, color temperature changes) using global average and max pooling statistics.
Spatial Attention Gate: Identifies locally affected regions (e.g., cast shadows, specular highlights) by condensing channel dimensions into spatial summary maps and processing them with a large 7×7 convolution kernel.
Contrast Attention Gate: Explicitly models the physical signature of lighting variation (edges) using a fixed Laplacian filter on a learned intensity projection. This gate reinforces degraded structural boundaries.

These gates are fused via element-wise multiplication and projected through a depthwise-separable convolution. Crucially, the residual projection is initialized to zero, ensuring the network begins training as an exact copy of the pre-trained SAM, preventing representation damage.

3. Training Strategy: Dual-Stream Pairwise Training

The framework utilizes a dual-stream setup where clean and variant images pass through the shared frozen encoder.

Shared Prompts: The same foreground points and bounding boxes are used as prompts for both streams to ensure prediction differences stem solely from illumination-induced feature variations.
Supervised Loss: A weighted blend of segmentation losses (BCE + Dice) for both clean and variant streams.
Consistency Loss: A novel feature consistency loss ( $L_{cons}$ ) that penalizes discrepancies between the sigmoid-activated outputs of the clean and variant streams. This loss enforces that the model produces identical predictions for the same scene regardless of lighting, without requiring additional annotations.
Curriculum-Based Weighting: The consistency loss weight is tuned to balance the supervised signal, preventing the model from being over-constrained during early training.

Key Contributions

LCA Module: A lightweight, dual-branch adapter that injects illumination-robust structural representations into foundation models without updating the pre-trained backbone weights.
Pairwise Training Strategy: A curriculum-based training approach utilizing a novel feature consistency loss to explicitly bridge the representational gap between ideal and adverse lighting conditions.
Comprehensive Empirical Study: A rigorous evaluation across three public benchmarks (COCO, VOC, Cityscapes) and a novel Unity-based synthetic dataset designed for physically accurate illumination evaluation.

Experimental Results

The approach was evaluated on COCO, VOC, Cityscapes, and the Unity synthetic dataset.

Performance: The LCA module (combined with decoder tuning) achieved superior mean Intersection over Union (mIoU) compared to the baseline SAM and decoder-only fine-tuning, particularly under lighting variant conditions. For instance, on the Cityscapes dataset, LCA+Dec achieved an mIoU of 0.784 under variant conditions, compared to 0.560 for the zero-shot SAM baseline.
Robustness: The method significantly reduced the performance gap between clean and variant images. In ablation studies, the LCA module alone improved mean IoU by 61.7% over the SAM baseline in challenging lighting scenarios.
Efficiency: The LCA module adds only ~2.4M trainable parameters (approx. 2.6% overhead) to the 93.7M SAM ViT-B model, maintaining computational efficiency.
Ablation: Studies confirmed that the depthwise-separable residual projection (V3 variant) was the key architectural driver for performance gains. The consistency loss weight ( $\lambda_c$ ) was found to be stable across a range of values, with 0.1 providing the best balance.

Significance and Claims

The paper claims that PLAP-LCA successfully bridges the domain gap caused by lighting variations, delivering superior lighting-robust segmentation. The authors emphasize that their approach enables safer autonomous driving at night, reliable medical imaging under variable clinical lighting, and robust agricultural monitoring across day-night cycles without expensive model retraining or costly real-world data collection. By treating lighting invariance as a first-class architectural and training problem rather than a secondary augmentation task, the work provides a scalable solution for deploying foundation models in safety-critical, real-world environments. The zero-initialization of the adapter ensures stable training, and the modular design makes illumination robustness accessible for resource-constrained applications.