⚡ electrical engineering

Evaluating an Adaptive Multispectral Turret System for Autonomous Tracking Across Variable Illumination Conditions

This paper presents an adaptive multispectral turret system that fuses RGB and long-wave infrared video streams at dynamically selected ratios to significantly improve autonomous object detection confidence and reliability across varying illumination conditions, from full light to no light.

Original authors: Aahan Sachdeva, Dhanvinkumar Ganeshkumar, James E. Gallagher, Tyler Treat, Edward J. Oughton

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aahan Sachdeva, Dhanvinkumar Ganeshkumar, James E. Gallagher, Tyler Treat, Edward J. Oughton

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific coffee mug on a table. If you are standing in bright sunlight, your eyes (which see color and texture) work perfectly. But if the lights go out, your eyes are useless, and you'd need to rely on feeling the heat of the mug with your hands (thermal vision) instead.

This paper is about teaching a robot turret (a camera on a motorized stand) to find objects like coffee mugs in any lighting condition, from bright day to pitch black. The researchers discovered that the robot needs to be "chameleon-like," changing how it sees the world depending on how much light is available.

Here is a breakdown of their work using simple analogies:

The Problem: One Size Doesn't Fit All

The researchers found that standard robot cameras (which only see color, like human eyes) fail when it gets dark. On the flip side, thermal cameras (which see heat) are great in the dark but look like blurry, colorless blobs in the bright sun.

If you try to use just one type of camera all the time, the robot gets confused. It's like trying to wear snowshoes in a desert; they work great in deep snow but are terrible on sand.

The Solution: The "Smart Mixer"

To fix this, the team built a system that acts like a smart audio mixer.

The Inputs: They have two "tracks" of video: one RGB (color) and one LWIR (thermal/heat).
The Mixing: They can blend these two tracks together in different ratios.
- 100% Color / 0% Heat: Pure color vision.
- 50% Color / 50% Heat: A perfect blend.
- 0% Color / 100% Heat: Pure thermal vision.

They didn't just pick one mix and hope for the best. Instead, they created a "menu" of 33 different robot brains (AI models). Each brain was trained to be an expert at a specific mix of color and heat, but only for a specific lighting condition (Bright, Dim, or Dark).

The Experiment: Training the Brains

The team set up a rotating platform with coffee mugs of different colors (white, black, orange, etc.). They filmed these mugs under three conditions:

Full Light: Like a sunny day (>1000 lux).
Dim Light: Like a cloudy day or a dim room (10–1000 lux).
No Light: Pitch black (<10 lux).

They created over 22,000 images by blending the color and thermal videos at 11 different "mixing ratios" (from 100% color down to 100% heat). They then trained a specific AI model for every single combination of Lighting Condition + Mixing Ratio.

The Results: Finding the Perfect Recipe

The results showed that the "best" mix changes depending on the light, just like your recipe for soup changes depending on the weather.

In Bright Light: The robot works best when it relies mostly on Color (80%) with a tiny pinch of Heat (20%). It's like eating a salad; you want the fresh veggies (color), but a little dressing (heat) helps it stick together.
In Dim Light: The robot needs a slightly heavier hand on the heat. The best mix was 90% Color / 10% Heat. The color is still king, but the heat helps the robot see the edges of the object better when the light is fading.
In Total Darkness: The robot has to flip the script. The best mix was 40% Color / 60% Heat. Even though it's dark, keeping a little bit of color information helps the robot understand the shape of the object, while the heat does the heavy lifting.

The "Composite Score" Winner:
The researchers didn't just look for the highest accuracy; they also looked for consistency. They found that in the dark, a 50/50 split had the highest average accuracy, but it was "jittery" (sometimes great, sometimes bad). The 40/60 split was slightly less accurate on average but much more stable and reliable. In a real-world robot, reliability is often more important than a tiny bit of extra accuracy.

The "Adaptive" System

The final system is like a smart thermostat for vision.

A small sensor measures the light in the room (like a thermostat measuring temperature).
If the light is bright, the system instantly switches to the "Bright Light Expert" brain (80% Color / 20% Heat).
If the lights go out, it instantly switches to the "Darkness Expert" brain (40% Color / 60% Heat).

Why This Matters

The paper proves that by simply switching the "recipe" based on the environment, the robot became significantly better at finding objects than robots that only use color cameras or only use thermal cameras.

In bright light, their system was about 10–21% more confident than standard cameras.
In the dark, the improvement was massive, up to 32% more confident.

The researchers also noted that the robot could find mugs it had never seen before (like a teal or yellow mug) because it learned the concept of the object, not just the specific color it was trained on.

In short: The paper shows that a robot doesn't need one super-powerful brain to see in all conditions. Instead, it needs a toolbox of specialized brains that it can swap out instantly depending on whether the sun is shining or the lights are off.

Technical Summary: Evaluating an Adaptive Multispectral Turret System for Autonomous Tracking Across Variable Illumination Conditions

Problem Statement
Autonomous robotic platforms are increasingly deployed in critical sectors such as emergency services and defense, requiring reliable real-time object detection across unpredictable environmental conditions. Traditional Red-Green-Blue (RGB) detection pipelines degrade significantly in low-light environments, while Long-Wave Infrared (LWIR) thermal systems, though effective in darkness, lack the color and texture information provided by visible spectrum sensors. Existing multimodal fusion approaches often struggle with a mismatch between benchmark performance and real-world operational reliability, particularly regarding the trade-offs between fusion strategies, computational efficiency on embedded platforms, and adaptability to dynamic illumination changes without human intervention.

Methodology
The authors developed an adaptive framework that fuses RGB and LWIR video streams to enhance autonomous tracking performance. The system architecture consists of a Raspberry Pi 5-based turret prototype equipped with an RGB camera (Emeet S600), a thermal camera (FLIR Vue Pro R), and an ambient light sensor (Adafruit VEML7700). The Pi forwards sensor data to a remote server for computationally intensive inference, returning detection coordinates to control stepper motors for pan and tilt.

The core experimental methodology involved:

Data Collection: Synchronized RGB and thermal video streams were recorded with target objects (mugs of varying colors) on a rotating platform. Approximately 22,000 fused images were generated and annotated.
Illumination Categorization: Scenes were classified into three categories based on lux readings: Full-light (>1000 lux), Dim-light (10–1000 lux), and No-light (<10 lux).
Fusion Strategy: Frames were fused using a linear blending formula ( $F = \alpha \cdot RGB_{norm} + (1 - \alpha) \cdot LWIR_{norm}$ ) across 11 ratios ( $\alpha$ from 1.0 to 0.0 in 0.1 increments), ranging from pure RGB to pure LWIR.
Model Training: The authors fine-tuned 33 distinct YOLOv11 nano (YOLOv11n) models. Each model was specialized for a specific combination of one fusion ratio and one illumination category. Two pre-trained baselines (YOLOv5n and YOLOv11n trained on COCO) were used for comparison.
Evaluation: Performance was measured using mean detection confidence across six mug colors (including two held-out test colors: teal and yellow). A composite score was calculated to balance mean confidence against standard deviation (variability) to identify the most robust models.

Key Contributions
The paper presents three primary contributions:

Practical Fusion Approach: A method for blending aligned RGB and LWIR images at varying opacity levels to improve target visibility across diverse lighting conditions.
Specialized Model Training: The training and evaluation of 33 YOLO models to identify optimal fusion configurations for specific lighting regimes, utilizing a dataset of over 22,000 annotated images which the authors have open-sourced.
Adaptive Framework: The implementation of a runtime system that dynamically selects the optimal detection model based on ambient illumination readings, allowing the turret to switch between lighting-specific models without manual intervention.

Results
Evaluation demonstrated that adaptive RGB–LWIR fusion significantly outperformed single-modality baselines (YOLOv5n and YOLOv11n) across all conditions:

Full-Light: The optimal configuration was 80% RGB / 20% LWIR, achieving a mean confidence of 92.8%. This significantly outperformed the YOLOv11n baseline (82.6%) and YOLOv5n (71.5%).
Dim-Light: The 90% RGB / 10% LWIR configuration yielded the highest performance with 92.0% mean confidence, surpassing the YOLOv11n (72.4%) and YOLOv5n (69.5%) baselines.
No-Light: Performance peaked at intermediate fusion levels. The 40% RGB / 60% LWIR model achieved 71.0% confidence. While the 50/50 ratio showed slightly higher mean confidence, the 40/60 ratio was selected as optimal due to significantly lower variability (higher composite score). Both fusion models outperformed the baselines (YOLOv11n: 54.1%; YOLOv5n: 38.4%), though the statistical significance was lower than in lit conditions.
Generalization: The system maintained high confidence on held-out mug colors (teal and yellow), indicating that the learned fusion behaviors generalized beyond the training set.

Significance and Claims
The authors claim that their work demonstrates the superiority of adaptive, illumination-aware fusion over fixed fusion strategies or single-modality detection. By dynamically selecting the optimal fusion ratio and corresponding specialized model based on real-time lux readings, the system achieves up to a 30% absolute gain in mean detection confidence compared to baseline YOLO models.

The paper emphasizes that a single, fixed fusion ratio is suboptimal for variable environments. Instead, the proposed approach of training condition-specific models allows the system to leverage the strengths of RGB (texture, color) in bright conditions and LWIR (thermal signatures) in darkness, while using intermediate blends to stabilize detection when one modality is degraded. The authors note that this approach simplifies deployment on embedded systems by relying on a single sensor reading for model switching rather than complex, computationally expensive online feature reweighting. The results suggest this framework is directly applicable to mission-critical tasks such as autonomous security turrets, mobile robotics, and search-and-rescue operations where lighting conditions can shift abruptly.

Limitations
The authors acknowledge several limitations, including a train-test domain shift where static training images were tested against dynamic motion, potential label noise from manual annotation, and the use of a single fixed thermal color palette. They also note that sensor noise and thermal reflections occasionally caused false positives. Future work is proposed to address these by expanding dataset diversity, applying data augmentation, and exploring temporal reasoning architectures.