🤖 AI

Adding Thermal Awareness to Visual Systems in Real-Time via Distilled Diffusion Models

The paper introduces FusionProxy, a real-time, plug-and-play image fusion module that leverages distilled diffusion models and complementary variance statistics to seamlessly integrate thermal and RGB data, thereby significantly enhancing the robustness and safety of visual perception systems in challenging conditions like nighttime and fog without requiring joint optimization.

Original authors: Yuchen Guo, Junli Gong, Wenjun Dong, Yiuming Cheung, Weifeng Su

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuchen Guo, Junli Gong, Wenjun Dong, Yiuming Cheung, Weifeng Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car at night in heavy fog. Your eyes (or a standard camera) can only see what's in the visible light spectrum—like looking through a dirty, dark window. You might miss a pedestrian in a black coat or a car with broken taillights because there isn't enough light to reflect off them.

Now, imagine you also have a pair of "heat goggles" (an infrared camera). These don't need light; they just see the warmth of living things and engines. The problem is, heat goggles are blurry and lack detail, while your regular camera is sharp but blind in the dark.

The Problem:
Scientists have tried to combine these two views before. Some methods create a perfect, crystal-clear "super-image" by taking thousands of guesses and averaging them out. But this is like trying to bake a cake by tasting it 1,000 times before serving it—it takes way too long. By the time the image is ready, you've already crashed. Other fast methods are quick but produce a blurry, low-quality mess that doesn't help much.

The Solution: FusionProxy
The authors of this paper created a new tool called FusionProxy. Think of it as a "smart translator" that instantly merges the sharp details of your regular camera with the heat-sensing power of the infrared camera.

Here is how they built it, using a few simple analogies:

1. The "Master Chefs" (The Teachers)

First, the researchers hired two expert "Master Chefs" (these are complex AI models called Diffusion Models). These chefs are incredibly talented at mixing the two camera views to create a perfect image, but they are very slow. They take their time, tasting and adjusting the image over and over to get it right.

The Trick: Instead of asking the chefs to cook every single time you drive, the researchers let the chefs cook a few times in advance and wrote down their "notes." They noted not just the final dish, but also where the chefs disagreed with each other.

2. The "Fast Food Trainee" (The Student)

Next, they trained a "Fast Food Trainee" (a lightweight AI model). This trainee is fast and can cook a meal in a split second, but usually, the food isn't very good.

The Training: The trainee didn't just copy the final dish. Instead, it learned from the notes the Master Chefs left behind.
- The "Confidence" Note: If the two chefs agreed on a specific spot (like a pedestrian's face), the trainee learned to paint that spot with high confidence. If the chefs were confused or saw different things, the trainee learned to be careful and not force a guess.
- The "Specialist" Note: The researchers also had the trainee look at the dish through the eyes of four different "Food Critics" (specialized AI models that understand shapes, textures, and objects). If one critic was confused about a specific part of the image, the trainee learned to listen more closely to the other critics who were sure about that part.

3. The Result: Instant, High-Quality Fusion

Once trained, the "Fast Food Trainee" (FusionProxy) can take a raw image from the regular camera and the heat camera and merge them instantly.

Speed: It works fast enough to run on a standard laptop or a car's computer (getting about 30 to 80 frames per second).
Quality: Even though it's fast, the image looks almost as good as the slow, perfect version made by the Master Chefs.
Plug-and-Play: The best part is that you don't need to retrain the car's existing safety systems. You just swap the regular camera feed for this new "super-feed," and the car's brain (which was trained only on regular cameras) suddenly becomes "heat-aware" without any extra work.

Real-World Impact

In tests, they put this system into a simulated driving environment (a video game world called CARLA) with heavy fog and darkness.

Without FusionProxy: The car's AI couldn't see a pedestrian and almost hit them.
With FusionProxy: The "super-feed" revealed the pedestrian clearly in the heat, and the car's AI successfully steered away to avoid a crash.

In Summary:
FusionProxy is like giving a fast, cheap camera the "superpowers" of a slow, expensive one. It uses a clever teaching method to learn how to combine heat and light instantly, making self-driving cars and security systems safer in the dark, without needing super-computers to do the math.

Technical Summary: FusionProxy

Problem Statement
Purely RGB-based vision systems struggle in challenging environmental conditions such as nighttime, fog, and glare, leading to degraded performance and safety risks. While infrared (thermal) imaging provides critical complementary information by capturing heat-emitting sources, existing high-fidelity fusion methods are impractical for real-time edge deployment. State-of-the-art diffusion-based fusion models (e.g., DDFM, Mask-DiFuser) produce superior quality but suffer from prohibitive latency due to iterative sampling, often requiring several seconds per frame. Conversely, real-time methods (e.g., TarDAL) sacrifice too much fusion quality to be effective. Furthermore, active sensors like LiDAR and radar face scalability limitations regarding interference and cost. The core challenge is to achieve diffusion-level fusion quality with the inference speed required for commodity hardware, while ensuring the output remains compatible with existing, frozen RGB-pretrained perception models without retraining.

Methodology: FusionProxy
The authors propose FusionProxy, a plug-and-play, real-time image fusion module that distills a diffusion teacher ensemble into a lightweight, deterministic student network. The framework operates in three key stages:

Diffusion Teacher Ensemble: Instead of relying on a single teacher, the method constructs an ensemble using two pre-trained diffusion fusion models with complementary training distributions: DDFM (trained on IR-VIS pairs for domain-grounded thermal modeling) and Mask-DiFuser (trained on natural images for broad priors and texture fidelity). For each input pair, $N$ samples are drawn from each teacher to form an ensemble.
Dual-Signal Distillation Loss: The student network is trained to approximate the ensemble mean using two complementary statistics derived from the teacher samples:
- Uncertainty-Weighted Pixel Supervision: The per-pixel variance of the teacher ensemble in the raw image space is used to weight pixel-level supervision. Pixels where the teacher ensemble agrees (low variance) receive higher weight, while ambiguous regions (high variance) are down-weighted to prevent the student from learning teacher noise.
- Spatially-Adaptive Multi-Foundation Alignment: To ensure compatibility with diverse downstream models, the student is aligned against a panel of frozen foundation backbones (VGG-16, DINOv2, CLIP, SAM). The per-pixel feature variance across teacher samples for each backbone serves as a routing signal. This allows the system to dynamically weight the contribution of different backbones based on their local informativeness (e.g., SAM for boundaries, CLIP for semantics) at each pixel location.
- Structural Consistency: An SSIM-based loss is added to preserve high-frequency details that might be smoothed out by averaging in pixel space.
Student Architecture and Robustness: The student is a lightweight, deterministic single-pass network (defaulting to a ConvNeXt V2-based U-Net) that processes IR and VIS inputs through dual encoders. To handle real-world sensor misalignment, the training process injects random affine perturbations (translation and rotation) to the IR input while supervising against the aligned teacher ensemble, forcing the student to implicitly learn compensation without adding inference cost.

Key Contributions

Real-Time Diffusion-Quality Fusion: FusionProxy is the first method to achieve diffusion-level fusion quality under strict real-time constraints (≥30 FPS) on commodity hardware (e.g., RTX 4070, RTX 3060, Apple M3), bridging the gap between high-fidelity diffusion models and deployable perception systems.
Dual-Signal Distillation Framework: The paper introduces a novel distillation loss that utilizes a single set of cached teacher forward passes to drive both uncertainty-weighted pixel supervision and spatially-adaptive multi-foundation alignment. This ensures the fused output is robust and compatible with heterogeneous, frozen downstream models.
Plug-and-Play Deployment: The method is designed as a fully independent module. Once trained, it can be integrated into any visual perception system (detection, segmentation, or closed-loop driving) as a drop-in replacement for the visible camera input, requiring no retraining or fine-tuning of the downstream models.

Experimental Results
Extensive experiments validate the efficacy of FusionProxy across multiple dimensions:

Fusion Quality: On the MSRS benchmark, FusionProxy achieves an mIoU of 65.4, which is within 0.5 points of the best non-real-time baseline (Mask-DiFuser at 65.9) but runs approximately 103× faster. It achieves the second-best learned-IQA scores (MUSIQ, CLIP-IQA) in the table, trailing only the non-real-time ControlFusion.
Downstream Performance: When plugged into frozen models without modification, the fused output significantly boosts performance. For YOLOv8 detection, mAP increased by +17.0; for SegFormer-B2 segmentation, mIoU increased by +16.3.
Closed-Loop Driving: In CARLA simulations under degraded visibility (dense fog, night glare), integrating FusionProxy into a frozen CILRS driving policy increased the success rate from 52.4% to 86.5% and reduced collision risk from 32.0 to 3.5.
Inference Speed: FusionProxy achieves real-time inference (84 FPS on RTX 4070, 47.7 FPS on RTX 3060, 37.8 FPS on Apple M3 at 480×640 resolution), outperforming all other diffusion-derived methods which are either non-real-time or significantly slower.

Significance and Claims
The paper claims that FusionProxy provides a flexible and generalizable solution for "all-day perception" by adding thermal awareness to existing RGB-based visual stacks without the need for specialized hardware or system retraining. By decoupling the student architecture from the teacher's heavy denoising network, the method allows lightweight backbones to inherit diffusion-grade fusion quality. The authors emphasize that this approach solves the scalability and latency issues of current fusion methods, making high-fidelity thermal integration practical for real-world autonomous driving and surveillance applications. The paper modestly notes a limitation: because the student is deterministic, it does not retain the generative diversity of the teacher ensemble, meaning it may not be suitable for applications requiring multiple plausible fusion outputs for uncertainty-aware reasoning.

1. The "Master Chefs" (The Teachers)

2. The "Fast Food Trainee" (The Student)

3. The Result: Instant, High-Quality Fusion

Real-World Impact

Technical Summary: FusionProxy

More like this