TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation

TherA is a novel framework that addresses the scarcity of thermal infrared data by combining a thermal-aware visual-language model with a latent diffusion translator to generate diverse, physically plausible, and controllable RGB-to-TIR images based on user prompts.

The Gist

Imagine you have a magical camera that can see the world in two different ways:

1. The "Daylight" View (RGB): This is what your eyes see. It shows colors, shapes, and textures.
2. The "Heat" View (Thermal): This is what a thermal camera sees. It doesn't see colors; it sees temperature. Hot things glow bright white, and cold things look dark blue or black.

For a long time, scientists have wanted to teach computers to translate the "Daylight" view into the "Heat" view automatically. This is super useful for things like self-driving cars (which need to see pedestrians at night) or search-and-rescue missions.

However, there's a big problem: Heat is tricky.

The Problem: The "Copy-Paste" Mistake

Imagine you have a photo of a car.

• Scenario A: The car is driving down the highway. The engine is hot, the tires are hot, and the exhaust is steaming.
• Scenario B: The same car is parked in a garage, turned off, and has been sitting there for hours.

If you ask a standard AI to turn the photo of the driving car into a thermal image, it might do a decent job. But if you ask it to turn the photo of the parked car into a thermal image, old AI models often make a silly mistake. They might still draw the car as "hot" because they just learned that "cars look like cars." They forget the physics: A parked car is cold.

Old AI models treat this like a simple art filter (like turning a photo black and white). But turning a photo into a thermal image isn't just about style; it's about physics. You need to know if the engine is running, if it's raining, or if the sun just set.

The Solution: TherA (The "Thermal Physicist" AI)

The authors of this paper created a new system called TherA. Think of TherA not just as a translator, but as a Thermal Physicist who helps a painter.

Here is how it works, using a simple analogy:

1. The Translator (The Painter)

This is the part of the AI that actually draws the thermal image. It's like a talented painter who knows how to use heat colors.

2. The Therapist (TherA-VLM)

This is the new, special part. Before the painter touches the canvas, they ask the Therapist (a specialized AI brain) for instructions.

• The Old Way: The painter just looked at the photo and guessed. "Oh, it's a car, so I'll make it hot."
• The TherA Way: The painter asks the Therapist: "Hey, look at this car. Is it running? Is it raining? Is it night time?"

The Therapist analyzes the photo and gives the painter a secret recipe (a "thermal embedding").

• Recipe for the driving car: "Make the engine and wheels glowing hot. Make the exhaust steamy."
• Recipe for the parked car: "Make the whole car cool and dark. No steam."
• Recipe for a rainy day: "Make everything look damp and cooler."

Why is this a Big Deal?

1. It's Controllable (The "Remote Control" Feature)
With TherA, you can change the story of the image just by typing a note or showing a reference picture.

• Text Control: You can type "Make it a rainy night," and TherA will instantly redraw the scene so the road looks wet and the car looks cooler, even if the original photo was a sunny day.
• Reference Control: You can show the AI a picture of a "parked car" and tell it, "Make this car look like that one." The AI will instantly cool down the car in the new image to match the reference.

2. It's Smarter About Physics
Because the Therapist understands the rules of heat (like how metal gets hot in the sun or how engines generate heat), the results look real. It doesn't just guess; it reasons. If you show it a car with no exhaust smoke and the engine off, it knows to draw it as cold.

3. It Solves the "Data Scarcity" Problem
Real thermal cameras are expensive, and taking thousands of thermal photos is hard. TherA can take millions of cheap, easy-to-get photos (like those from your phone) and turn them into realistic thermal photos. This gives scientists a massive library of "fake" thermal data to train better self-driving cars and robots.

The Bottom Line

Think of TherA as the difference between a photocopier and a chef.

• A photocopier (old AI) just copies the colors and shapes, often getting the "temperature" wrong.
• TherA is a chef who tastes the ingredients (the photo), understands the recipe (the physics of heat), and cooks up a brand new dish (the thermal image) that tastes exactly right, whether it's a hot summer day or a cold winter night.

This breakthrough means we can finally generate realistic thermal images on demand, helping robots and cars "see" the heat of the world much better than ever before.

Technical Summary

1. Problem Statement

Thermal Infrared (TIR) imaging is crucial for robust perception in low-visibility conditions (e.g., night, fog, smoke). However, acquiring large-scale, annotated TIR datasets is prohibitively expensive due to the cost of sensors and the difficulty of labeling low-texture imagery. While physics-based simulators exist, they often lack scene diversity and require extensive engineering to model specific thermal behaviors.

Current solutions rely on RGB-to-TIR image translation to synthesize pseudo-TIR data. However, existing methods suffer from two major limitations:

1. Lack of Thermal Physics: Most approaches treat translation as a simple style transfer or pixel-level mapping based on RGB priors. They fail to account for physical factors like material emissivity, active heat sources, and environmental conditions (time of day, weather). This leads to thermally implausible outputs (e.g., a parked car appearing as hot as a moving one, or buildings lacking realistic heat distribution).
2. Lack of Controllability: Existing models enforce a deterministic one-to-one mapping, ignoring the inherent one-to-many nature of thermal imagery (where one RGB scene can correspond to multiple valid thermal states depending on context). They offer no mechanism for users to explicitly control thermal attributes like weather or object activity.

2. Methodology

The authors propose TherA, a framework consisting of two synergistic stages: a Thermal-Aware Vision-Language Model (TherA-VLM) and a VLM-Conditioned Latent Diffusion Model.

A. The R2T2 Dataset

To train the system, the authors constructed R2T2, a dataset of 100k+ aligned RGB-TIR-text triplets.

• Construction: They utilized multimodal LLMs (Gemini 2.5 Pro) to analyze paired RGB-TIR images and generate structured, canonical text descriptions.
• Schema: Unlike standard captions, the text follows a physics-grounded schema encoding:
  • Scene: Time of day, weather, location.
  • Objects: Category, material, position.
  • Heat Emission State: Active (self-heating, e.g., running engine) vs. Passive (environment-heated, e.g., parked car).
  • Prior: Thermal context not visible in RGB (e.g., solar loading on pavement).

B. TherA-VLM (Thermal-Aware Vision-Language Model)

• Architecture: Based on fine-tuned LLaVA 1.5.
• Function: It takes an input RGB image and a user prompt (or reference image) and outputs a structured thermal embedding ($h_N$).
• Innovation: Instead of generating free-form text, it predicts a compact schema representing physical thermal attributes. This embedding captures the "thermal context" of the scene, bridging the gap between visual appearance and physical heat emission.

C. VLM-Conditioned Diffusion Translator

• Architecture: A Latent Diffusion Model (based on Stable Diffusion) with a modified UNet.
• Input: The UNet receives 8 channels: 4 for the noisy TIR latent and 4 for the RGB guidance latent.
• Conditioning: The standard CLIP text embedding is replaced by the Thermal Embedding ($h_N$) from TherA-VLM.
  • A TE-Adapter (a two-layer feed-forward network) projects the VLM's high-dimensional embedding into the UNet's cross-attention space.
• Training: The VLM is frozen; only the UNet and the TE-Adapter are trained using a diffusion noise-prediction loss.
• Inference (Dual-CFG): The model uses Dual Classifier-Free Guidance (CFG) to balance visual structure (from RGB) and thermal semantics (from VLM), allowing for fine-grained control.

D. Controllability Mechanisms

TherA supports two modes of control without altering scene geometry:

1. Text-Guided Control: Users provide natural language prompts (e.g., "rainy weather," "nighttime") to modulate global thermal attributes.
2. Reference-Guided Control: Users provide a reference RGB image (e.g., an image of an active car) to transfer specific thermal states (active vs. passive) to the target image.

3. Key Contributions

1. Thermal-Aware VLM Conditioning: Introduction of TherA-VLM, which generates physically grounded, structured thermal embeddings to guide diffusion, moving beyond simple RGB-centric style transfer.
2. Controllable Thermal Modulation: The first framework to enable explicit control over thermal appearance (weather, time of day, object activity) via text or reference images.
3. R2T2 Dataset: Release of a large-scale, structured RGB-TIR-text dataset designed specifically for learning thermal physics.
4. State-of-the-Art Performance: Achieved significant improvements in translation quality and zero-shot generalization compared to existing baselines.

4. Experimental Results

The method was evaluated on M3FD, FLIR, and CART datasets against a comprehensive suite of baselines (including InstructPix2Pix, DiffV2IR, F-ViTA, and ThermalGen).

• Quantitative Performance:
  • TherA achieved SOTA results across all metrics (PSNR, SSIM, FID, LPIPS).
  • On the M3FD benchmark, it improved PSNR by 0.57 dB over the previous best (DiffV2IR).
  • Zero-Shot Generalization: When trained only on R2T2 and tested on unseen datasets (CART, FLIR), TherA outperformed all baselines by a large margin (e.g., +33% average improvement in metrics), demonstrating robust generalization to unseen domains.
• Qualitative Results:
  • Physical Plausibility: Correctly distinguishes between active (hot) and passive (cool) objects (e.g., moving cars vs. parked cars), a failure point for previous methods.
  • Controllability: Successfully simulates day-night cycles and weather changes (rain, snow) while maintaining structural integrity.
• Downstream Tasks:
  • Segmentation: Pre-training segmentation models on TherA-generated pseudo-TIR data yielded higher mIoU scores on real thermal datasets compared to other translation methods.
  • Image Matching: Improved cross-modal RGB-TIR feature matching performance significantly.

5. Significance and Impact

• Solving the Data Bottleneck: TherA provides a scalable solution for generating high-quality, diverse pseudo-TIR data, reducing the reliance on expensive sensor collection.
• Physics-Grounded AI: By integrating vision-language reasoning with physical thermal principles, the paper demonstrates that semantic understanding of "heat" is essential for realistic thermal synthesis, moving the field beyond pure pixel-level style transfer.
• Versatility: The controllability feature allows researchers to generate specific thermal scenarios (e.g., "nighttime with active vehicles") on demand, facilitating the creation of targeted datasets for autonomous driving and surveillance.
• Limitations: The current system operates on relative thermal imagery (normalized contrast) rather than absolute radiometric values, limiting its use for metric temperature estimation. It also relies on sufficient RGB visibility; severely degraded RGB inputs (e.g., total darkness) can lead to suboptimal thermal predictions.

In conclusion, TherA represents a paradigm shift in RGB-to-TIR translation, establishing a new standard for physically plausible, controllable, and generalizable thermal image synthesis.