💻 computer science

ThermoAct:Thermal-Aware Vision-Language-Action Models for Robotic Perception and Decision-Making

This paper introduces ThermoAct, a novel Vision-Language-Action framework that integrates thermal sensor data with visual and linguistic inputs to enhance robotic perception, safety, and task execution in human-robot collaboration environments.

Original authors: Young-Chae Son, Dae-Kwan Ko, Yoon-Ji Choi, Soo-Chul Lim

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Young-Chae Son, Dae-Kwan Ko, Yoon-Ji Choi, Soo-Chul Lim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to help out in your kitchen. You tell it, "Please bring me a cold Coke."

A standard robot, equipped with only "eyes" (cameras that see color), looks at the table. It sees a red can and a blue can. It might grab the red one because it looks like a Coke, but it has no idea if that can is warm from sitting in the sun or cold from the fridge. It's like trying to pick the ripest fruit in a basket while wearing sunglasses that only show you the shape, not the texture or temperature.

ThermoAct is a new way of teaching robots that gives them a "sixth sense": the ability to feel heat.

Here is how the paper explains this breakthrough, broken down into simple concepts:

1. The Robot's New "Superpower" (Thermal Vision)

The researchers gave the robot a special camera that sees heat instead of just colors.

The Analogy: Think of a standard camera like a regular pair of glasses. The new thermal camera is like night-vision goggles for heat. It can see that a hair straightener is glowing hot (dangerous!) or that a cup of water is steaming warm, even if the room is dark or smoky.
Why it matters: Now, the robot can actually follow your command to "pick up the coldest drink" or "turn off that hot appliance" because it can literally feel the temperature difference.

2. The "Brain" and the "Hands" (The Two-Step Team)

Teaching a robot to do complex things all at once is hard, especially when you don't have thousands of examples of it doing those things. The researchers used a clever two-part team structure:

The "Brain" (The VLM Planner): This is like a smart project manager. It looks at the big picture (the thermal images and your voice command) and breaks the big job into tiny, easy steps.
- Example: If you say "Get me a cold Coke," the Brain doesn't just say "Grab can." It thinks: "Is there a cold can? No? Okay, go to the ice maker, fill a cup, then get the Coke."
The "Hands" (The VLA Executor): This is the robot's muscle memory. It takes the tiny steps from the Brain and actually moves the arms to do the work.
- The Magic: Because the "Brain" does the hard thinking, the "Hands" only need to learn simple tasks. This means the robot can learn to do complex, heat-sensitive jobs without needing a massive library of training data.

3. Real-World Tests: What Could It Do?

The team tested this robot in five different scenarios, acting like a helpful but cautious assistant:

The "Warm Water" Test: The robot had to find a cup of warm water among cold ones. With thermal vision, it picked the right one every time. Without it, it was just guessing.
The "Safety" Test: Imagine a hair straightener left on and glowing hot. A normal robot might just move it aside, potentially burning itself or starting a fire. The ThermoAct robot saw the heat, realized it was dangerous, and turned it off first.
The "Conveyor Belt" Test: The robot had to spot a battery that was overheating while it was moving on a belt. It successfully grabbed the hot one and left the cool ones alone.

4. The Results: Why This is a Big Deal

The paper found that giving the robot this "heat sense" made it much smarter and safer.

Success Rate: When the robot had to deal with temperature (like picking the warmest cup), its success rate jumped from about 40% to over 80%.
Safety: It stopped making dangerous mistakes, like trying to touch a hot object.
Efficiency: Even though they didn't have a huge database of thermal images to train on, the "Brain and Hands" team structure allowed the robot to learn quickly and work reliably.

The Bottom Line

ThermoAct is like giving a robot a pair of thermal goggles and a smart manager.

Before, robots were like people trying to cook in the dark; they could see shapes but couldn't tell if the stove was hot.
Now, with ThermoAct, the robot can "feel" the heat, understand the context ("Oh, that's hot, I should turn it off"), and execute the task safely.

This technology is a major step toward robots that can work safely alongside humans in real-world environments, handling everything from serving cold drinks to preventing fires.

1. Problem Statement

Current Vision-Language-Action (VLA) models have demonstrated significant success in robotic manipulation by integrating visual (RGB) and linguistic data. However, they suffer from a critical limitation: the inability to perceive thermal properties.

Safety Risks: Robots cannot distinguish between hot and cold objects, leading to potential hazards (e.g., touching a hot hair straightener or an overheated battery).
Task Failure: Robots fail at commands requiring temperature discrimination, such as "pick up the coldest Coke" or "bring me warm water."
Data Scarcity: Unlike RGB data, large-scale thermal datasets for pre-training VLAs do not exist, making end-to-end training for complex thermal reasoning inefficient and prone to failure in long-horizon tasks.

2. Methodology: ThermoAct Framework

The authors propose ThermoAct, a hierarchical framework that integrates thermal sensing into robotic decision-making. The system consists of two main components:

A. Hierarchical Architecture

Instead of training a single end-to-end VLA for complex thermal reasoning, ThermoAct employs a two-stage approach:

VLM Planner (High-Level): Uses a large Vision-Language Model (specifically Gemini 2.0 Flash) to interpret natural language commands and decompose them into executable sub-tasks.
- Input: User command, RGB image (wrist), Thermal image (external), and a structured guideline prompt.
- Function: Analyzes the environment using thermal cues (e.g., identifying a hot object) and generates a sequence of low-level sub-task descriptions.
VLA Executor (Low-Level): A fine-tuned VLA model (based on $\pi_0$ ) that executes the specific actions defined by the planner.
- Input: Sub-task prompt, real-time thermal image ( $256 \times 256 \times 3$ ), wrist RGB image, and robot state vector.
- Output: Joint angles, gripper control, and a termination flag.

B. Data Processing and Training

Thermal Preprocessing: Raw thermal data ( $256 \times 192$ ) is normalized within a $20^\circ\text{C}$ – $35^\circ\text{C}$ range, quantized to 8-bit, and mapped to the INFERNO pseudocolor palette (dark purple for cold, bright yellow/white for hot). This converts thermal data into an RGB-like format compatible with standard VLA encoders.
Dataset: The authors collected 50 demonstrations per task using a Kinova Gen3 Lite robot. The dataset includes synchronized RGB and thermal streams at 15 Hz.
Fine-Tuning: The VLA executor is fine-tuned using LoRA (Low-Rank Adaptation) to adapt efficiently to the limited thermal data.

3. Key Contributions

Thermal-Aware VLA Framework: The first integration of thermal imaging into a VLA pipeline, enabling robots to perceive physical temperature states and make safety-critical decisions.
Hierarchical Planning for Thermal Tasks: Demonstrates that using a VLM as a high-level planner effectively overcomes the scarcity of thermal pre-training data by decomposing complex thermal reasoning into manageable sub-tasks.
Real-World Validation: Extensive experiments in diverse scenarios, ranging from daily tasks (selecting warm/cold drinks) to safety-critical operations (turning off overheated appliances).

4. Experimental Results

The study evaluated the system across five tasks involving daily manipulation and safety scenarios, comparing ThermoAct (RGB-T) against a baseline RGB-only VLA.

Performance on Thermal Tasks:
- Thermal Perception: ThermoAct achieved an average success rate of 82% on thermal-specific sub-tasks, compared to 42% for the RGB-only baseline (a ~40% improvement).
- Specific Tasks:
  - Pick Warmest Cup: Improved from 30% (30 episodes) to 90% (50 episodes).
  - Turn Off Hot Hair Straightener: Improved from 50% to 90%.
  - Pick Overheated Battery (Dynamic): Achieved 80% success on a moving conveyor belt.
Generalization to Non-Thermal Tasks:
- ThermoAct maintained robust performance on non-thermal tasks (e.g., picking apples), with success rates comparable to the RGB baseline (e.g., 80% vs. 72.5% in Task 1), proving that adding thermal input does not degrade general manipulation capabilities.
Hierarchical vs. Flat VLA:
- A flat, end-to-end VLA attempting to learn the full task sequence achieved near 0% success due to the complexity and data scarcity.
- The hierarchical approach (VLM + VLA) achieved an overall average success rate of 76.8%, significantly outperforming the flat model (18%) and the RGB-only baseline (59.4%).

5. Significance and Future Work

Safety Enhancement: The framework enables robots to proactively detect and mitigate environmental hazards (fire risks, burns) that are invisible to standard RGB cameras.
Data Efficiency: The study proves that a hierarchical architecture allows robots to learn complex thermal behaviors with small datasets (50 episodes), bypassing the need for massive pre-training thermal datasets.
Limitations & Future Directions:
- The system occasionally struggled with depth perception when objects were outside the wrist camera's field of view, leading to hovering or grasping failures.
- Future work suggests fusing thermal data with depth sensors and developing dynamic modality fusion strategies to handle diverse environmental conditions more robustly.

Conclusion: ThermoAct successfully bridges the gap between visual and thermal perception in robotics. By combining the reasoning capabilities of VLMs with the action execution of VLAs, it enables safer, more intelligent, and temperature-aware robotic interactions in real-world environments.