🤖 AI

See the past: Time-Reversed Scene Reconstruction from Thermal Traces Using Visual Language Models

This paper proposes a novel framework that combines Visual-Language Models with a constrained diffusion process to reconstruct plausible past scene states from fading thermal traces left by human interactions, enabling time-reversed imaging up to 120 seconds prior.

Original authors: Kebin Contreras, Luis Toscano-Palomino, Mauro Dalla Mura, Jorge Bacca

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kebin Contreras, Luis Toscano-Palomino, Mauro Dalla Mura, Jorge Bacca

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a room and see an empty chair. To your eyes, it looks just like any other chair. But if you could see the "heat" left behind, you might see a warm, ghostly outline of a person who sat there just a few minutes ago. This paper is about teaching computers to use those invisible heat ghosts to figure out what happened in the room just moments before.

Here is a simple breakdown of how they did it:

The Core Idea: The "Heat Ghost"

When a person sits on a chair, leans against a wall, or touches an object, their body heat leaves a temporary mark. Think of it like a warm footprint in the snow or a fading echo. Even after the person leaves, that spot stays slightly warmer than the rest of the room for a while.

Standard cameras (like the one on your phone) only see the "now." They can't see the heat, so they can't tell if someone just left. But thermal cameras can see these fading heat marks. The researchers wanted to use these heat marks as a "time machine" to reconstruct what the scene looked like a few seconds or minutes ago.

The Problem: Heat is Vague

The problem is that a heat mark is blurry. It tells you something was there, but it doesn't tell you what it was or exactly how it was positioned. It's like seeing a shadow and trying to guess the exact shape of the object casting it.

The Solution: A Team of AI Detectives

The researchers built a system that acts like a team of two AI detectives working together to solve the mystery of "what happened here?"

The Describer (The "Eye"): First, they use a powerful AI (a Visual-Language Model) to look at the current scene. It sees the normal photo (RGB) and the heat photo (Thermal). It acts like a detective who looks at the evidence and writes a short story: "A person was just sitting here; you can see the heat on the chair."
The Artist (The "Hand"): Next, they use a second AI (a Diffusion Model) that is really good at drawing pictures. They give this artist the current photo and the short story written by the first detective. The artist is told: "Draw what this room looked like a few minutes ago, based on this story."

By combining the heat clues with the story, the artist can "reverse the clock" and draw a plausible picture of the past.

What They Tested

They created a special dataset called TraceDataset. It's like a training gym for their AI, featuring 80 different scenarios where people performed simple actions:

Sitting on a chair.
Leaning against a wall.
Touching an object.

They took photos immediately after the action, and then again 5, 15, 30, and even 120 seconds later. They asked the AI to guess what the scene looked like at those earlier times.

The Results: How Far Back Can We See?

The results were promising, but with limits:

The Sweet Spot: The AI could successfully reconstruct the scene up to 30 seconds in the past with high accuracy. It could even guess things like "a person was holding a book" just by looking at the heat trace.
The Limit: As time went on, the heat "ghosts" faded away. By 120 seconds (2 minutes), the heat traces were too weak to give specific details. The AI could still guess the general shape of the room, but it couldn't tell exactly what the person was doing.
The Magic Ingredient: The system worked best when the "Describer" AI wrote a clear story. If they just gave the "Artist" the photos without the story, the results were much worse. The story acted as a guide to keep the drawing realistic.

The Bottom Line

This paper proves that we can use the fading heat left behind by humans to "see" the immediate past. It's not a full time machine yet—it only works for a couple of minutes and needs controlled conditions—but it's the first step toward a technology that can look at a room and say, "I know someone was just sitting here, and here is what they looked like."

Problem Statement

The paper addresses the challenge of recovering recent past states of a scene from current observations. While traditional temporal reasoning in computer vision focuses on predicting future frames (e.g., via Deep Voxel Flow or PredNet), inferring the past from a single static observation is highly ambiguous when relying solely on RGB data, as multiple past trajectories can result in the same current state. The authors propose leveraging thermal imaging to resolve this ambiguity. Since humans (approx. 37°C) are typically warmer than their surroundings, physical interactions (sitting, leaning, touching) leave residual heat traces that decay over seconds to minutes. These "fading imprints" serve as passive temporal codes, offering indirect evidence of recent events that are invisible to standard RGB cameras but detectable in the infrared spectrum.

Methodology

The proposed framework is a time-reversed reconstruction system that synthesizes a plausible RGB image from a time $\Delta$ in the past ( $\hat{x}^{t-\Delta}_{RGB}$ ) using current multimodal inputs: a co-registered RGB frame ( $x^t_{RGB}$ ) and a thermal frame ( $x^t_{Thermal}$ ). The method consists of three primary components:

Multimodal Input Encoding: The system acquires aligned RGB and thermal images at time $t=0$ . The RGB image provides spatial context, while the thermal image encodes the residual heat signatures of recent human-environment interactions.
VLM-Based Guidance (Semantic Priors):
- A pre-trained Visual-Language Model (VLM), specifically GPT-5, is used in inference mode with frozen parameters.
- It takes the RGB image, the thermal image, and a textual prompt ( $p_{desc}$ ) as input.
- It generates a structured textual description ( $p_{out}$ ) of the scene, explicitly identifying thermographic traces and describing the event that occurred seconds or minutes ago (e.g., "a person was recently sitting on the chair").
- Crucially, the authors do not fine-tune the VLM; they rely on its zero-shot capability to interpret thermal traces and convert them into semantic constraints.
Constrained Diffusion Framework (Image Synthesis):
- A pre-trained diffusion model (Gemini 2.5 Flash Image, also known as NanoBanana) is used to generate the past frame.
- The model is not retrained or fine-tuned. Instead, its denoising trajectory is constrained via conditioning.
- The generation process ( $z_{t-1} = g(z_t, x^t_{RGB}, x^t_{Thermal}, p_{out}, p_{gen})$ ) is guided by the current RGB/thermal inputs, the semantic description ( $p_{out}$ ), and a generative prompt ( $p_{gen}$ ) instructing the model to depict the scene "a few seconds earlier" while preserving environmental fidelity.

Dataset and Experimental Setup

To evaluate the method, the authors introduced TraceDataset, a multimodal dataset containing 80 controlled human-environment interaction scenes.

Actions: Sitting on a chair, leaning against a wall, and touching objects.
Protocol: Subjects maintained contact for 30 seconds. RGB and thermal images were captured at time delays of 5s, 15s, 30s, and 120s after contact ceased.
Hardware: FLIR ONE camera (RGB: 1440×1080; Thermal: 80×60, 8.7 Hz).
Ground Truth: The frame captured at 0s (immediately after contact) serves as the reference.

Evaluation Metrics:

Low-level: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM).
High-level: Mean Per Joint Position Error (MPJPE) for pose estimation and Overall Accuracy (OA) for semantic segmentation.

Key Results

Ablation Study:
- RGB-only: Failed to reveal the previous state, often hallucinating plausible but unconstrained frames.
- RGB + Thermal: Improved performance by introducing complementary cues, allowing recovery of basic actions (e.g., sitting).
- Full Model (RGB + Thermal + Descriptor): Achieved the best results. The inclusion of the VLM-generated semantic descriptor increased Overall Accuracy (OA) by 10% and PSNR by 1.3 dB compared to the thermal-only baseline. This confirms that the structured description acts as a strong prior, anchoring the diffusion process to physically consistent semantics.
Generator Comparison:
- The proposed framework was tested against several state-of-the-art generative models (DALL-E 3, Grok, Pixverse, Flux Kontext, Seeddream 4.0).
- Gemini 2.5 Flash Image achieved the highest performance across almost all metrics (highest OA, lowest MPJPE, highest PSNR/SSIM), demonstrating superior ability to integrate multimodal cues and preserve temporal consistency.
Temporal Reconstruction Range:
- Reconstructions remained stable and accurate up to 15 seconds.
- Performance showed noticeable degradation beyond 30 seconds as thermal traces weakened.
- At 120 seconds, the model could preserve the global structure of the scene but failed to capture fine details or precise interactions. The effective temporal horizon is established as one to two minutes.

Significance and Claims

The paper claims to present the first proof-of-concept framework for reconstructing complete past scene states (full-scene appearance) from thermal traces.

Novelty: It is the first work to operationalize residual heat signatures as a computational signal for time-reversed image synthesis, moving beyond previous pose-level inference or short-horizon predictions.
Modularity: The framework is model-agnostic and modular. It does not require retraining the generative backbone, allowing it to benefit from future advances in VLMs and diffusion models without structural changes.
Feasibility: Controlled experiments validate that plausible recovery of past frames is achievable up to two minutes post-interaction, establishing a foundational step toward practical time-reversed imaging for applications in forensics, scene analysis, and security.

The authors remain modest regarding the current scope, noting that the study is limited to controlled scenarios with single subjects and that future work is required to extend the framework to real-world environments with multiple subjects and variable illumination.