ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery

1. Problem Statement

Vision-Language Models (VLMs) have achieved remarkable success on RGB imagery but fail to generalize to thermal imagery. Thermal sensing is critical in scenarios where visible light is absent or insufficient (e.g., night surveillance, search and rescue, medical screening, autonomous driving). Unlike RGB images, which encode color and texture, thermal images encode physical temperature.

The core problem identified is that current VLMs, trained predominantly on RGB data, lack the perceptual and reasoning capabilities to:

• Ground language in physical temperature signals.
• Interpret false-colored thermal maps (colormaps).
• Perform temperature-grounded reasoning (e.g., estimating absolute temperatures or comparing thermal intensities).
• Avoid relying on "language priors" (e.g., defaulting to a human body temperature of 36.8°C regardless of the image content).

There is a significant lack of systematic benchmarks and datasets with dense, semantic temperature annotations to evaluate these capabilities.

2. Methodology

The authors introduce ThermEval, a comprehensive framework consisting of a benchmark (ThermEval-B) and a new dataset (ThermEval-D).

A. ThermEval-B (The Benchmark)

ThermEval-B evaluates VLMs across seven tasks organized by increasing difficulty, designed to probe foundational thermal understanding primitives:

1. T1 & T2: Modality Identification: Can the model distinguish thermal from RGB images? (T2 adds robustness testing against various colormap transformations like Magma, Viridis, Summer, etc.).
2. T3: Human Counting: Counting pedestrians in thermal scenes (using FLIR and LLVIP datasets).
3. T4: Colorbar Interpretation: Detecting the presence of a colorbar, locating it, and extracting min/max temperature values (OCR task).
4. T5: Thermal Reasoning: Comparing relative temperatures between body parts (e.g., "Which person has a hotter nose?") or ranking body parts by thermal intensity.
5. T6: Absolute Temperature Estimation: Estimating temperature at specific coordinates, marked pixels, or semantic regions.
6. T7: Depth Robustness: Estimating temperatures of semantic regions at varying distances (2ft, 6ft, 10ft) to test robustness to imaging distance.

B. ThermEval-D (The Dataset)

To support T4–T7, the authors created ThermEval-D, the first dataset to provide:

• Dense per-pixel temperature maps (ground truth).
• Semantic body-part annotations (forehead, chest, nose, full body) across diverse indoor and outdoor environments.
• Scale: Over 1,000 images with 35 human participants, collected with informed consent and ethical approval.
• Annotation Quality: High inter-annotator agreement (IoU > 0.72, Dice > 0.84) and precise temperature consistency (std dev ~0.18°C).

C. Evaluation Protocol

• Models Evaluated: 25 VLMs ranging from 0.3B to 235B parameters, including open-source (InternVL, Qwen, LLaVA, MiniCPM, etc.) and closed-source (Gemini 3 Pro/Flash, etc.) models.
• Settings: Standard zero-shot prompting, context-augmented prompting (ablation study), and supervised fine-tuning (SFT) on a representative model (Qwen-VL 2.5).
• Parsing: An LLM-based parser (Gemini 2.5) was used to standardize heterogeneous model outputs for evaluation.

3. Key Contributions

1. ThermEval-B Benchmark: The first structured benchmark with ~55,000 thermal Visual Question Answering (VQA) pairs across seven tasks, moving beyond simple classification to complex reasoning and regression.
2. ThermEval-D Dataset: A novel dataset with dense per-pixel temperature annotations and semantic body-part segmentation, filling a critical gap in available thermal data resources.
3. Systematic Evaluation: A comprehensive analysis of 25 VLMs revealing that current models fail at core thermal tasks regardless of parameter scale.
4. Insight into Failure Modes: Identification that failures are due to a lack of cross-modal grounding (inability to map visual patterns to physical temperature) rather than model capacity.

4. Key Results

• Modality Identification (T1/T2): Most models perform well on raw thermal vs. RGB distinction but degrade significantly under colormap transformations, suggesting reliance on low-level color statistics rather than modality-invariant features.
• Human Counting (T3): Performance varies widely. While larger models (e.g., InternVL 38B) improve, errors remain high (MAE ~3.0 on FLIR) compared to humans (MAE ~1.73).
• Colorbar Interpretation (T4): Many models fail to extract numerical values from colorbars, exhibiting hallucinations (e.g., generating huge numbers) or decimal-shift errors.
• Thermal Reasoning (T5): Models struggle to compare temperatures. They often default to language priors (e.g., assuming a specific body part is always hotter) rather than analyzing the image. Scaling (larger models) does not consistently improve performance.
• Temperature Estimation (T6/T7):
  • Models fail to map pixels to temperature values accurately.
  • Many models output fixed values (e.g., 37°C) regardless of the input, indicating a reliance on language priors.
  • Even the best zero-shot models have MAEs > 3.5°C, far from the human baseline (~2.7°C).
• Prompt Engineering vs. Fine-Tuning:
  • Prompting: Adding contextual descriptions about thermal physics helps with basic modality identification but fails to improve reasoning or estimation tasks.
  • Supervised Fine-Tuning (SFT): SFT on Qwen-VL 2.5 significantly improves performance, allowing it to outperform much larger zero-shot models (including 235B parameter models) and approach human-level performance. However, it still leaves a 1–2°C error margin, which is unacceptable for safety-critical applications like fever screening.

5. Significance and Conclusion

• Thermal Understanding is an Open Challenge: The paper demonstrates that current VLMs, even large ones, cannot effectively reason about thermal data without specific domain grounding. The limitation is not model size but the lack of training on physical sensor modalities.
• Need for Physical Grounding: The results suggest that future VLM pretraining must include physical sensor modalities (thermal, depth, X-ray) rather than relying solely on RGB-centric representations.
• Benchmark Impact: ThermEval provides a diagnostic framework to drive progress in thermal vision-language modeling, highlighting that simple scaling is insufficient and that architectural or training innovations are required to bridge the gap between visual appearance and physical temperature.

In summary, ThermEval exposes a critical gap in multimodal AI: while models can "see" thermal images, they cannot "understand" the physics of heat within them. The work calls for a paradigm shift in training data and evaluation to enable reliable thermal AI for real-world safety-critical applications.