💻 computer science

FlameVQA: A Physically-Grounded UAV Wildfire VQA Benchmark with Radiometric Thermal Supervision

This paper introduces FlameVQA, a novel multiple-choice visual question answering benchmark for UAV-based wildfire monitoring that leverages paired RGB and radiometric thermal imagery to enable temperature-grounded reasoning, while evaluating current multimodal large language models to highlight their strengths in cross-modal tasks and limitations in smoke-obscured detection and coverage estimation.

Original authors: Mobin Habibpour, John Spodnik, Niloufar Alipour Talemi, Fatemeh Afghah

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mobin Habibpour, John Spodnik, Niloufar Alipour Talemi, Fatemeh Afghah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a wildfire from a drone's camera. Usually, you just look at a standard color photo (RGB). But in a wildfire, thick smoke can hide the flames, or you might see a smoldering ember that looks like just a pile of leaves to the naked eye. It's like trying to find a warm cup of coffee in a dark room just by looking at it; you might miss it entirely.

This paper introduces FlameVQA, a new "test" designed to see how well Artificial Intelligence (AI) can act as a smart assistant for drone pilots fighting wildfires. Here is the breakdown in simple terms:

1. The Problem: "Seeing" isn't enough

Standard AI models are great at looking at color photos. But in a wildfire, color photos are often misleading. Smoke hides the fire, and heat can exist without visible flames. The authors argue that to truly understand a fire, an AI needs to "see" the heat, not just the light.

2. The Solution: A "Thermal X-Ray" Vision Test

The researchers built a benchmark (a standardized test) called FlameVQA.

The Data: They used a dataset called FLAME 3, which contains pairs of images for every single frame: a standard color photo AND a "radiometric thermal" image. Think of the thermal image as an X-ray that shows exactly how hot every single pixel is, giving the AI a temperature reading for the whole scene.
The Test: Instead of just asking "Is there a fire?", the test asks 34 different multiple-choice questions per image. These questions cover real-world firefighter needs, such as:
- "Is there a fire hidden under this thick smoke?"
- "How much of the forest is currently burning?"
- "Where is the hottest spot located?"
- "Is it safe for the drone to fly here, or is the smoke too thick?"

3. How They Made the Test Reliable (The "Truth Engine")

Creating a test where the answers are 100% correct is hard because humans can disagree on what they see in smoke. To fix this, the authors used a clever "hybrid" method to generate the correct answers (the "Ground Truth"):

The AI Assistant: They first asked a powerful AI to guess the answers.
The Physics Rulebook: Then, they checked those guesses against the actual temperature numbers from the thermal files. If the thermal file says a spot is 500°C, the AI must say it's a fire, regardless of what the color photo looks like. This is like using a thermometer to settle a debate about whether a cup is hot.
The Logic Check: They ran logical checks (e.g., "If the AI says there is no fire, it cannot also say there is a hotspot").
Human Review: Finally, human experts checked the tricky cases to ensure everything made sense.

4. The Results: AI is Good, But Not Perfect

The researchers tested two popular AI models (LLaVA and Qwen-VL) on this new test.

The Good News: When the question was about combining the color photo and the heat map (e.g., "Is the fire visible in the color photo, or only in the heat map?"), the AI did very well. It successfully used the "X-ray" vision.
The Bad News: The AI struggled with two specific things:
1. Heavy Smoke: When smoke was very thick, the AI often failed to detect that a fire was even present, even with the thermal data.
2. Counting and Estimating: The AI was bad at estimating how much of the area was covered by fire or smoke (e.g., "Is it 20% or 40%?"). It's like the AI can point to the fire but can't tell you how big the room is.

5. The Bottom Line

The paper concludes that while current AI models are getting better at looking at wildfires, they aren't ready to replace human experts yet. They need more specific training to handle the chaos of smoke and the precise math of fire coverage.

In short: FlameVQA is a new "driver's license test" for AI drones. It proves that giving AI a "heat vision" camera helps it see better, but it still needs more practice to accurately judge how big the fire is and to spot fires hidden deep in thick smoke.

Technical Summary: FlameVQA

Problem Statement

Wildfire monitoring via Unmanned Aerial Vehicles (UAVs) requires reliable reasoning over complex aerial scenes where smoke, scale variations, and occlusions often render RGB-only interpretation insufficient. While existing remote sensing Visual Question Answering (VQA) benchmarks (e.g., RSVQA, HRVQA) address high-resolution scenes, they predominantly focus on RGB optical imagery and general-purpose semantic queries. These benchmarks lack evaluation capabilities for hazard-centric domains where modality-specific signals—specifically thermal signatures revealing obscured hotspots, smoldering regions, and residual heat—are critical for safety and situational awareness. Furthermore, existing UAV wildfire datasets have primarily supported detection, segmentation, or classification tasks rather than multimodal question answering that integrates perception with higher-level operational reasoning.

Methodology

Dataset Construction: FlameVQA

FlameVQA is a multiple-choice VQA benchmark built upon the FLAME 3 dataset, which consists of synchronized aerial RGB and radiometric thermal imagery captured by UAVs during prescribed burns at three sites: Sycan Marsh, Willamette Valley, and Shoetank.

Modalities: Each sample includes an RGB image, a color-mapped thermal visualization, and a radiometric thermal TIFF file containing per-pixel temperature values. This allows for temperature-grounded reasoning beyond visual cues.
Question Suite: The benchmark comprises 34 multiple-choice questions per image, featuring carefully curated distractors and randomized option ordering to mitigate language-model biases. The questions are categorized into six operational capability groups:
1. Presence/Detection: Identifying fire, smoke, thermal activity, and safety-relevant context.
2. Classification: Assigning labels for fire behavior, vegetation, fuel condition, and accessibility.
3. Distribution/Segmentation: Characterizing fire/fuel/smoke distribution and coverage estimates above specific temperature thresholds.
4. Localization/Direction: Providing spatial grounding for hotspots, smoke origins, and man-made elements.
5. Cross-Modal Reasoning: Evaluating joint reasoning over RGB and thermal signals to address modality gaps and occlusions.
6. Flight Planning: Assessing UAV-centric factors such as viewpoint, altitude, and safety risks near flames.

Labeling and Quality Control Pipeline

To ensure label reliability at scale, the authors employ a hybrid labeling engine:

MLLM-Assisted Generation: Gemini 2.5 Pro generates initial answers using RGB images, thermal visualizations, and a compact temperature summary (min/max, threshold exceedance) derived from the radiometric TIFF.
Deterministic Verification: For temperature-critical questions, labels are validated against the raw radiometric TIFF using NumPy to inspect per-pixel temperatures. Physical rules (e.g., binary hotspot masks based on temperature thresholds $\tau_{fire}$ ) and metadata (EXIF GPS/altitude combined with terrain models) provide objective, reproducible supervision.
Consistency Checks: Logical constraints are applied across semantically linked questions (e.g., if "No fire" is selected, hotspot questions must return "No hotspots"). Violations are flagged for human expert review.
Human Auditing: A targeted human evaluation was conducted on the Willamette Valley subset (chosen for dense smoke and ambiguity) focusing on seven challenging questions (covering presence under smoke, coverage estimation, and cross-modal reasoning).

Key Results

Human vs. Automated Label Agreement

On the Willamette subset, the automated pipeline (MLLM + rules) achieved a 70.78% match rate with human expert ground truth across seven challenging question types.

High Agreement: Cross-modal reasoning (CMR) and spatial localization (LD) showed strong agreement (82–90%).
Low Agreement: Presence detection under heavy smoke (PD5) and smoke coverage estimation (DS5) showed the lowest agreement (36% and 45%, respectively), indicating inherent ambiguity even for human experts in these specific scenarios.

Baseline MLLM Performance

Two open-source Multimodal Large Language Models (MLLMs), LLaVA-1.6-7B and Qwen3-VL-8B-Instruct, were evaluated on the benchmark:

Overall Performance: Qwen-VL outperformed LLaVA across all categories, achieving 48.65% overall accuracy compared to LLaVA's 36.23%.
Strengths: Both models performed well on Cross-Modal Reasoning (85–87%), suggesting they can effectively exploit joint RGB–thermal cues when the task is explicit.
Weaknesses: Significant failures were observed in Presence Detection under heavy smoke (PD5: 14–45%) and Distribution/Coverage Estimation (DS: 18–25%). These results highlight current limitations in spatial aggregation and percentage-based reasoning within complex wildfire scenes.

Significance and Contributions

The paper claims the following contributions:

First Dedicated Benchmark: FlameVQA is the first VQA benchmark specifically for wildfire monitoring that couples RGB imagery with radiometric thermal supervision, enabling physically grounded, safety-critical reasoning.
Operational Focus: The benchmark targets realistic operational intelligence tasks (perception, localization, quantification, and flight planning) rather than simple detection, providing a challenging evaluation for multimodal models in disaster response.
Baseline and Analysis: By evaluating representative MLLMs, the work establishes baselines and reveals consistent limitations in current state-of-the-art models regarding safety understanding, multimodal grounding in occluded scenarios, and multi-image reasoning.
Scalable Pipeline: The proposed hybrid labeling approach (MLLM + deterministic rules + consistency checks) offers a scalable method for constructing benchmarks under limited annotation budgets.

The authors conclude that while current MLLMs show promise when explicit cross-modal cues are available, they require domain-specific adaptation to effectively support disaster and wildfire monitoring, particularly in scenarios involving heavy smoke and complex spatial reasoning. The dataset and code are released as open-source to facilitate future research.