WildFireVQA: A Large-Scale Radiometric Thermal VQA Benchmark for Aerial Wildfire Monitoring

1. Problem Statement

Wildfire monitoring relies heavily on timely situational awareness from airborne platforms (UAVs). While Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs) show promise in general aerial visual question answering (VQA), existing benchmarks suffer from critical limitations in the context of wildfire management:

• Lack of Thermal Grounding: Most datasets rely solely on RGB imagery, failing to evaluate reasoning based on temperature, which is crucial for detecting smoldering fires, hotspots obscured by smoke, and fire intensity.
• Operational Gap: Current benchmarks focus on generic land-cover or post-disaster assessment rather than the specific, safety-critical reasoning required for tactical wildfire response (e.g., flight planning, hazard localization, and fuel analysis).
• Annotation Reliability: Generating reliable ground truth for complex thermal scenarios is difficult due to the ambiguity of visual cues and the need for physical consistency across frames.

2. Methodology

A. Dataset Construction (WildFireVQA)

The benchmark is built upon the FLAME 3 dataset, which provides synchronized RGB and radiometric thermal imagery from UAVs over prescribed burns.

• Data Composition: 6,097 RGB–thermal samples. Each sample includes:
  1. RGB Image.
  2. Color-mapped thermal visualization (for human inspection).
  3. Radiometric Thermal TIFF: The core innovation, containing per-pixel absolute temperature values (in °C).
• Question Design: 34 distinct questions per frame, totaling 207,298 multiple-choice questions. These are categorized into six operational domains:
  1. Presence & Detection: Identifying fire, smoke, and assets.
  2. Classification: Categorizing fire behavior, vegetation, and fuel types.
  3. Distribution & Segmentation: Estimating fire spread and area coverage.
  4. Localization & Direction: Pinpointing hotspot locations and smoke drift.
  5. Cross-Modal Reasoning: Integrating RGB and thermal cues to resolve occlusions.
  6. Flight Planning: Assessing safety risks and flight constraints.

B. Annotation and Labeling Pipeline

To ensure high reliability in a safety-critical domain, the authors employed a hybrid annotation strategy:

1. MLLM-Generated Proposals: Used Gemini 2.5 Pro to generate initial answers based on RGB, thermal visualizations, and a retrieved radiometric summary (min/max/std dev temperatures, % of pixels >200°C and >400°C).
2. Sensor-Driven Deterministic Labeling: For questions requiring precise physical measurements, the authors bypassed MLLM inference entirely, using algorithms on the raw thermal TIFFs:
   • UAV Altitude: Calculated from GPS/EGM96/SRTM data.
   • Hotspot Detection: Pixels >200°C are classified as active fire; connected components are analyzed for size and ground-projected area.
   • Spatial Distribution: Uses Principal Component Analysis (PCA) on hotspot centroids to classify fire as "Linear," "Concentrated," or "Scattered."
   • Object Counting: Used YOLO-World (open-vocabulary detector) for counting emergency vehicles to avoid MLLM hallucinations.
3. Consistency Verification:
   • Intra-Frame: Logic checks to ensure answers to related questions (e.g., "No Fire" vs. "Hotspot Location") do not contradict.
   • Inter-Frame: Used ORB feature matching to identify near-duplicate frames and enforce label consistency across time.

C. Evaluation Protocol

The authors evaluated four representative MLLMs (LLaVA-v1.6, Qwen3-VL, InternVL2, MiniCPM-V2) under three conditions:

1. RGB Only: Standard visual input.
2. Thermal Only: Input of color-mapped thermal images.
3. Retrieval-Augmented Generation (RAG): Input of RGB/Thermal images + textual radiometric statistics (e.g., "Max temp: 450°C, 5% pixels >400°C").

3. Key Contributions

1. First Radiometric Thermal VQA Benchmark: Introduces WildFireVQA, the first large-scale benchmark explicitly designed to evaluate temperature-grounded reasoning for wildfires, moving beyond simple fire detection.
2. Hybrid Annotation Framework: Proposes a rigorous pipeline combining MLLM generation with deterministic sensor-based algorithms and manual verification to ensure physical accuracy in labels.
3. Comprehensive Evaluation: Establishes baselines for state-of-the-art MLLMs, analyzing how they perform with different modalities and the impact of explicit thermal statistics.
4. Open-Source Release: The dataset and code are publicly available to foster research in autonomous wildfire monitoring.

4. Experimental Results

• Modality Performance: Across all models, RGB imagery remains the strongest modality, outperforming thermal-only inputs. This suggests current MLLMs struggle to interpret raw thermal visualizations without explicit guidance.
• Impact of RAG:
  • Stronger Models Benefit: Qwen3-VL-8B achieved the highest overall accuracy (54.76%) when using RGB + retrieved thermal statistics. InternVL2 also showed significant gains (+5.00% in Thermal setting) with RAG.
  • Weaker Models Struggle: MiniCPM-V2 saw performance decrease with RAG, indicating an inability to effectively integrate structured numeric metadata with visual inputs.
• Task-Specific Insights:
  • Models performed best on "Presence & Detection" (up to 78% accuracy) but struggled with "Flight Planning" and "Localization."
  • Retrieval augmentation provided the most significant gains for tasks directly grounded in temperature data (e.g., hotspot intensity, coverage estimation).

5. Significance and Future Work

• Safety-Critical Benchmarking: WildFireVQA fills a critical gap in evaluating AI for disaster response, highlighting that current models are not yet robust enough for fully autonomous tactical decision-making in complex fire scenarios.
• Value of Physical Grounding: The results demonstrate that while visual appearance (RGB) is dominant, explicit temperature cues significantly boost performance for capable models, validating the need for radiometric data in future wildfire AI systems.
• Limitations: The current RAG approach provides temperature statistics as text rather than requiring models to infer them directly from thermal pixels. Future work aims to develop end-to-end models that can perform direct thermal inference.

In conclusion, WildFireVQA establishes a new standard for multimodal wildfire intelligence, proving that while current MLLMs have limitations, the integration of radiometric thermal data and structured reasoning is essential for the next generation of autonomous fire-monitoring systems.