🤖 machine learning

Does Your Wildfire Prediction Model Actually Work, or Just Score Well?

This paper introduces WILDFIRE-FM, the first foundation model specifically pretrained for wildfire prediction, and proposes a fixed-contract evaluation framework to demonstrate that wildfire transfer conclusions are highly sensitive to evaluation design and task formulation.

Original authors: Yangshuang Xu, Yuyang Dai, Liling Chang, Qi Wang, Yushun Dong

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Yangshuang Xu, Yuyang Dai, Liling Chang, Qi Wang, Yushun Dong

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict where a wildfire will start and how fast it will spread. You have two types of "experts" you can hire:

The Generalist: A highly educated weather forecaster who knows everything about wind, rain, and global climate patterns. They are great at predicting rain in London or snow in Tokyo, but they've never specifically studied fire.
The Specialist: A firefighter who has spent their entire life studying fire behavior, local terrain, and how flames react to dry grass.

This paper asks a simple but tricky question: Is the Generalist actually good at predicting fires, or do they just look good on paper because we are grading them with the wrong test?

The authors, a team from Florida State University and Northeastern University, argue that the answer is: It depends entirely on how you grade the test.

Here is the breakdown of their findings using simple analogies:

1. The "Generalist" Problem

The paper introduces a new model called WILDFIRE-FM. Think of this as the Specialist. It was trained from scratch using data specifically about fires, weather, trees, and hills.

Then, they tested it against ten famous "Generalist" models (like Pangu-Weather or ClimaX). These Generalists were trained on massive amounts of general weather data but were not trained specifically for fire.

2. The "Grading Rubric" Trap (The Matching Rule)

The biggest discovery in the paper is about how we measure success. In wildfire prediction, being "close" is often good enough. If a model predicts a fire will start in a specific forest, and it actually starts in the next forest over, a real-world fire department might still consider that a useful warning.

However, the authors found that the "Generalist" models looked terrible if you graded them with a Strict Rubric (Exact Matching).

The Strict Rubric: "Did you predict the fire in the exact same square inch at the exact same second?"
The Result: Under this strict rule, the Generalists scored near zero. They looked useless.

But, when the authors switched to a Relaxed Rubric (Tolerated Matching)—allowing for a few miles of error or a few hours of delay—the same Generalist models suddenly looked amazing. Their scores jumped from "useless" to "very good."

The Analogy: Imagine a dart player who hits the bullseye 10 feet to the left.

If the rule is "Hit the exact center," they get a 0.
If the rule is "Hit anywhere on the board," they get a 100.
The paper says: Don't just tell me the score; tell me what the rules were, or the score means nothing.

3. The "Head-Selection" Trap

The paper also found that the way you choose the "final answer" from a model matters.

Imagine the model gives you a list of 100 possible outcomes, ranked by how likely they are.
Method A (Ranking): You pick the one that looks most likely on paper.
Method B (Decision): You pick the one that actually works best in a real-world scenario (like minimizing false alarms).

The authors found that sometimes, the "Ranking" method picks a model that looks great on paper but fails in practice. This is called "Selection Regret." It's like hiring a chef because they have the highest number of 5-star reviews (Ranking), only to find out they can't cook the specific dish you need (Decision).

4. The Solution: The "Fixed Contract"

Because the scores change so wildly based on the rules, the authors created a new way to test models called a Fixed-Contract Evaluation.

Think of this like a legal contract before a game starts. Before you compare the Specialist (WILDFIRE-FM) to the Generalists, you must agree on:

The Task: Are we predicting fire start or fire spread?
The Metric: How do we measure success? (F1 score, error rate, etc.)
The Matching Rule: How much error do we allow? (Exact? 5 miles? 10 miles?)
The Scope: Are we looking at the whole world or just fire-prone areas?

The Result of the Contract:
Once they locked in these rules and compared everyone fairly:

The Specialist (WILDFIRE-FM) consistently outperformed the Generalists in predicting fire occupancy and spread.
The Generalists were not "bad," but they were inconsistent. Sometimes they looked great, sometimes terrible, depending entirely on which "contract" (rules) you used.
For some specific tasks (like predicting smoke or extreme heat), a few Generalists performed just as well as the Specialist.

The Bottom Line

The paper concludes that you cannot simply say "Model A is better than Model B" for wildfires.

If you change the rules of the game (how much error you allow, or how you pick the winner), the winner changes. The authors built a new model (WILDFIRE-FM) that is specifically designed for fire, but their most important contribution is a new rulebook (the Fixed-Contract Framework) to ensure that when we compare AI models for disasters, we are comparing apples to apples, not apples to oranges.

In short: A model might look like a hero or a villain depending entirely on how you grade its homework. This paper provides the standardized grading sheet to make sure we know who is actually doing the best job.

Technical Summary: Does Your Wildfire Prediction Model Actually Work, or Just Score Well?

Problem Statement
Wildfire prediction is critical for early warning and resource allocation, yet existing Earth Foundation Models (Earth FMs) are pretrained for general atmospheric and geophysical objectives rather than specific wildfire forecasting. This creates a domain mismatch: wildfire dynamics depend on complex interactions between weather, vegetation, topography, fuel conditions, and active-fire behavior, which are not explicitly modeled during general pretraining. Furthermore, evaluating the transfer of these general models to wildfire tasks is fraught with instability. Wildfire events are sparse in space and time, making evaluation conclusions highly sensitive to three sources of variability:

Matching Rules: Different rules (e.g., exact cell-time overlap vs. spatial/temporal tolerance) yield drastically different performance scores (e.g., F1) for the same model output.
Head-Selection Metrics: The choice of metric used to select a lightweight adapter (e.g., PR-AUC vs. decision-based F1) can favor different heads from the same frozen representation, leading to inconsistent rankings.
Task Forms: Different tasks (occupancy, spread, regression, retrieval) operate over different prediction units and metric families, making scores across tasks non-comparable even under the same backbone.

Standard geospatial benchmarks often fail to control for these variables, leading to non-comparable scores and potentially misleading conclusions about model viability.

Methodology
The authors propose a two-pronged approach: introducing a domain-specific backbone and establishing a rigorous evaluation framework.

WILDFIRE-FM (The Backbone):
The authors introduce WILDFIRE-FM, the first foundation model pretrained specifically for wildfire prediction. Unlike general Earth FMs adapted post-hoc, WILDFIRE-FM is trained directly on fire-relevant multimodal data, including:
- Dynamic Weather: NOAA HRRR fields (temperature, wind, CAPE, precipitation, etc.) at 6-hour intervals.
- Active-Fire Observations: NASA FIRMS detections mapped to a 5km grid.
- Static Context: LANDFIRE fuel models, canopy cover, housing density, and population data.
- Training Strategy: A compact U-Net architecture trained on 32x32 tiles with fire-centered sampling to handle sparsity. The model uses a spatial-support objective (dilated targets) to reduce sensitivity to small grid displacements.
Fixed-Contract Evaluation Framework:
To address evaluation instability, the authors define a Fixed-Contract Evaluation problem. A contract $C$ is a tuple $(T, M, \Lambda, \Omega, A)$ specifying:
- $T$ : Task (e.g., occupancy, spread).
- $M$ : Metric (e.g., F1, RMSE).
- $\Lambda$ : Matching rule (Exact, Tolerated, or Union).
- $\Omega$ : Evaluation scope (Global vs. Fire-prone regions).
- $A$ : Allowed lightweight-head family.
Comparisons are only valid if all five components are identical. The framework introduces two controlled checks:
- Fixed-Output Check: Holds the model output and label fixed while varying the matching rule ( $\Lambda$ ) to isolate the effect of tolerance on performance.
- Fixed-Feature Check: Holds the frozen features and contract fixed while varying the selection metric to measure "selection regret" (the performance loss incurred by using a ranking metric like PR-AUC instead of a decision metric like F1 for head selection).

Key Contributions

WILDFIRE-FM: A specialized foundation model that learns representations aligned with physical fire processes rather than relying on transfer from general atmospheric objectives.
Fixed-Contract Framework: A formal protocol for wildfire transfer evaluation that explicitly controls for matching rules, head-selection metrics, and task forms, separating evaluation artifacts from representation quality.
Systematic Benchmark: A comprehensive comparison of WILDFIRE-FM against ten general-purpose Earth FMs (including Prithvi-WxC, Aurora, ClimaX, Pangu-Weather, etc.) across six task forms under matched contracts.

Results
The experiments address three research questions regarding evaluation sensitivity:

Matching-Rule Sensitivity (RQ1): Under fixed outputs, the choice of matching rule determines whether a model appears usable. Under strict "exact matching," most frozen Earth FMs achieve near-zero F1 scores, rendering them indistinguishable from baselines. However, under "tolerated" or "union" matching (reflecting operational early-warning needs), these same models recover substantial performance. This indicates that prior claims of model viability may be entirely dependent on undisclosed tolerance assumptions.
Head-Selection Sensitivity (RQ2): Under fixed features, selection metrics introduce significant backbone-dependent variation. Some backbones show near-zero regret between ranking and decision metrics, while others (particularly under global scopes with severe class imbalance) show large regret, indicating that PR-AUC-based selection can lead to suboptimal decision performance.
Task-Form Consistency (RQ3): Model rankings are not consistent across task forms.
- Burned Area Regression: Preserves the primary ordering where WILDFIRE-FM leads frozen models.
- Analog Retrieval & Smoke PM2.5: The primary ordering does not hold; general remote-sensing backbones (e.g., AlphaEarth) match WILDFIRE-FM closely, while atmospheric FMs show near-zero correlation on smoke tasks.
- Extreme Heat: Exhibits massive variance, with atmospheric FMs ranging from near-reference performance to complete failure depending on the backbone. Aggregating scores across these tasks without respecting contract boundaries would produce rankings dominated by scale artifacts rather than transfer quality.

Significance and Claims
The paper claims that reliable wildfire transfer evaluation is substantially more difficult than standard benchmark settings suggest. The authors argue that wildfire transfer conclusions are highly sensitive to evaluation design and task formulation. Scores computed under different settings (e.g., different matching rules or scopes) are not directly comparable.

The significance of this work lies in providing a foundation for future wildfire-specific Earth FM research and benchmarking. By introducing WILDFIRE-FM and the fixed-contract framework, the authors provide a system to:

Distinguish between genuine representation quality and evaluation artifacts.
Prevent overconfident benchmark claims that rely on implicit, favorable protocol choices.
Guide real-world intervention and resource allocation by ensuring that model evaluations align with the specific decision tasks (e.g., early warning vs. damage assessment) they are intended to support.

The authors remain modest, noting that their conclusions apply specifically to the task forms, scopes, and eligibility decisions used in the study, and that the evaluation bounds rather than replaces the need for primary decision-task evaluation.