⚡ electrical engineering

Evaluating the Generalizability of Foundation Models for Extreme Environmental Events: Case Study of California Wildfire PM2.5

This study benchmarks six time series foundation models against traditional deep learning baselines for forecasting extreme wildfire PM2.5 in California, revealing that fully trained recurrent models consistently outperform both zero-shot and fine-tuned foundation models in accuracy and extreme event detection, thereby challenging the assumption that larger pretrained models universally dominate environmental forecasting.

Original authors: Yongcan Huang, Li Jiang, Ze Yu Liu

Published 2026-09-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yongcan Huang, Li Jiang, Ze Yu Liu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Air pollution is one of the most persistent threats to public health in the modern world, but few events are as sudden or as dangerous as the smoke from a wildfire. When a fire ignites, it does not just release a steady stream of smoke; it sends up massive, unpredictable spikes of tiny particles that can travel hundreds of miles and settle deep into the lungs. These particles, known as PM2.5, are so small they can enter the bloodstream, and during severe fire events, their concentration can rise to levels hundreds of times higher than what is considered safe. Predicting these spikes is a matter of life and death, allowing communities to issue evacuation orders, close schools, and prepare hospitals before the air becomes toxic. For decades, scientists have tried to forecast these events using complex physics-based models that simulate how smoke moves through the atmosphere, but these systems are often too slow or require too much data to be useful in real-time emergencies. In recent years, a new approach has emerged: using artificial intelligence to learn patterns from past data. The most promising of these are "foundation models," massive computer programs trained on vast amounts of general time-based data, from stock markets to weather patterns, designed to predict the future of almost any sequence of numbers without needing specific training for each new task.

A team of researchers set out to test whether these powerful, general-purpose artificial intelligence models could handle the extreme and chaotic reality of wildfire smoke. They focused on California, a state that has suffered through a decade of increasingly severe fire seasons. The team gathered a massive dataset containing hourly air quality readings from 79 monitoring stations across the state, covering the period from 2013 to 2025. This collection included data from 1,375 distinct wildfire incidents, capturing everything from mild background pollution to extreme events where particle levels soared to nearly 1,250 micrograms per cubic meter. The researchers wanted to see if the new foundation models, which had never seen this specific type of data before, could predict the next six, twelve, or twenty-four hours of air quality better than older, specialized computer programs that were trained specifically on wildfire data. To ensure a fair test, they designed a rigorous evaluation method where the computer models were asked to predict the air quality of a fire they had never seen before, using only data from other fires to learn. This approach prevented the models from simply memorizing the details of a specific fire and forced them to truly understand the underlying patterns of smoke behavior.

The results of this study revealed a surprising truth about the capabilities of these large artificial intelligence systems. Despite their reputation for solving complex problems and their ability to learn from massive amounts of data, the general-purpose foundation models did not outperform the older, specialized models when it came to predicting the most dangerous moments of a wildfire. The most accurate predictions came from a compact, fully trained computer program known as a bidirectional long short-term memory network, or BiLSTM. This model, which was built specifically for this task and trained on the historical wildfire data, consistently made the smallest errors in its forecasts. It was particularly skilled at identifying when air quality would cross the threshold into "hazardous" levels, a critical capability for public safety. In contrast, the general foundation models, which were tested without any specific training on the fire data, performed only slightly better than a simple guess that assumed the air quality would stay the same as it was in the last hour. While these large models could predict the average air quality reasonably well, they struggled significantly when it came to the extreme spikes. One of the foundation models, in particular, produced a few massive errors that skewed its overall performance, causing it to miss the most severe peaks entirely or predict them at the wrong time.

The researchers also tested whether they could improve the performance of these large models by giving them a small amount of specific training on the fire data, a technique known as fine-tuning. This process allowed the models to adjust their internal settings to better fit the unique behavior of wildfire smoke. While this fine-tuning did help, it did not close the gap. Even after being adjusted, the foundation models still could not match the accuracy of the specialized model that had been trained from the ground up on the fire data. The study suggests that the sheer size and general nature of the foundation models are not enough to overcome the unique challenges of predicting extreme environmental events. These models, trained on vast amounts of ordinary data, tend to smooth out the wild, jagged spikes that characterize wildfire smoke, effectively averaging out the danger. The specialized model, having learned the specific rhythm and intensity of fire smoke over many years, retained the ability to anticipate those sudden, life-threatening surges.

This finding offers a clear path forward for scientists and public health officials who rely on air quality forecasts. It suggests that for the most critical, high-stakes predictions—such as warning a community of imminent hazardous air—specialized, smaller models trained on specific historical data remain the most reliable tool. While the large, general-purpose artificial intelligence models show promise for general forecasting and can be improved with some specific training, they are not yet a replacement for the dedicated systems that have learned the specific language of wildfire smoke. The study underscores that in the face of rare and extreme events, the depth of domain-specific knowledge often outweighs the breadth of general knowledge. For communities living in fire-prone regions, this means that the most effective early warning systems will likely continue to be those built by experts who have studied the specific behavior of smoke in their local environment, rather than relying solely on the broad, general intelligence of massive, pre-trained algorithms.

Technical Summary: Evaluating the Generalizability of Foundation Models for Extreme Environmental Events

Problem Statement

Accurately forecasting extreme spikes in PM2.5 concentrations caused by wildfire smoke remains a critical challenge for public health, particularly for issuing timely evacuation orders and hospital surge preparations. While Time Series Foundation Models (TSFMs) have demonstrated strong performance on general benchmarks through zero-shot inference and parameter-efficient adaptation, their efficacy under extreme, out-of-distribution (OOD) conditions—specifically the heavy-tailed, non-stationary dynamics of wildfire smoke—remains untested.

Existing literature suffers from two primary limitations:

Evaluation Protocols: Most studies rely on chronological train-test splits or single-site settings. These methods often leak information from the same fire event between training and testing, failing to assess a model's ability to generalize to a completely unseen incident.
Domain Applicability: It is unclear whether large-scale pretraining on general corpora transfers effectively to the specific, episodic, and extreme nature of wildfire PM2.5, or if lightweight fine-tuning (e.g., LoRA) can bridge the gap between foundation models and domain-specific supervised learning.

Methodology

Dataset and Preprocessing

The study utilizes a 12-year (2013–2025) hourly dataset comprising 1,375 wildfire incidents across 79 EPA-certified monitoring sites in California. The dataset includes 1.73 million incident-aligned records, with PM2.5 concentrations ranging from 0 to 1,249 µg/m³. Notably, the data is heavily skewed: 93.8% of readings fall in the "Good-to-Moderate" range, while only 0.28% reach "Hazardous" levels (>225.5 µg/m³).

Preprocessing involved:

Event Alignment: Pairing each wildfire incident with its nearest monitoring station (within 100 km).
Gap Handling: Avoiding interpolation to prevent artificial dynamics; data was segmented into continuous blocks.
Normalization: Per-site z-score normalization for trained baselines; native units for foundation models (with specific handling for Time-MoE).
Windowing: A 48-hour input context was used to forecast 6, 12, and 24-hour horizons.

Evaluation Protocol: Leave-One-Incident-Out (LOIO)

To prevent temporal leakage and simulate the operational scenario of predicting a new fire with no prior incident-specific history, the authors employed a Leave-One-Incident-Out (LOIO) cross-validation protocol.

Grouping: All time windows associated with a single wildfire incident are assigned to the same fold.
Stratification: Folds are stratified by the peak concentration of the incidents to ensure balanced severity distribution across training and testing sets.
Validation: In each fold, the model is trained on all other incidents and tested exclusively on the held-out incident.

Models Evaluated

The study benchmarks ten configurations across three categories:

Trained Deep Learning Baselines: LSTM, BiLSTM, and Transformer (fully trained from scratch on the wildfire data).
Zero-Shot Foundation Models: TimesFM, Chronos-2, Moirai-2, and Time-MoE (pretrained weights frozen, no gradient updates).
LoRA Fine-Tuned Foundation Models: Chronos-2 and Time-MoE adapted using Low-Rank Adaptation (LoRA) on the training folds only.

Metrics

Performance was evaluated using:

Regression Metrics: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Coefficient of Determination ( $R^2$ ).
Exceedance Detection: F1-scores at EPA AQI thresholds (35.5, 55.5, 125.5, and 225.5 µg/m³) to assess the ability to predict hazardous events.

Key Results

1. Superiority of Trained Recurrent Baselines

Contrary to the assumption that larger pretrained models universally dominate, the fully trained BiLSTM achieved the best performance across all metrics:

Lowest MAE: 5.16 µg/m³ (compared to 5.56–5.92 µg/m³ for zero-shot foundation models).
Highest Exceedance F1: At the Hazardous threshold (>225.5 µg/m³), BiLSTM achieved an F1 of 0.63, significantly outperforming the best zero-shot foundation model (0.54) and the LoRA-fine-tuned variants.
Stability: The BiLSTM maintained consistent performance across all forecast horizons (6, 12, 24 hours).

2. Limitations of Zero-Shot Foundation Models

Modest Gains: Zero-shot TSFMs improved slightly over a persistence baseline in aggregate MAE but failed to approach the performance of trained recurrent models.
Tail Instability: Zero-shot Chronos-2 exhibited severe instability. While its MAE was competitive (5.61 µg/m³), its RMSE was exceptionally high (23.45 µg/m³) with a negative $R^2$ (-0.075). This indicates sporadic, large-magnitude errors that are penalized heavily by squared-error metrics but invisible to absolute-error metrics.
Horizon Degradation: The performance gap between trained baselines and foundation models widened as the forecast horizon increased from 6 to 24 hours.

3. Impact of LoRA Fine-Tuning

Partial Remediation: LoRA fine-tuning substantially improved the performance of both Chronos-2 and Time-MoE. For Chronos-2, it repaired the tail instability (RMSE dropped from 23.45 to 15.67 µg/m³; $R^2$ improved from -0.08 to 0.60).
Persistent Gap: Despite improvements, neither LoRA-fine-tuned variant surpassed the fully trained BiLSTM or LSTM on any metric. The authors suggest that lightweight adaptation recalibrates magnitude scales but fails to instill the fine-grained temporal dynamics required for extreme-event exceedance.

Key Contributions

Benchmark and Protocol: The release of a 12-year, 79-site California wildfire PM2.5 benchmark and the introduction of the LOIO evaluation protocol, which prevents incident-level leakage and directly simulates the prediction of new, unseen fire events.
First TSFM Benchmark for Extreme Wildfire PM2.5: The first systematic evaluation of time series foundation models (four zero-shot, two LoRA-fine-tuned) against fully trained deep learning baselines specifically for extreme, episodic wildfire smoke rather than routine urban pollution.
Counter-Intuitive Empirical Finding: The demonstration that a compact, fully trained BiLSTM consistently outperforms all TSFM configurations (zero-shot and fine-tuned) in both aggregate error and extreme-event exceedance detection.
Deployment Guidance: Practical recommendations clarifying that while zero-shot models offer a baseline, domain-specific supervised training remains essential for hazardous-level forecasting, and LoRA fine-tuning serves as a viable rapid-deployment path only when site-specific historical data is unavailable.

Significance and Claims

The paper challenges the prevailing assumption that architectural scale and large-scale pretraining automatically substitute for domain-specific training data when the target involves rare, extreme events. The authors claim that:

Generalization Limits: Large-scale pretraining on general corpora transfers only weakly to the heavy-tailed, non-stationary dynamics of wildfire smoke.
Evaluation Necessity: Standard chronological splits are insufficient for extreme-event forecasting; incident-level holdout is necessary to reveal model failures (such as the tail instability of Chronos-2) that would otherwise be masked.
Operational Reality: For sites with accumulated fire history, a small, fully trained recurrent model remains the most effective and efficient choice for exceedance-critical forecasting. Foundation models, even with fine-tuning, do not yet match this performance in the extreme regime.

The study concludes that while foundation models offer promise for rapid deployment in data-scarce scenarios, they currently do not replace the need for domain-specific supervised training when the objective is the reliable prediction of hazardous air quality events.