Extreme Weather Bench: A framework and benchmark for evaluation of high-impact weather

Technical Summary: Extreme Weather Bench (EWB)

Problem Statement
The rapid evolution of Artificial Intelligence (AI) weather models has outpaced the development of standardized verification frameworks capable of assessing their performance on high-impact weather events. Current evaluation practices for AI and Numerical Weather Prediction (NWP) models predominantly rely on global-scale metrics, such as Root Mean Squared Error (RMSE) and Anomaly Correlation Coefficient (ACC) on prognostic variables like 500mb geopotential height. These metrics often reward smooth fields rather than physically realistic weather patterns and fail to capture the "value" of a forecast to end-users. Furthermore, existing benchmarks often lack specific case studies for extreme phenomena or rely on reanalysis datasets (e.g., ERA5) which may contain biases and non-stationarity. This gap makes it difficult to determine if models are truly consistent, high-quality, and valuable for predicting events that cause significant societal harm, such as heat waves, tropical cyclones, and severe convective outbreaks.

Methodology
The authors introduce Extreme Weather Bench (EWB), a community-driven, open-source benchmark suite designed to evaluate both AI and traditional NWP models against a standard set of high-impact weather events. The methodology is structured around three core components:

1. Case Study Selection: EWB curates a dataset of events spanning 2020–2024, a period largely independent of the training data for many global AI models. The suite covers five categories of high-impact phenomena:

   • Heat waves
   • Major large-scale freeze events
   • Severe convective days
   • Tropical cyclones (TCs)
   • Atmospheric rivers (ARs)
     To address the "forecaster's dilemma" (where models might over-predict extremes to maximize scores), EWB includes a set of "marginal events" (e.g., non-extreme temperature days or days with low severe weather risk) for comparison.

2. Observation-Based Verification: Unlike benchmarks that rely solely on reanalysis, EWB prioritizes ground-truth observations.

   • Temperature: Uses hourly surface observations from the Global Historical Climatology Network (GHCNh).
   • Severe Weather: Utilizes storm reports from NOAA (US), the Canadian Severe Storms Laboratory, and manually curated reports from Australia to create "Practically Perfect Hindcast" (PPH) regions.
   • Tropical Cyclones: Uses IBTrACS data for track and landfall verification.
   • Atmospheric Rivers: Relies on ERA5 Integrated Vapor Transport (IVT) due to the lack of global, real-time, quality-controlled precipitation radar data, though the framework allows for future integration of datasets like IMERG/GPM.

3. Impact-Based Metrics: EWB moves beyond global RMSE by defining metrics tailored to specific hazards and end-user needs. These include:

   • Lead Time: How far in advance a major event is predicted.
   • Regional Error: Regional MAE and RMSE for temperature extremes (focusing on daily maximums and minimums).
   • Spatial Overlap: Intersection over Union (IOU) and Critical Success Index (CSI) for severe weather and AR footprints.
   • Displacement: Spatial and temporal errors for TC landfalls.
   • False Alarm Ratio (FAR): To assess over-prediction on marginal days.

The framework includes open-source Python code for event identification, tracking (including a new Python-based AR tracker), and metric calculation.

Key Contributions

• First Comprehensive High-Impact Benchmark: EWB is presented as the first open-source, community-driven suite specifically designed to verify consistency and quality for high-impact extreme events across both AI and NWP models.
• Diverse Observation Sources: The integration of ground-based observations (GHCNh, storm reports) alongside reanalysis data provides a more robust verification standard than reanalysis-only approaches.
• Impact-Oriented Metrics: The suite introduces metrics that directly address the "value" of a forecast, such as the ability to predict the timing of a heat wave or the spatial accuracy of a TC landfall, rather than just global field statistics.
• Mitigation of Evaluation Biases: By including marginal events, EWB provides a mechanism to detect and penalize models that over-predict extremes to game global scores.
• Open-Source Ecosystem: The code and data are publicly available to facilitate reproducibility and community expansion, aiming to become a standard similar to ImageNet for computer vision.

Results
The paper demonstrates the utility of EWB by evaluating state-of-the-art AI models (AIFS-Single, GraphCast, Pangu-Weather) against the ECMWF IFS HRES baseline across the curated event set:

• Heat Waves: While all models performed well on global RMSE, performance varied significantly by region. AIFS-Single was the only AI model to outperform HRES across all lead times for predicted event highs in the 2021 Pacific Northwest heat wave, though none outperformed HRES on event lows.
• Freeze Events: AI models generally struggled to predict event lows at longer lead times, indicating a warm bias. AIFS-Single showed a faster ability to correct this bias globally and in North America compared to other AI models.
• Convective Outbreaks: AI models generally outperformed HRES in Critical Success Index (CSI) and False Alarm Ratio (FAR) when evaluated against Practically Perfect Hindcast regions. However, they tended to over-predict risk on marginal severe days.
• Atmospheric Rivers: AI models generally outperformed HRES in CSI and spatial displacement for North America, though performance was mixed in Europe and Australia.
• Tropical Cyclones: AI models showed mixed results. They mostly outperformed HRES in pressure predictions for Western Hemisphere TCs but had higher wind speed errors in the Eastern Hemisphere. Early initialization times often resulted in larger track errors, though some models (e.g., GraphCast on TC Beryl) locked onto tracks earlier than others.

Significance and Claims
The authors claim that EWB represents a critical step toward making AI weather models "trustworthy" and "good" in the sense defined by Murphy (consistency, quality, and value). By shifting the focus from global statistical scores to specific, high-impact case studies and impact-based metrics, EWB enables:

1. True Model Comparison: Facilitating direct, apples-to-apples comparisons across different model architectures and types (AI vs. NWP).
2. Scientific Innovation: Providing a standardized testing ground that can drive improvements in model development, similar to how ImageNet advanced computer vision.
3. Community Growth: As an open-source project, it invites the global weather community to expand the event catalog (e.g., to droughts, floods, blizzards) and refine metrics, ensuring the benchmark evolves with the science.

The paper modestly acknowledges limitations, such as the reliance on ERA5 for atmospheric rivers due to data availability and the current focus on deterministic forecasts, with probabilistic extensions planned for future work. The ultimate goal is to establish EWB as a standard for evaluating the ability of models to predict the "tails of distributions" that have the most material impact on life and property.