Developing Machine Learning-Based Watch-to-Warning Severe Weather Guidance from the Warn-on-Forecast System

This study develops and evaluates a machine learning framework using Warn-on-Forecast System output to predict 2-6 hour severe weather probabilities, demonstrating that histogram gradient-boosted tree and U-Net models outperform traditional updraft helicity baselines, with the former achieving superior metrics and the latter providing smoother spatial guidance.

The Gist

The Big Picture: Bridging the "Watch-to-Warning" Gap

Imagine you are watching a storm roll in. You have two types of information:

1. The Radar (Nowcasting): This is like looking out your window. You can see the rain right now and guess where it's going in the next 30 minutes. It's great for immediate action, but it can't see around the corner.
2. The Supercomputer (Long-term Forecast): This is like a weather model that predicts the general weather for the next 3 days. It's good for planning a picnic, but it's often too vague to tell you if your specific street will get hit by a tornado in two hours.

The Problem: There is a dangerous "blind spot" between 2 and 6 hours out. The radar can't see that far ahead, and the supercomputer is too fuzzy to be precise. This is the "Watch-to-Warning" gap. If a storm is forming, you need to know exactly where it will hit in the next few hours so you can issue a warning before it's too late.

The Solution: Teaching Computers to "Read the Tea Leaves"

The researchers used a powerful weather simulation system called WoFS (Warn-on-Forecast System). Think of WoFS as a high-tech weather simulator that runs 36 different scenarios at once (an "ensemble") to guess how a storm might behave.

However, WoFS is like a raw, unedited movie. It shows the storm moving, but it doesn't explicitly say, "There is a 70% chance of a tornado here." It just shows swirling winds and rain. Meteorologists usually have to squint at the data and guess the risk.

Enter Machine Learning (ML):
The authors trained two different types of "AI students" to look at the raw WoFS data and translate it into a clear, easy-to-read probability map (e.g., "60% chance of severe hail here").

They tested two different "students":

1. The HGBT (Histogram Gradient Boosted Tree): Think of this as a super-organized detective. It looks at thousands of specific clues (wind speed, humidity, rotation) and makes a decision based on a strict set of rules it learned. It's fast, logical, and very good at finding patterns in lists of numbers.
2. The U-Net: Think of this as an artistic painter. Instead of looking at a list of numbers, it looks at the whole picture (the map) at once. It understands how the storm looks as a shape. It's great at smoothing things out and seeing the "big picture" context.

The Experiment: The "Practice Run"

The researchers fed these AI students data from 108 real storm days between 2019 and 2023. They asked the AI to predict severe weather (tornadoes, giant hail, damaging winds) for the 2-to-6-hour window.

They compared the AI's predictions against a "Baseline" (a standard, non-AI method that meteorologists currently use).

The Results: The AI Wins, But with Different Styles

1. Both AI students were better than the Baseline.
Just like a seasoned coach is better than a rookie, both the "Detective" (HGBT) and the "Painter" (U-Net) were more accurate at predicting severe weather than the old-school method. They caught more storms and made fewer false alarms.

2. The Detective (HGBT) was the most precise.
The HGBT model was the best at getting the math right. It gave the most reliable scores. However, it had a quirk: it was a bit shy. Even when it was very sure a tornado was coming, it would only say, "There's a 60% chance." It rarely went higher than that.

3. The Painter (U-Net) was more dramatic and smooth.
The U-Net produced maps that looked smoother and more connected (less "pixelated"). It was also the only one brave enough to say, "There is a 100% chance of a tornado here!"

• The Catch: Sometimes the U-Net was a bit too confident (over-predicting), but its ability to give a "100%" warning is valuable when you need to tell people to get to a shelter immediately.

Why This Matters

This study proves that we can use AI to fill the dangerous 2-to-6-hour gap.

• Before: You might have to wait until the storm is right on top of you to get a specific warning.
• Now (and soon): We can use these AI tools to give you a "heads up" 2 to 6 hours in advance, telling you exactly which neighborhoods are in the danger zone.

The Bottom Line

The researchers built a new "translator" that turns complex, confusing computer weather models into clear, life-saving warnings. While the "Detective" AI (HGBT) is slightly more accurate, the "Painter" AI (U-Net) is better at giving a clear, high-confidence picture of the danger. Together, they help us get from "watching the storm" to "warning the people" much faster and more accurately.

Technical Summary

1. Problem Statement

Forecasting severe weather (tornadoes, damaging winds, large hail) in the "watch-to-warning" timeframe (0–6 hours prior to an event) remains a significant challenge. While Convection-Allowing Models (CAMs) like the Warn-on-Forecast System (WoFS) provide storm-scale guidance, they do not explicitly predict severe hazards. Forecasters currently rely on proxy fields (e.g., updraft helicity) and subjective heuristics, which lack probabilistic rigor and often fail to generalize across different severe weather regimes.

While Machine Learning (ML) post-processing of CAM output has shown promise for very short lead times (0–3 hours), its application to the 2–6 hour lead time window is underexplored. This specific window is critical for decision-making but poses a unique challenge: it is too long for traditional radar-based nowcasting yet requires higher spatial/temporal precision than typical synoptic-scale guidance.

2. Methodology

Data Source and Preparation

• Dataset: The study utilized 108 cases from the 2019–2023 NOAA Hazardous Weather Testbed Spring Forecasting Experiments (HWT-SFE).
• Model Input: Forecasts from the NSSL Warn-on-Forecast System (WoFS), a 36-member multi-physics ensemble updated every 15 minutes.
• Target: Probabilistic forecasts of "any" severe weather (tornado, wind >50 kts, hail >1 inch) within a 36 km radius of a grid point, valid for the 2–6 hour forecast window.
• Features:
  • 21 WoFS Fields: Including updraft helicity (UH), CAPE, shear, vorticity, and HAILCAST.
  • Preprocessing: Data was coarsened to a 9-km grid. Intrastorm variables were processed with maximum filters; environmental variables with uniform filters.
  • Observations: MRMS composite reflectivity was included as an input for the Deep Learning model but excluded from the tabular model.
• Split: 75 cases for training, 33 cases for testing (random date-based split).

Models Compared

The study compared two ML approaches against a calibrated baseline:

1. Baseline (WoFS NMEP): A Neighborhood Maximum Ensemble Probability (NMEP) derived from 2–5 km Updraft Helicity (UH), calibrated using isotonic regression.
2. Histogram Gradient-Boosted Trees (HGBT): A traditional ML method using scikit-learn's HistGradientBoostingClassifier. It utilized 175 input features derived from ensemble statistics across multiple spatial scales (9, 27, 45 km radii).
3. U-Net (Deep Learning): A convolutional neural network (CNN) architecture designed for image-to-image translation. It processed 63 input channels (including MRMS reflectivity) on a 96x96 grid. It features skip connections to preserve fine-scale details and uses a sigmoid activation for probability output.

Evaluation Metrics

Performance was evaluated using:

• Performance Diagram: Analyzing Probability of Detection (POD) vs. Success Ratio (SR).
• Normalized Area Under the Performance Diagram Curve (NAUPDC): To summarize overall skill.
• Normalized Critical Success Index (NCSI): Maximum CSI normalized for base rate.
• Reliability Diagrams & Brier Skill Score (BSS): To assess probability calibration.
• Statistical Significance: Determined via block bootstrapping (N=100) and permutation testing.

3. Key Contributions

• Extension of ML to 2–6 Hour Lead Times: This is one of the first studies to successfully apply ML post-processing to WoFS output specifically for the 2–6 hour "watch-to-warning" gap, a period previously dominated by less precise guidance.
• Architectural Comparison: It provides a direct comparison between a high-performance tree-based method (HGBT) and a deep learning architecture (U-Net) for severe weather probability forecasting.
• Probabilistic Calibration: The study demonstrates that ML models can generate well-calibrated probabilistic forecasts that outperform raw ensemble probabilities derived from physical proxy fields.

4. Results

Performance Metrics

• Superiority over Baseline: Both the HGBT and U-Net models significantly outperformed the calibrated WoFS UH baseline (p < 0.0001) in terms of NAUPDC and BSS.
• Threshold Sensitivity: The ML models showed the most significant improvement at higher probability thresholds (40–60%), where they achieved higher Success Ratios (SR) than the baseline.
• HGBT vs. U-Net:
  • HGBT: Achieved the highest overall skill scores (NAUPDC and BSS). It was the most well-calibrated model. However, its predicted probabilities were capped at approximately 60%.
  • U-Net: Produced spatially smoother guidance (typical of CNNs) and was capable of predicting probabilities up to 100%. While slightly less calibrated than HGBT (tending to overpredict in the 30–80% range), its ability to output high-confidence probabilities was noted as potentially valuable for decision-makers.

Spatial Characteristics

• The ML models generally captured the broad areas of severe weather identified by the baseline but provided more precise spatial localization.
• Displacement errors persisted in ML models, as they inherit underlying errors from the WoFS ensemble forecasts.
• The U-Net produced smoother fields, whereas the HGBT output was more discontinuous but often more spatially precise in high-probability cores.

5. Significance and Limitations

Significance

• Bridging the Gap: The study validates that ML can effectively fill the critical "watch-to-warning" gap, providing probabilistic guidance that is more skillful than current operational proxy-based methods for the 2–6 hour window.
• Operational Potential: The HGBT model is already running quasi-operationally, demonstrating the feasibility of integrating these tools into real-time forecasting workflows.
• Decision Support: The ability to predict probabilities up to 100% (U-Net) offers a new level of confidence for forecasters, potentially improving warning issuance and lead times.

Limitations & Future Work

• Input Disparity: The U-Net and HGBT did not use identical inputs (U-Net included MRMS; HGBT used multi-scale manual features), making a direct "apples-to-apples" comparison of architecture vs. data difficult.
• Generalizability: The dataset is limited to warm-season cases in the Central/Mid-Atlantic US; year-round or global generalization is unproven.
• Explainability: The study did not investigate why the models made specific predictions (feature importance or learned relationships).
• Ensemble Statistics: The optimal way to extract information from ensemble members for ML inputs requires further research.

In conclusion, this paper establishes that both traditional ML (HGBT) and Deep Learning (U-Net) are effective tools for post-processing CAM output to generate severe weather probabilities in the 2–6 hour lead time, offering a significant improvement over current baseline methods.