Huracan: A skillful end-to-end data-driven system for ensemble data assimilation and weather prediction

Technical Summary: Huracan – A Skillful End-To-End Data-Driven System for Ensemble Data Assimilation and Weather Prediction

Problem Statement
Operational weather forecasting has traditionally relied on Numerical Weather Prediction (NWP), a computationally intensive process involving data assimilation (DA) and physics-based forecasting models. While recent machine learning (ML) models have demonstrated superior deterministic forecasting capabilities compared to NWP, they remain dependent on NWP-derived initial conditions (e.g., from ECMWF HRES or ENS). This dependency creates an upper limit on their forecast skill. Although a few end-to-end data-driven DA systems have been proposed (e.g., Aardvark, FuXi Weather, GraphDOP), none have yet matched the forecast skill of state-of-the-art NWP competitors when coupled with a prediction model. The challenge lies in effectively handling the "ill-behaved" nature of raw observation data (sparsity, misalignment, quality issues) and the complexity of DA processes (quality control, error estimation, bias correction) within a purely data-driven framework.

Methodology
The authors propose Huracan, an observation-driven, end-to-end system that integrates ensemble data assimilation with ensemble weather prediction, relying solely on observations as inputs.

• Data Sources: The system ingests diverse Level-1 and Level-2 observations, including microwave and hyperspectral infrared sounders (Metop, NOAA, JPSS), GNSS radio occultation, geostationary satellite radiance, and in-situ observations (radiosondes, land stations). The training dataset spans 2010–2022, with 2023 and 2024 reserved for validation and testing.
• Data Processing: Raw observations undergo quality control (removing implausible values), compression via autoencoders (for hyperspectral and radio occultation data), and interpolation to a unified 1-degree grid. Metadata such as viewing angles and time differences are included as input channels.
• Architecture: The system employs two models (assimilation and forecasting) sharing a modified Spherical Fourier Neural Operator (SFNO) architecture. Key modifications include:
  • Replacing Multi-Layer Perceptrons (MLPs) in SFNO blocks with Swin Transformer blocks to enhance modeling capability.
  • Aggressive compression of filter weights to reduce parameters by 12-fold and mitigate overfitting.
  • Integration of noise embeddings generated from random Gaussian noise to handle stochasticity, replacing standard layer normalizations with conditional layer normalizations.
• Training Strategy:
  1. A deterministic base model is trained on 36 years of ERA5 data using L1 loss.
  2. The base model is fine-tuned with noise injection and a Continuous Ranked Probability Score (CRPS) objective to create a stochastic base.
  3. Assimilation Model: Initialized from the stochastic base, it undergoes an 8-step autoregressive fine-tune using ERA5 ground truth and observation channels from a 15-year dataset.
  4. Forecasting Model: Undergoes a 6-step autoregressive fine-tune.
  5. Joint Fine-tuning: Both models are fine-tuned together (8 assimilation steps + 8 forecasting steps) to produce the final ensemble system.
• Baseline: A "Huracan-Hybrid" baseline is created by fine-tuning the forecasting model on ECMWF analysis and using ECMWF ENS initial conditions, serving as a strong comparison point.

Key Contributions

1. First End-to-End Ensemble System: Huracan is the first system to provide both ensemble initial conditions and end-to-end ensemble weather forecasts using only observations, completing the operational loop with uncertainty estimates.
2. Competitive Skill: The system achieves accuracy comparable to ECMWF ENS (the state-of-the-art NWP competitor) despite using a smaller volume of observation data and a 1-degree resolution.
3. Efficiency: The assimilation process takes a single model step, orders of magnitude faster than traditional NWP DA systems.
4. Performance Metrics: Huracan matches or exceeds the CRPS of ECMWF ENS on 80.2% of variable and lead time combinations (559 out of 697) when evaluated against each model's own analysis. Even against the stronger Huracan-Hybrid baseline, it performs as well or better on 65.1% of combinations.

Results

• Variable Performance: Huracan demonstrates robust skill in predicting temperature and humidity, often outperforming ECMWF ENS. It shows slight degradation in geopotential and pressure compared to NWP, particularly in the mid-troposphere.
• Ensemble Behavior: The system exhibits higher ensemble spread initially (due to independent member initialization rather than perturbation of a control member) but shows lower spread at 3–5 day lead times, potentially indicating overfitting.
• Visualization: Ensemble mean values align well with ECMWF ENS large-scale patterns, though Huracan's means are slightly smoother. Ensemble standard deviations are generally more spread out in Huracan compared to the concentrated, localized perturbations of ECMWF ENS.

Significance and Claims
The paper claims that Huracan represents a major step forward in end-to-end data-driven weather prediction. By successfully coupling ensemble data assimilation with forecasting, it demonstrates that ML-based systems can rival whole NWP systems in both initial condition generation and forecast accuracy. The authors assert that this work opens opportunities for further improving operational forecasting, potentially revolutionizing the field by removing reliance on NWP-derived initial conditions. However, the authors modestly acknowledge current limitations, including performance gaps on pressure/geopotential variables and a noticeable performance drop between training and test years (suggesting overfitting), which require further research. They also note that while the system learns biases inherent in ERA5 during training, the long autoregressive fine-tuning and noise injection help mitigate this, allowing the trained system to run independently of NWP indefinitely.