Connecting the Dots: A Machine Learning Ready Dataset for Ionospheric Forecasting Models

Technical Summary: Connecting the Dots: A Machine Learning Ready Dataset for Ionospheric Forecasting Models

Problem Statement
Operational forecasting of the ionosphere faces critical challenges due to sparse observations, complex geospatial couplings, and the urgent need for accurate predictions to support Global Navigation Satellite Systems (GNSS), aviation safety, and satellite operations. While the availability of ionospheric observations has grown significantly, a major bottleneck remains: the lack of machine learning (ML)-ready datasets. Existing data products are heterogeneous in resolution and format, often sparse, and require extensive preprocessing before they can be utilized for training. This absence of standardized, ML-ready resources hinders the systematic comparison of models and slows the development of data-driven space weather research.

Methodology
To address these gaps, the authors present a curated, open-access dataset that integrates diverse ionospheric and heliospheric measurements into a coherent, modular structure. The methodology involves:

• Data Integration: The dataset harmonizes multi-source data spanning multiple modalities and temporal cadences. Key sources include:
  • Solar Drivers: Solar Dynamic Observatory (SDO) extreme ultraviolet flux embeddings, solar irradiance indices (F10.7, S10.7, etc.), and solar wind parameters (velocity, interplanetary magnetic field) from NASA OMNI2.
  • Geomagnetic Drivers: Activity indices (Kp, AE, SYM-H, Ap) and solar wind measurements.
  • Ionospheric Observations: NASA JPL's Global Ionospheric Maps of Total Electron Content (GIM-TEC) at 15-minute intervals (dense) and Madrigal sparse TEC maps at 5-minute intervals.
  • Geospatial Features: Orbital mechanics data (solar/lunar zenith angles, distances) and Quasi-Dipole coordinate transformations.
• Temporal and Spatial Alignment: The dataset is aligned to the temporal range of the SDO Foundation Model (May 13, 2010 – August 1, 2024). The authors implement a robust preprocessing pipeline to handle inconsistent cadences and missing values. Missing data is standardized as NaNs. A forward-filling strategy is employed for short gaps, with a defined "rewind time" (typically the native cadence, or 50 minutes for OMNI data) to prevent the propagation of stale data.
• Event Cataloging: A physics-based event catalog is generated using NOAA G-levels (derived from Kp indices) to classify geomagnetic storm durations. This classification ensures proper model validation by preventing data leakage where portions of the same storm event are scattered across training and validation sets.
• Model Benchmarking: Using this dataset, the authors trained and benchmarked several spatiotemporal ML architectures for forecasting vertical TEC under both quiet and active geomagnetic conditions. These models include an LSTM baseline, a Spherical Neural Operator (SFNO), and a GraphCast-inspired architecture.

Key Contributions

1. A Novel Heterogeneous Dataset: The primary contribution is a unified, ML-ready dataset that aligns sparse and dense TEC maps with solar and geomagnetic driver data, a combination not previously available in a single product.
2. Standardized Preprocessing Pipeline: The authors provide a codebase (publicly available on GitHub) that handles the alignment, normalization, and splitting of data, specifically addressing the challenges of missing values and non-standard encoding conventions found in raw space weather data.
3. Event-Based Validation Framework: The inclusion of a geomagnetic storm event catalog allows for rigorous model validation that respects the temporal continuity of physical events, mitigating data leakage issues common in time-series forecasting.
4. Baseline Performance: The paper demonstrates that models trained on this dataset (specifically the IonCast suite) outperform standard persistence forecasts, achieving accurate TEC predictions up to 12-hour lead times.

Results
The study reports that the IonCast models, trained on the 15-minute cadence aligned data product with JPL dense TEC maps as targets, successfully forecast vertical TEC. The models demonstrate superior performance compared to baseline persistence forecasts across varying geomagnetic conditions. The dataset and codebase are made publicly available on AWS S3 and GitHub to facilitate further research.

Significance
The paper positions this work as a foundational step toward building "digital twins" of the ionosphere. By providing a standardized, multi-modal dataset, the authors aim to unlock the full potential of data-driven space weather research. The work supports both scientific inquiry into Sun-Earth interactions and operational forecasting efforts, enabling the development and systematic comparison of next-generation forecasting models that were previously hindered by data fragmentation. The authors emphasize that creating robust ML-ready datasets is a necessary prerequisite for advancing the field beyond current operational frameworks.