Towards a Fully Automated Pipeline for Short-Term Forecasting of In Situ Coronal Mass Ejection Magnetic Field Structure

This paper presents and evaluates a fully automated pipeline that integrates remote sensing, deep learning, and iterative flux rope modeling to provide real-time, short-term forecasts of the magnetic field structure of coronal mass ejections at L1, demonstrating promising accuracy while highlighting limitations due to event complexity.

The Gist

Imagine the Sun is a giant, fiery lighthouse that occasionally sneezes massive clouds of magnetic gas into space. These sneezes are called Coronal Mass Ejections (CMEs). When one of these clouds hits Earth, it can act like a giant electromagnetic hammer, potentially knocking out satellites, disrupting GPS, and even causing blackouts in our power grids.

For a long time, space weather forecasters have been like meteorologists trying to predict a storm, but with a major catch: they can tell you when the storm cloud will arrive, but they can't tell you how hard it will hit. It's like knowing a hurricane is coming, but not knowing if it will be a gentle breeze or a Category 5 disaster.

This paper introduces a new, fully automated system called NEXUS (think of it as a "Space Weather Autopilot") designed to solve this problem. Its goal is to not only predict when a CME will hit Earth but also to forecast the strength of its magnetic punch in real-time, without needing a human to sit at a computer and tweak the numbers.

Here is how NEXUS works, broken down into three simple steps using everyday analogies:

1. The Crystal Ball (ELEvo)

First, NEXUS looks at the Sun. When a CME is spotted, the system uses a model called ELEvo.

• The Analogy: Imagine throwing a pebble into a pond. You know the pebble's speed and the direction you threw it, but the water has currents and ripples that might slow it down or push it sideways. ELEvo is like a smart calculator that simulates how that pebble (the CME) will travel through the "ocean" of solar wind. It predicts: "Based on the speed and direction, this cloud will likely hit Earth in about 2 days."

2. The Motion Sensor (ARCANE)

Once the system knows a CME is coming, it waits. But space is vast, and sometimes the cloud misses us. NEXUS needs to know for sure when the cloud actually arrives.

• The Analogy: Think of ARCANE as a high-tech motion sensor in a hallway. Instead of a human watching a video feed, this is an AI "robot eye" that constantly scans the stream of solar wind data. As soon as the "magnetic obstacle" (the front of the CME) bumps into our sensors, the robot shouts, "It's here!" It can even tell the difference between the messy, turbulent front of the storm (the sheath) and the organized, dangerous core (the magnetic rope).

3. The Crystal Ball Re-Read (3DCORE)

This is the magic part. Once the storm hits, NEXUS doesn't just wait for it to pass. It starts guessing what the rest of the storm will look like.

• The Analogy: Imagine you are walking through a dark tunnel and you feel the first few seconds of a wind gust. You can't see the whole tunnel, but you can guess how strong the rest of the wind will be based on how hard the first gust hit you.
  • NEXUS uses a model called 3DCORE to do exactly this. As the first hour of the CME passes, the system builds a 3D model of the magnetic structure.
  • It then says, "Okay, the first hour was strong and twisted like a left-handed screw. Based on that, the next 10 hours will probably be even stronger."
  • Every hour, as new data comes in, it updates its guess, refining the forecast like a GPS that reroutes you as traffic changes.

What Did They Find?

The researchers tested this "Autopilot" on data from 2013 to 2025. Here is the verdict:

• It Works (Mostly): For about 61 specific events where the CME was a "clean" magnetic structure (like a perfect rope), NEXUS could predict the magnetic strength and timing with surprising accuracy just a few hours after the storm hit.
• The "Good Enough" Rule: The system is usually off by about 5 hours on timing and 10 units of magnetic strength. In the world of space weather, that's actually pretty good for a first guess!
• The Limitation: The system works best on "clean" storms. Many CMEs are messy, tangled, or deformed (like a crumpled piece of paper rather than a neat rope). When the storm is messy, the model struggles because it's trying to fit a square peg into a round hole.
• The "More Data" Myth: Interestingly, waiting for more of the storm to pass didn't always make the forecast much better. If the storm is fundamentally messy, seeing more of it doesn't help the model understand it better. The problem isn't lack of data; it's that the storm is too complex for the simple model to handle.

Why This Matters

Before NEXUS, forecasting the magnetic punch of a CME required a human expert to manually analyze data, a process that takes too long for real-time warnings.

NEXUS is the first system that runs entirely on its own. It connects the dots from the Sun to Earth, automatically deciding when to sound the alarm and how bad the hit will be. While it's not perfect yet, it proves that we can build a "self-driving car" for space weather.

The Bottom Line: We are moving from just knowing when the storm is coming, to knowing how hard it will hit, all without needing a human to press the buttons. It's a giant leap toward protecting our technology from the Sun's temper tantrums.

Technical Summary

1. Problem Statement

Coronal Mass Ejections (CMEs) are the primary drivers of severe space weather, posing risks to satellites, power grids, and navigation systems. While predicting the arrival time of CMEs has reached a plateau (typically with errors >13 hours), predicting the internal magnetic field structure (specifically the southward $B_Z$ component) remains the critical bottleneck for accurate geomagnetic forecasting.

Current operational challenges include:

• Complexity: CME structures are often distorted, non-ideal, and interact with the solar wind, making simple flux rope models insufficient.
• Human Dependency: Traditional flux rope reconstruction requires manual selection of intervals and initial parameters, preventing real-time, autonomous operation.
• Data Scarcity: Most CMEs are measured at only a single point in space (1 AU), limiting the ability to infer global structure.
• Need for Automation: There is a lack of fully automated pipelines that can process continuous data streams to provide short-term forecasts of the remaining magnetic profile once a CME is detected.

2. Methodology: The NEXUS Pipeline

The authors present NEXUS (Near-real-time Event detection and eXtrapolation using a Unified Space weather pipeline), a fully automated system that couples three distinct models to forecast CME magnetic fields at the L1 Lagrange point. The pipeline operates without human intervention once triggered.

A. Data Sources

• Remote Sensing: CCMC DONKI database for CME launch parameters (time, speed, direction).
• In Situ Data: NOAA Real-Time Solar Wind (RTSW) data (magnetic field, plasma density, temperature, speed) at 10-minute resolution.
• Ground Truth: HELIO4CAST ICMECAT catalog (used for validation).

B. The Three-Stage Architecture

1. Arrival Time Prediction (ELEvo):

   • Model: Elliptical Evolution (ELEvo), a drag-based propagation model.
   • Function: Takes CME parameters from DONKI to predict if an Earth impact is likely and estimates the arrival time window ($t_e \pm 2\sigma$).
   • Role: Defines the temporal search window for in situ detection, reducing false positives.

2. In Situ Event Detection (ARCANE):

   • Model: A deep learning model (Convolutional Neural Network) trained on solar wind data.
   • Function: Processes sliding windows of in situ data to classify the solar wind state into three classes: Background, Sheath, and Magnetic Obstacle (MO).
   • Role: Automatically identifies the onset of the CME sheath and the MO within the ELEvo-predicted window.

3. Iterative Reconstruction & Forecasting (3DCORE):

   • Model: 3DCORE, a semi-empirical flux rope model assuming a Gold-Hoyle configuration with elliptical cross-sections.
   • Function: Uses an Approximate Bayesian Computation - Sequential Monte Carlo (ABC-SMC) algorithm.
   • Process:
     • Once the MO onset is detected, the model fits the observed partial data (first 1–12 hours).
     • It generates an ensemble of parameter combinations to predict the remaining unobserved portion of the magnetic profile.
     • Reconstructions are updated iteratively (at 2, 3, 6, and 12 hours) as new data arrives.
     • A final reconstruction is performed using the full event duration for validation.

3. Key Contributions

• First Fully Automated Pipeline: NEXUS is the first system to integrate remote-sensing arrival prediction, deep-learning-based in situ detection, and iterative flux rope reconstruction into a single, autonomous workflow.
• Short-Term Forecasting Concept: The study introduces and validates the concept of "Short-Term Forecast," where the model predicts the future magnetic field profile based on only the initial fraction of the observed MO, rather than waiting for the full event to pass.
• Large-Scale Statistical Evaluation: Unlike previous studies relying on case-by-case analysis, this work evaluates the pipeline on 3,870 DONKI entries (2013–2025), resulting in 61 high-quality events with clear counterparts in the ICMECAT for rigorous statistical analysis.
• Operational Realism: The study strictly mimics real-time conditions by using only data that would have been available at the time of the event (e.g., excluding delayed catalog entries).

4. Results

The evaluation focused on 61 events where ARCANE detections, ICMECAT entries, and 3DCORE reconstructions aligned.

• Forecast Performance:

  • Accuracy: Forecasts based on the first few hours of MO data achieved performance comparable to full-event reconstructions.
  • Typical Errors:
    • Timing: $\sim$5 hours error in the timing of magnetic field extrema (e.g., min $B_Z$).
    • Magnitude: $\sim$10 nT error in field strength metrics.
  • Stability: The Root Mean Square Error (RMSE) for total field strength ($B_T$) and $B_Z$ remained relatively stable as more data was added, indicating that additional data does not significantly correct model errors.

• Systematic Biases:

  • Underestimation: The model systematically underestimates the maximum field strength ($B_T$) and overestimates the minimum $B_Z$ (making the southward field appear less severe).
  • Cause: This suggests that many CMEs deviate from the idealized flux rope assumptions (e.g., due to erosion, deformation, or complex interactions) and cannot be fully captured by the simplified 3DCORE geometry.

• Data Source Discrepancies:

  • Significant inconsistencies were found between DONKI arrival times, ICMECAT intervals, and ARCANE detections. Only a fraction of ELEvo-predicted arrivals had counterparts in all other datasets, highlighting the inherent subjectivity in defining ICME boundaries.

5. Significance and Implications

• Operational Feasibility: The study proves that fully autonomous, real-time short-term forecasting of CME magnetic structures is technically feasible.
• Model Limitations: The results indicate that for many events, the dominant error source is model inadequacy (the inability of a simple flux rope to represent complex ejecta) rather than a lack of observational data. Simply waiting longer for more data does not improve the forecast if the underlying physics model is too simple.
• Future Directions:
  • Upstream Monitors: Extending the pipeline to "sub-L1" monitors (e.g., at L5 or upstream of L1) to increase lead time.
  • Complex Models: Integrating deformable flux rope models or multi-point constraints to better handle non-ideal CME structures.
  • Geoeffectiveness: Coupling the magnetic field forecasts directly to geomagnetic indices (e.g., Dst, SYM-H) to provide direct impact assessments.

Conclusion: NEXUS represents a major step toward operational space weather forecasting, shifting the focus from "when will it arrive?" to "how severe will it be?" while highlighting the need for more sophisticated physical models to handle the complexity of real-world CMEs.