ARCANE -- Early Detection of Interplanetary Coronal Mass Ejections

This paper introduces ARCANE, a novel framework that achieves robust, early detection of interplanetary coronal mass ejections in streaming solar wind data by leveraging a ResUNet++ machine learning model, which significantly outperforms threshold-based baselines while maintaining high performance even with real-time data constraints.

The Gist

Imagine the Sun is a giant, temperamental lighthouse that occasionally sneezes massive clouds of charged particles into space. These sneezes are called Interplanetary Coronal Mass Ejections (ICMEs). When they hit Earth, they can act like a cosmic storm, potentially frying satellites, disrupting GPS, and knocking out power grids.

For decades, scientists have tried to build an automatic "storm alarm" that can spot these sneezes the moment they arrive at Earth's doorstep. But it's been like trying to identify a specific person in a crowd while wearing foggy glasses and only seeing them for a split second.

Enter ARCANE, a new "digital detective" introduced in this paper. Here is how it works, explained simply:

1. The Problem: The "Foggy Window"

Imagine you are standing at a train station (Earth) trying to spot a specific train (the ICME) coming down the tracks.

• The Old Way: Scientists used to wait until the entire train had passed the station to say, "Ah, that was the train!" By then, the storm has already hit.
• The Data Problem: Real-time data from satellites is often "noisy" or has gaps, like looking through a dirty window. Previous models were trained on "perfect" data (like a high-definition photo), so they got confused when looking through the "dirty" real-time window.

2. The Solution: ARCANE (The Early Bird)

The authors built ARCANE (Automatic Real-time deteCtion ANd forEcast). Think of ARCANE not as a person waiting for the whole train, but as a super-smart security guard who can identify a train just by seeing the very front of it (the engine) or even the smoke puffing out before it arrives.

• The Brain: ARCANE uses a type of Artificial Intelligence called a "ResUNet++." Imagine this as a highly trained eye that has looked at millions of solar wind patterns. It knows exactly what the "signature" of a storm looks like, even if the data is a bit messy.
• The Training: Instead of training on perfect, clean data, the team fed ARCANE the "real-world" messy data (NOAA's real-time feed). This is like training a firefighter in a smoky, chaotic room rather than a clean classroom. The result? ARCANE is much tougher and doesn't get confused when the real storm hits.

3. How It Works in Real-Time

Usually, to be sure a storm is coming, you might need to wait 8 hours to see the whole structure. ARCANE is designed to make a call much faster.

• The "Waiting Game": The researchers tested how long ARCANE needs to "wait" before making a decision.
  • If it waits 30 minutes, it catches about 71% of the storms, but it might raise a few false alarms (thinking a small gust is a hurricane).
  • If it waits 16 hours, it becomes a perfect detective, but by then, the storm has already arrived.
• The Sweet Spot: The paper found that ARCANE can detect the most dangerous storms with a reasonable delay (about 24% of the storm's total duration) while still being accurate enough to be useful. It's like spotting the dark clouds on the horizon and saying, "Storm's coming," rather than waiting for the rain to hit your face.

4. The Results: A Better Alarm System

The team compared ARCANE to an old-school method that just looks for simple numbers crossing a line (like a thermometer hitting a specific degree).

• The Old Method: It was like a motion sensor that goes off every time a cat walks by. It raised too many false alarms.
• ARCANE: It's like a facial recognition system. It knows the difference between a cat and a tiger. It successfully identified the "high-impact" storms (the tigers) that matter most for protecting our technology, while ignoring the smaller, harmless ones.

5. Why This Matters

This isn't just about better science; it's about early warnings.

• The Analogy: If you know a tornado is coming in 10 minutes, you can't do much. If you know it's coming in 30 minutes, you can put your kids in the basement. If you know it's coming in an hour, you can shut down the power grid to prevent fires.
• ARCANE gives us that extra 30 minutes to an hour of warning. It allows us to switch to "safe mode" for satellites and power grids before the worst of the storm hits.

The Bottom Line

The paper presents a modular, flexible framework that acts as a real-time weather radar for space. It admits it's not perfect (it still misses some small storms), but it is a massive leap forward in spotting the "big bad" solar storms early, using data that is actually available right now, not just data from a perfect lab.

In short: ARCANE is the early-warning system we've been waiting for to keep our technology safe from the Sun's temper tantrums.

Technical Summary

1. Problem Statement

Interplanetary Coronal Mass Ejections (ICMEs) are primary drivers of space weather disturbances, posing significant risks to technological infrastructure and human spaceflight. While manual identification of ICMEs in in situ solar wind data is possible, it is time-consuming, subjective, and inconsistent across different event catalogs.

Existing automatic detection methods face two major hurdles in operational settings:

1. Data Quality: Most models are trained on high-resolution "science-quality" data (e.g., OMNI), whereas real-time operational data (NOAA RTSW) is lower quality, noisier, and contains gaps.
2. Real-Time Constraints: Traditional methods often require observing the entire ICME structure (including the sheath and magnetic cloud) before making a classification. However, early warning systems require detection as soon as initial signatures (shocks, sheaths, or early magnetic obstacles) appear, without waiting for the full event to pass.

2. Methodology

The ARCANE Framework

The authors introduce ARCANE (Automatic Real-time deteCtion ANd forEcast), a modular machine learning framework designed specifically for streaming solar wind data under realistic operational constraints.

• Data Source: The study utilizes the NOAA Real-Time Solar Wind (RTSW) dataset (1998–2024), primarily from the DSCOVR spacecraft (with ACE as backup) at the L1 Lagrange point.
  • Parameters: Six independent variables: Magnetic field components ($B_X, B_Y, B_Z$), total magnitude ($|B|$), proton density ($N_p$), proton temperature ($T_p$), bulk velocity ($V$), and plasma beta ($\beta$).
  • Preprocessing: Data is resampled to a 10-minute resolution to balance temporal detail with data gap mitigation. Linear interpolation is used for gaps < 6 hours.
  • Labeling: Based on the HELIO4CAST ICME catalog (v2.3). Labels cover the entire ICME structure, including the sheath region, not just the magnetic cloud.

Model Architecture

• Core Model: A modified ResUNet++ architecture adapted for 1D time-series segmentation.
  • Unlike YOLO-based approaches that use bounding boxes, ResUNet++ performs pointwise segmentation, which is better suited for capturing the gradual onset of ICMEs in streaming data.
  • The model uses 1D convolutions, Batch Normalization, ReLU, Squeeze-and-Excitation blocks, and Attention mechanisms.
• Training Strategy:
  • Nested Cross-Validation: An outer loop splits data by year (1 year held out as test), and an inner loop splits the remaining years for training/validation. This prevents data leakage and ensures robustness across different solar cycle phases.
  • Loss Function: Weighted Dice Loss to handle class imbalance (ICMEs are rare events).
  • Sampling: Positive samples (last timestep in a window is an ICME) are weighted 10x higher than negative samples.

Evaluation Strategy: Early Detection

A novel evaluation protocol was developed to simulate real-time decision-making:

• Sliding Window: Data is processed in 1024-timestep windows (~7 days).
• Waiting Time ($\delta$): The model's prediction is extracted at various time steps within the window. $\delta$ represents the "waiting time" (how long the model observes a data point before classifying it).
  • $\delta = 0.5$ hours: Immediate classification.
  • $\delta = 16$ hours: Classification after observing a significant portion of the event.
• Metrics: Precision, Recall, F1-Score, and a new metric called Delay (time difference between the true start of the event and the first detection, adjusted for the waiting time $\delta$).
• Baseline: A threshold-based classifier derived from Lepping et al. (2005), adapted for real-time use with thresholds on $B_{max}$, $\beta$, and $T_p$.

3. Key Contributions

1. First Real-Time ICME Detection Framework: ARCANE is the first framework explicitly designed and evaluated for early ICME detection in streaming real-time data, rather than retrospective analysis of full structures.
2. Robustness to Data Quality: The study demonstrates that training directly on lower-quality real-time data (RTSW) yields better operational performance than training on high-quality science data and applying it to real-time streams.
3. Quantification of the "Early Detection" Trade-off: The paper systematically quantifies the trade-off between timeliness (low $\delta$) and accuracy (F1-Score). It shows that while performance improves with longer observation windows, useful detection is possible very early.
4. Modular Open-Source Framework: The code and data are publicly available, allowing for the integration of new data sources (e.g., Solar Orbiter) and future model improvements.

4. Results

• Performance vs. Baseline:

  • The ResUNet++ model significantly outperforms the threshold-based baseline.
  • At a waiting time of $\delta = 0.5$ hours:
    • ARCANE: F1-Score = 0.37, Precision = 0.25, Recall = 0.71.
    • Threshold Baseline: F1-Score = 0.18, Precision = 0.10, Recall = 0.71.
  • The baseline suffers from a high rate of false positives due to small-scale fluctuations exceeding static thresholds.

• Impact of Waiting Time ($\delta$):

  • As $\delta$ increases, Precision improves (fewer false alarms) while Recall remains relatively stable or slightly decreases.
  • Maximum F1-Score (0.48) is reached at $\delta = 16$ hours, suggesting that observing the full sheath region is necessary to perfectly distinguish CME-driven sheaths from other structures (like SIRs).
  • However, high-impact events (strong magnetic fields $>20$ nT, high velocities) are detected reliably even at low $\delta$.

• Event Characteristics:

  • The model successfully prioritizes high-impact events. False Negatives (missed events) are predominantly low-magnitude events ($|B| < 20$ nT, $V < 600$ km/s), which are less geoeffective.
  • The average detection delay is 24.1% of the event duration when processing only a minimal portion of the data.

5. Significance and Outlook

• Operational Readiness: ARCANE represents a substantial step toward automated, real-time space weather monitoring. It is already deployed as a prototype at the Austrian Space Weather Office.
• Strategic Trade-offs: The framework allows operators to tune the detection threshold. Lowering the threshold prioritizes early warnings (crucial for critical infrastructure) at the cost of precision, while raising it improves accuracy for confirmed events.
• Future Directions:
  • Integration: Combining ARCANE with physics-based propagation models (e.g., ELEvo) to refine arrival time predictions.
  • Severity Classification: Future catalogs need to include severity labels to help the model distinguish between minor and major storms.
  • Multi-Spacecraft Data: Integrating data from upstream monitors (e.g., Solar Orbiter, STEREO) to provide even earlier warnings before the ICME reaches L1.

In conclusion, ARCANE proves that deep learning models can effectively detect ICMEs in noisy, real-time data streams, offering a viable path toward reliable early warning systems for space weather hazards.