A Leakage-Controlled Time-Series Forecasting Benchmark for GOES Soft X-ray and Proton Flux

The Sun is not a steady, unchanging star. It is a dynamic engine that occasionally erupts, sending out bursts of radiation and high-energy particles that race toward Earth at the speed of light. When these eruptions strike our planet, they can disrupt the technology we rely on every day, from the satellites guiding our GPS to the power grids that light our cities. To protect these systems, scientists need to know not just that an eruption is coming, but exactly how it will behave: how fast it will rise, how intense it will become, and how long it will last. For decades, researchers have tried to predict these events using computer models, but a new study suggests that the way we have been testing these models has been hiding a critical flaw.

A team of researchers at the Korea Astronomy and Space Science Institute has built a new, stricter test to see if artificial intelligence can truly forecast the intensity of solar flares and the flow of energetic protons. Instead of simply asking a computer to guess whether a flare will happen or not, they asked it to predict the continuous, minute-by-minute rise and fall of radiation levels. They gathered data spanning nearly three decades, covering multiple cycles of solar activity, and carefully arranged the testing so that the computer could not access future information. This rigorous approach revealed a surprising truth: while some models are excellent at predicting the general, quiet background of solar activity, they largely fail to predict the sudden, dangerous spikes that pose the greatest risk to our technology.

The core of this work is a new benchmark, a standardized test designed to eliminate a common mistake known as "data leakage." In many previous studies, researchers shuffled their data randomly before training their models. This is like studying for a history exam by reading the answers before you see the questions. Because solar activity follows a timeline, shuffling the data allowed the models to learn patterns from the future that they would never have in a real-world scenario. The new benchmark fixes this by splitting the data into strict, non-overlapping time blocks. The models are trained on past data and tested only on future data, ensuring that any skill they show is genuine prediction, not memory. The researchers tested ten different modern deep learning models against two specific targets: the soft X-ray flux, which measures the intensity of solar flares, and the proton flux, which measures the flow of high-energy particles that can damage spacecraft.

When the researchers ran the tests, the results were a mix of success and sobering limitation. One model, a segment-recurrent network, proved to be the most accurate at predicting the overall shape of the radiation curves. It consistently produced the smallest errors when measuring the difference between its predictions and the actual data. However, this overall accuracy told only half the story. When the researchers looked specifically at the ability to predict the sudden, impulsive spikes that define a dangerous solar flare, the best-performing model fell short. It became so focused on getting the average, quiet background right that it smoothed out the sharp peaks, effectively missing the most critical moments. In fact, for predicting these sudden spikes, the best model's skill score sat on the "persistence floor," performing no better than a simple guess that the current conditions would just stay the same, with a score near 0.10.

This disconnect between overall accuracy and the ability to spot dangerous events is the paper's most significant finding. The researchers found that minimizing the average error actually encourages models to ignore the rare, extreme events because those events are so infrequent compared to the quiet background. A model that tries to be perfect on average will naturally drift toward predicting the calm, boring parts of the data, leaving the violent spikes unforecasted. Interestingly, this problem only appeared for the solar flares. When predicting the flow of energetic protons, which tend to persist for days once they start, the models that were good at the overall math were also good at spotting the events. The two types of solar activity behave differently: the flares are sudden and chaotic, while the proton flows are more like a slow, rolling wave that is easier to track.

The study also compared these new computer models against the current operational forecasts used by the National Oceanic and Atmospheric Administration. For the proton flows, the new models performed competitively with the existing human-led systems. But for the solar flares, none of the artificial intelligence models could outperform the more sophisticated baseline of assuming that if an event just occurred, another will follow. While the best model did perform nearly double the skill of a trivial "flat-value" guess (which assumes conditions never change), it still fell significantly short of the operational reference and the more realistic persistence baseline. Even the most advanced deep learning architectures, which have revolutionized many other fields, could not crack the code of predicting the exact timing and intensity of a flare more than a few hours in advance. The researchers noted that this difficulty might not be a failure of the computer models themselves, but a fundamental limit of the data; the sudden onset of a flare appears to be inherently unpredictable from the history of the radiation stream alone.

Ultimately, this work serves as a necessary reality check for the field of space weather forecasting. It demonstrates that a model can have a low error rate and still be useless for its intended purpose if it fails to capture the extreme events. The researchers have released their data, code, and testing framework to the public, inviting other scientists to try to solve this problem under the same strict rules. By removing the possibility of accessing future information and focusing on the specific skill of spotting dangerous spikes, they have created a clear target for the next generation of forecasting tools. The path forward, they suggest, may require new types of models that are specifically designed to value the rare, violent moments over the quiet average, or perhaps a shift toward using different kinds of solar data that might hold the keys to predicting these sudden eruptions. Until then, the ability to forecast the most dangerous solar storms remains an open challenge, one that requires us to look beyond simple accuracy scores to understand what our models are truly learning.