💻 computer science

Automating data splitting and hyper-parameters tuning in an Echo State Network-based model for Dynamic Aperture prediction

This paper investigates the impact of different data partitioning strategies on Echo State Network performance for predicting Dynamic Aperture in hadron storage rings, specifically exploring the automation of the training-test split boundary using the derivative of analytical scaling laws derived from the Nekhoroshev theorem.

Original authors: Quentin Bruant, Barbara Dalena, Massimo Giovannozzi, Maxime Casanova

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Quentin Bruant, Barbara Dalena, Massimo Giovannozzi, Maxime Casanova

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the future path of a swarm of bees inside a giant, twisting, invisible maze. In the world of particle physics, these "bees" are beams of protons zipping around a massive ring called a storage ring. The "maze" is made of powerful magnets that guide them. But here's the catch: the magnets aren't perfect. They have tiny wobbles and imperfections, just like a real-world maze might have a slightly crooked wall. Because of these wobbles, the bees don't just fly in perfect circles; they wiggle, dance, and sometimes, if they get too close to the edge, they crash into the walls and disappear.

Scientists call the safe zone where the bees can fly forever without crashing the "Dynamic Aperture." It's like the size of the safe playground inside the maze. Figuring out exactly how big this playground is usually requires running millions of computer simulations, checking every possible path the bees could take. It's so computationally heavy that it's like trying to count every grain of sand on a beach by picking them up one by one. To speed things up, researchers have started using a special type of AI called an "Echo State Network." Think of this AI as a super-smart student who watches a short video of the bees flying and then tries to guess where they will be a long time later. But for this student to be accurate, you have to show them just the right amount of training data—not too little, or they get confused; not too much, or they waste time.

This paper is about teaching that AI student how to learn more efficiently. The researchers wanted to figure out the perfect way to split their data: how much should the student study (training), how much should they practice on (validation), and how much should they be tested on (testing)? They discovered that by looking at how fast the "safe playground" is shrinking over time, they could automatically decide when to stop the simulations and let the AI take over. They also built a smart system that automatically picks the best settings for the AI, so scientists don't have to guess and check by hand. The result is a method that cuts down the time needed to simulate these particle beams by a factor of 20, making it much faster to design the next generation of particle accelerators.

The Story of the Smart Student and the Shrinking Playground

In the world of particle accelerators like the Large Hadron Collider (LHC), scientists need to know the "Dynamic Aperture" (DA). You can think of the DA as the size of the safety zone for a beam of particles. If the particles stay inside this zone, they circulate happily for hours. If they drift outside, they hit the walls and are lost. To find the exact size of this zone, physicists usually run massive computer simulations that track particles for millions of turns around the ring. This is incredibly slow and expensive in terms of computer power.

To solve this, the authors of this paper used a machine learning technique called an Echo State Network (ESN). Imagine the ESN as a student who watches a short clip of the particles moving and then tries to predict their path far into the future. In a previous study, the authors showed that this "student" could predict the future size of the safety zone quite well, especially when combined with a mathematical formula (called a Scaling Law) that describes how the zone shrinks over time.

However, there was a problem: the researchers had to manually decide how to split their data. They needed to give the student some data to learn from (training), some to practice on (validation), and some to test their final skills on (testing). If they gave the student too little data to learn from, the student would fail the test. If they gave them too much, they wasted precious simulation time.

The Big Discovery: The "Shrinking" Clue
The authors asked a simple question: Can we automate this decision? They noticed that the size of the safety zone doesn't shrink at a constant speed. At first, it drops quickly, like a ball bouncing down a steep hill. But after a while, the drop slows down, and the zone shrinks very gradually, like a ball rolling on flat ground.

The researchers found that the "speed" of this drop (the derivative of the scaling law) is the key. They discovered that if they stopped the simulations and let the AI take over exactly when the drop started to slow down, the AI's predictions were incredibly accurate. By looking at this "speed," they could automatically tell the computer: "Okay, you've simulated enough turns. Now, let the Echo State Network finish the job."

The Results: Speed and Accuracy
When they tested this idea on data from the High-Luminosity LHC (HL-LHC) and a mathematical model called the Hénon map, they found some great results:

The Magic Number: They determined that the AI needs to see about 30% to 35% of the total data to learn properly. Anything less, and the predictions get messy.
The 20x Speed-Up: By using this automatic splitting method, they could stop the slow computer simulations early and let the AI predict the rest. This made the whole process 20 times faster for the HL-LHC simulations.
Tiny Errors: Even when predicting far into the future (up to 1,000 times longer than the data they simulated), the error in their predictions was only about 1%. That's like guessing the distance to the moon and being off by only a few meters.

Teaching the Student to Choose Its Own Settings
The paper also tackled another headache: choosing the right "settings" (hyper-parameters) for the AI. Usually, scientists have to guess these settings, try them out, and hope for the best. The authors used a smart tool called Bayesian Optimization (specifically a method called Tree-structured Parzen Estimator) to let the computer find the best settings automatically.

They ran this optimization process 50 times to see how consistent it was. The results were very stable:

The average error stayed below 1%.
Even the "worst-case" scenarios (the most difficult simulations) rarely exceeded an error of 2%.
This proved that the AI doesn't just get lucky once; it consistently finds the right settings to make accurate predictions.

Why This Matters
This work doesn't just make the math look pretty; it solves a real bottleneck in designing future particle accelerators. By automating the data splitting and the tuning of the AI, scientists can now predict how their machines will perform over long periods without waiting days or weeks for computer simulations to finish. It's like giving the particle physicists a crystal ball that works 20 times faster than their old telescope, allowing them to design safer, more powerful machines for exploring the universe.

Technical Summary: Automating Data Splitting and Hyper-parameters Tuning in an Echo State Network-based Model for Dynamic Aperture Prediction

Problem Statement
The prediction of Dynamic Aperture (DA)—the extent of the phase-space region where bounded motion occurs in hadron storage rings—is critical for the design and optimization of particle accelerators like the LHC and HL-LHC. Traditional DA determination relies on computationally expensive numerical simulations over vast ensembles of initial conditions, often limited to short time scales (e.g., $10^6$ turns) due to CPU constraints, whereas operational time scales require extrapolation over much longer durations.

Previous work [1] established that combining Echo State Networks (ESN), a type of Reservoir Computing (RC), with analytical scaling laws (SL) derived from the Nekhoroshev theorem could significantly improve prediction accuracy and reduce computational costs. However, the performance of this hybrid approach depends heavily on how the dataset is partitioned into training, validation, and test sets, as well as on the specific hyper-parameters of the ESN. The current study addresses the lack of automation in these processes, aiming to determine optimal data splitting strategies and automate hyper-parameter tuning to ensure robust, reproducible results without manual intervention.

Methodology
The authors utilize two datasets: one generated from numerical simulations of the HL-LHC transverse beam dynamics (using 60 seeds to account for magnetic imperfections) and another based on a modulated 4D Hénon map. The core methodology involves:

Hybrid Modeling: The study employs a model where the DA evolution is predicted by fitting a Scaling Law (SL) to tracking data in the training and validation sets, while the ESN is trained on the same data to predict the DA for the test set (turns beyond the simulation limit). The final prediction combines the SL fit and the ESN output (SL+ESN).
Data Partitioning Analysis: The authors investigate the impact of varying the ratio of data allocated to training, validation, and test sets. They specifically analyze the derivative of the analytical Scaling Law ($dSL/dN$) at the boundaries of these sets. The hypothesis is that the derivative's behavior (transitioning from a steep drop to a gradual decrease) can serve as an automated metric to determine the optimal split point between the training/validation set and the test set.
Automated Hyper-parameter Optimization: To replace manual tuning, the study implements a Bayesian Optimization approach using the Tree-structured Parzen Estimator (TPE) algorithm via the hyperopt library. This optimizes six ESN hyper-parameters: reservoir dimension ( $N_r$ ), number of stacked reservoirs ( $L$ ), spectral radius ( $\rho$ ), discarded input data ( $B_I$ ), time discretization constant ( $\Delta t$ ), and the Tikhonov regularization coefficient ( $\beta$ ).
Ensemble Approach: Consistent with previous work, the study uses an ensemble of ESN realizations for each seed to mitigate algorithmic randomness, averaging the results to generate final predictions.

Key Results

Optimal Data Splitting: The analysis reveals that a training share of approximately 35% (and as low as 30% for the SL+ESN model) is sufficient to achieve a Relative Root Mean Square Error (RRMSE) in the test set of a few percent. The study identifies a "plateau" in performance between 30% and 45% training share.
Derivative-Based Automation: The absolute value of the SL derivative and its dispersion across the 60 seeds evolve monotonically with the training share. The authors demonstrate that a fixed threshold on the mean absolute derivative or its standard deviation can automatically identify the point where the data exhibits asymptotic behavior suitable for validation, effectively automating the split between training/validation and test sets.
Hyper-parameter Optimization: Using TPE-based Bayesian optimization over 50 independent runs, the study found that the SL+ESN model is highly robust. The median RRMSE for the SL+ESN model remained below 1%, with the maximum mean RRMSE across different hyper-parameter sets staying below 2%. In contrast, the ESN-only model showed slightly higher variability, though still performing well.
Performance Gain: The combined SL+ESN approach allows for the replacement of expensive tracking simulations in the test set with ESN predictions. For the HL-LHC simulations, this resulted in a speed-up factor of 20. The prediction accuracy for turns three orders of magnitude beyond the test set remained around 1% RRMSE.

Significance and Claims
The paper claims to provide a more efficient and robust procedure for applying machine learning to accelerator physics problems. Specifically, it establishes that:

Automation is feasible: The derivative of the analytical scaling law can be used to automatically determine the optimal data partitioning, removing the need for heuristic or manual selection of training/test boundaries.
Robustness of Hybrid Models: The combination of the Nekhoroshev-based Scaling Law with ESN predictions yields superior reliability compared to using either method alone, particularly in low-data regimes.
Automated Tuning: Bayesian optimization (TPE) successfully identifies optimal ESN hyper-parameters, providing a quantifiable estimation of uncertainty related to these parameters and confirming the stability of the surrogate models.

The authors conclude that these findings enable a more streamlined workflow for future applications to new accelerator configurations, ensuring that the computational savings (speed-up factors) are achieved without sacrificing the accuracy required for reliable long-term DA prediction.

The Story of the Smart Student and the Shrinking Playground

Technical Summary: Automating Data Splitting and Hyper-parameters Tuning in an Echo State Network-based Model for Dynamic Aperture Prediction

More like this