First Experimental Demonstration of Reinforcement Learning-Based Tuning on the PSI Injector 2 Cyclotron

Technical Summary: First Experimental Demonstration of Reinforcement Learning-Based Tuning on the PSI Injector 2 Cyclotron

Problem Statement
Reliable operation of high-power proton cyclotrons is a critical prerequisite for Accelerator Driven Systems (ADS) and large-scale scientific applications. Traditionally, beam tuning in these machines is performed manually, a process that is often slow, sub-optimal, and difficult to execute during faults or changing environmental conditions. While machine learning (ML) has shown promise in linear accelerators and light sources, its application to high-power cyclotrons remains largely unexplored. The nonlinear and operating-point-dependent nature of cyclotron beam dynamics—where responses to coil currents and RF voltages vary significantly with turn number and machine working point—renders fixed linear response matrices insufficient. The authors address the need for autonomous control strategies capable of compensating for routine operational drifts and fast fault scenarios within the stringent reliability requirements of ADS-class drivers.

Methodology
The authors developed and deployed a Reinforcement Learning (RL) framework on the Injector 2 cyclotron at the Paul Scherrer Institut (PSI). The system utilized the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm within a custom OpenAI Gym-compatible environment interfaced with the machine's EPICS control system via PyEPICS.

• State and Action Space: The observation vector included 14 continuous variables: eight phase probes (MIF1–MIF8), two loss monitors, extracted beam current, estimated turn number, and temperature readings. The action space comprised 14 continuous variables: twelve trim-coil currents, the main correction coil current (AIHS), and the voltage of resonator 3 (CI3V).
• Physics-Informed Adaptations: To bridge the gap between abstract policy learning and practical tuning, the authors introduced specific accelerator-physics adaptations:
  • Overshooting Strategy: A technique was implemented to accelerate magnetic field settling times. By applying a temporary overshoot to the main coil current, the settling time was reduced from approximately 60 seconds to 10 seconds, increasing the number of agent-environment interactions per training session by a factor of six.
  • Reward Shaping: A physics-motivated reward function was designed to balance beam quality, trim-coil usage, and safety. The reward penalized beam losses and excessive local corrections while prioritizing phase alignment, particularly at outer probes near extraction.
  • Interlock Handling: A layered safety logic was implemented to halt RL control and revert to a safe mode if beam losses exceeded thresholds or if current stability was compromised.
• Training Strategies: The campaign evaluated three initialization approaches: training from scratch (random weights), transfer learning (reusing policies from adjacent turn numbers), and pretraining using a data-driven surrogate model (Mixture Density Network) trained on 27 million samples of historical operational data.
• Experimental Campaign: The study spanned a 12-day campaign covering five distinct machine configurations (turn numbers 60, 72, 73, 74, and 89) representing nominal, reduced, and degraded operating modes. Training was conducted at a low beam current (20 µA) to minimize activation risks, followed by overnight autonomous evaluation and generalization tests at higher currents (up to 800 µA).

Key Results

• Convergence and Performance: The RL agent successfully tuned the machine across all five configurations. Stable policies were obtained within a few hours of online training (typically under 1,000 timesteps). The agent achieved phase alignment within ±1° to ±1.5° of the reference and reduced beam losses by nearly two orders of magnitude, often reaching sub-nA levels.
• Surrogate Pretraining: Pretraining with the surrogate model significantly accelerated convergence when the operational regime matched the training data (e.g., Turn 73 reduced training time from ~4 hours to ~2.3 hours). However, in degraded configurations with significantly altered dynamics (Turn 89), pretraining introduced bias and hindered performance, highlighting the need for regime matching.
• Generalization: Policies did not generalize directly across different turn numbers without adaptation, confirming the strong dependence of beam dynamics on the turn number. However, "warm-starting" from previous policies reduced adaptation time compared to training from scratch.
• High-Current Generalization: A policy trained exclusively at 20 µA successfully generalized to beam currents up to 800 µA without additional training. In all high-current tests, the agent restored optimal conditions in at most five steps without triggering interlocks, though tighter action constraints were required to manage increased sensitivity.
• Autonomous Operation: During overnight evaluation runs, the trained agents maintained stable operation for extended periods (up to 12 hours), autonomously compensating for environmental drifts and small perturbations without human intervention.

Significance and Claims
The paper claims this work constitutes the first experimental demonstration of RL-assisted tuning on a high-power cyclotron. The results demonstrate that RL can be reliably deployed for autonomous closed-loop tuning within strict safety boundaries, achieving convergence in hours even under varying machine conditions.

The authors emphasize that this work does not yet demonstrate recovery from major hardware failures on the timescale required for ADS (seconds), but rather establishes a critical intermediate step: autonomous compensation of operational drifts and perturbations over timescales of seconds to hours. The study validates the feasibility of using physics-guided RL environments for pre-experiment benchmarking and confirms that ML-based control can operate safely and effectively in real-world high-power accelerator environments. These findings provide a foundation for scaling the methodology toward the full HIPA complex and future ADS-class drivers, aiming to establish transparent, certifiable ML control architectures capable of sustaining long-term, high-current operation with minimal human intervention.