🔬 physics

Reinforcement learning for vertical position control on the EXL-50U spherical tokamak

This paper presents an experimentally validated reinforcement learning framework for vertical position control on the EXL-50U spherical tokamak, which achieves millimeter-scale tracking accuracy comparable to traditional PID controllers while consistently reducing actuator effort, thereby demonstrating the feasibility of learning-based control for future fusion systems.

Original authors: Lei Xing, Huicong Ma, Changquan Yu, Xuanhe Wang, Jiayi Zhi, Pei Guo, Mengyao Li, Zhengyuan Chen, Yapeng Zhang, Guoyang Shi, Dongkai Qi, Xiang Gu, Siqi Ding, Yong Liu, Jianguo Chen, Tianyuan Liu, the E

Published 2026-08-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lei Xing, Huicong Ma, Changquan Yu, Xuanhe Wang, Jiayi Zhi, Pei Guo, Mengyao Li, Zhengyuan Chen, Yapeng Zhang, Guoyang Shi, Dongkai Qi, Xiang Gu, Siqi Ding, Yong Liu, Jianguo Chen, Tianyuan Liu, the EXL-50U Teama

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quest for clean, limitless energy, scientists are trying to recreate the power of the sun here on Earth. They do this by trapping super-hot gas, called plasma, inside a magnetic cage shaped like a doughnut. This process, known as magnetic confinement fusion, requires the plasma to be held perfectly still. If the gas drifts even slightly off-center, it can hit the walls of the machine, cool down instantly, and the reaction stops. This is especially tricky in a specific type of machine called a spherical tokamak, which is shaped more like a cored apple than a flat doughnut. These machines are designed to be compact and efficient, but their unique shape makes the plasma naturally unstable, prone to shooting up or down like a balloon that won't stay in place. Keeping this floating fireball steady is the difference between a successful experiment and a sudden, violent crash.

For years, researchers have relied on standard, rule-based computer programs to keep the plasma centered. These programs act like a thermostat, constantly checking the position and making small adjustments to the magnetic fields. While they work well enough for some setups, they struggle when the plasma is stretched out or when the machine is pushed to its limits. In recent experiments on a machine called EXL-50U in China, these standard controls often left the plasma wobbling significantly, sometimes drifting by several centimeters. Such large movements are dangerous; they can damage the machine and prevent the use of advanced heating systems. To solve this, a team of scientists turned to a different kind of intelligence: machine learning. Instead of programming the computer with rigid rules, they taught it to learn how to control the plasma by practicing in a highly detailed virtual simulation, much like a pilot training in a flight simulator before ever touching a real plane.

The researchers built a digital twin of the EXL-50U machine, a virtual environment that mimics the real physics of the plasma and the electrical circuits with extreme precision. Inside this simulation, they tested a new type of controller based on reinforcement learning. This is a method where an artificial agent learns by trial and error, receiving a reward for keeping the plasma steady and a penalty for letting it drift. The team trained this digital agent on one specific experimental run and then challenged it to control the plasma on a completely different run, where the conditions were slightly different. They compared this learning-based approach against the standard rule-based controller and another advanced method known as a linear quadratic regulator, which relies on complex mathematical models.

The results from the simulation were revealing. The standard rule-based controller could keep the plasma on target, but it required the machine's magnets to work very hard, using a lot of electrical power to make constant corrections. The model-based approach struggled to stay steady when the conditions changed, often leaving the plasma in a slightly wrong position. The reinforcement learning agent, however, learned to guide the plasma with remarkable smoothness. It tracked the desired path just as accurately as the standard controller but did so with significantly less effort from the magnets. This efficiency is crucial because it means the machine can run longer and more stably without overloading its power systems. To ensure the learning agent could handle the inevitable differences between the virtual world and reality, the team added a simple correction mechanism that helped it adjust for small errors, a technique that proved vital for maintaining precision.

Encouraged by these virtual successes, the team took the trained artificial intelligence out of the computer and into the real machine. They deployed the learning controller on the actual EXL-50U tokamak, letting it take over control during live experiments. In more than ten separate shots, the system successfully kept the plasma vertically stable within the designated time window. In a direct comparison across seven of these shots, the learning controller matched the tracking accuracy of the standard controller, keeping the plasma within a few millimeters of its target. At the same time, it consistently used less electrical power to achieve this stability. This demonstrated that a machine learning system could not only survive the harsh environment of a fusion reactor but could also outperform traditional methods in terms of efficiency.

However, the experiment also highlighted the limits of the current technology. When the plasma current began to drop at the end of an experiment, the learning controller, which had been trained on steady conditions, started to lose its grip, and the plasma began to drift again. This showed that while the agent was excellent at handling stable phases, it still needed more robust tools to adapt to rapidly changing conditions. The researchers noted that future improvements would likely require the system to identify changes in the machine's behavior in real-time and adjust its strategy on the fly. Despite this, the successful deployment marks a significant step forward. It proves that learning-based control is a viable path for managing the complex, unstable physics of fusion, offering a practical route toward the more stable and efficient reactors needed to bring clean energy to the grid.

Technical Summary: Reinforcement Learning for Vertical Position Control on EXL-50U

Problem Statement
Vertical position control is critical for sustaining high-performance operation in spherical tokamaks, particularly as operations push toward higher plasma elongation. On the EXL-50U spherical tokamak, uncontrolled vertical displacement events (VDEs) constitute a primary cause of disruptions (accounting for 77% of disruptions in a recent campaign of 251 shots). While the existing Proportional-Integral-Derivative (PID) controller maintains millimeter-level stability for limiter plasmas, it struggles with divertor configurations, often exhibiting residual vertical oscillations of approximately 5 cm. These large motions degrade discharge reliability and impair radio-frequency coupling. The challenge lies in developing a controller that offers robust stabilization and reduced actuator effort under the stringent constraints of fast-growing vertical instabilities, actuator limits, and model-plant mismatches inherent in real-world fusion devices.

Methodology
The authors propose a reinforcement learning (RL)-based control framework validated through high-fidelity simulation and on-device experiments. The methodology is structured around four layers:

Plant Modeling: A high-fidelity, discharge-reconstructed simulation environment was developed. This uses a linear time-varying (LTV) state-space model derived from physics-based plasma–circuit equations (rigid-plasma RZIP model coupled with vessel eddy currents). The model is updated every 10 ms using experimental equilibrium data to track evolving operating points.
Controller Baselines: Two baselines were established for systematic comparison:
- PID: The existing operational model-free controller.
- LQR (Linear Quadratic Regulator): A model-based optimal controller synthesized from a reduced-order plant (two-state model capturing the unstable growth rate).
Primary Controller (RL): A reinforcement learning policy was trained using Proximal Policy Optimization (PPO). The agent interacts with the full-order high-fidelity plant, observing vertical displacement and selecting normalized voltage commands. The reward function encourages the reduction of absolute vertical error.
Sim-to-Real Remediation: To address residual model-plant mismatch and training-deployment differences, integral compensation was introduced as a "coarse remedy" layer. For LQR, this was implemented via state augmentation (LQRI); for RL, via an external integral term attached to the observation or policy output (RLI).

Key Contributions

High-Fidelity Simulation Environment: The establishment of a discharge-reconstructed simulation environment that integrates physics-based models with experimental equilibrium information, enabling systematic controller synthesis and sim-to-real evaluation.
Systematic Benchmarking: A rigorous comparison of PID, LQR, and RL controllers under identical plant dynamics, actuator constraints, and measurement imperfections.
Experimental Validation: The first deployment of an RL-based vertical position controller on the EXL-50U spherical tokamak, demonstrating stable closed-loop regulation across more than ten discharges.

Results

Simulation Performance:
- Tracking Accuracy: The RL policy achieved tracking accuracy comparable to the operational PID controller. Pure LQR failed to settle within the 5 mm error band due to model-plant mismatch, but the integral-augmented LQR (LQRI) restored stability.
- Actuator Effort: RL consistently required lower vertical-stabilization coil effort ( $V_{rms}$ ) than both PID and LQR.
- Integral Compensation: The addition of integral terms (LQRI and RLI) significantly improved performance. RLI achieved the lowest Integrated Time-weighted Absolute Error (ITAE) among all tested controllers while maintaining moderate actuator effort.
- Cross-Shot Transfer: A fixed RLI policy, trained on shot #17567, successfully stabilized vertical positions across four different reconstructed plants (shots #16235, #16911, #17287, #17567) without per-shot retuning.
On-Device Experiments:
- Stability: The RL controller successfully took over vertical control in more than ten discharges, maintaining stable regulation within the controlled windows.
- Accuracy: In seven representative discharges, RL maintained millimeter-scale tracking accuracy (Mean Absolute Error typically 1–5 mm), comparable to the operational PID controller.
- Efficiency: Across all seven compared shots, RL consistently reduced the actuator effort ( $V_{rms}$ ) compared to the PID baseline.
- Limitations: During the current ramp-down phase (e.g., shot #20177), where the plasma equilibrium changes rapidly, the frozen RL policy showed degraded tracking quality compared to the flattop phase, indicating a need for stronger adaptation mechanisms.

Significance and Claims
The paper claims to demonstrate the feasibility of learning-based plasma control on a real spherical tokamak. It establishes a practical pathway toward future fusion control systems by showing that RL can achieve performance comparable to established engineering baselines (PID) while reducing actuator stress. The work positions RL not as a replacement for all control logic, but as a primary controller augmented by integral compensation to handle mismatch. The authors explicitly note that while the current integral compensation is a "coarse entry," future work will require more robust designs, such as real-time system identification or robust synthesis (e.g., $H_\infty$ methods), to handle rapid phase changes like current ramp-downs. The study does not claim RL outperforms PID on all metrics in all conditions but highlights its favorable accuracy-effort trade-off and potential for systematic gain selection once a credible training environment is available.

Technical Summary: Reinforcement Learning for Vertical Position Control on EXL-50U

More like this