Parallelized Real-time Physics Codes for Plasma Control on DIII-D

This paper presents a real-time safe multi-threading library developed for the DIII-D plasma control system that successfully optimizes the execution of the TORBEAM and STRIDE physics codes to under 20 ms and 100 ms, respectively, enabling crucial electron cyclotron wave propagation and stability limit calculations for future fusion power plants.

The Gist

Imagine a fusion reactor like the DIII-D tokamak as a giant, super-hot, swirling storm of electricity (plasma) that needs to be held perfectly still inside a magnetic bottle. If the storm gets too wild, it can crash into the walls and destroy the machine. To keep it safe, a "Plasma Control System" (PCS) acts as the pilot, constantly making tiny adjustments.

However, the storm changes faster than a human can react. The pilot needs a super-fast computer brain that can predict the storm's behavior and suggest adjustments in the blink of an eye. This is where the paper comes in.

The Problem: The "One-Worker" Bottleneck

Imagine you are a chef trying to cook a massive banquet. You have a recipe (a physics code) that tells you how to cook the food. But you only have one chef (a single computer processor core) doing all the chopping, stirring, and baking. If the recipe is too complex, the chef gets overwhelmed, the food burns, and the banquet fails.

In the world of fusion, these "recipes" are complex physics simulations (like TORBEAM and STRIDE) that calculate how to heat the plasma or check if it's about to become unstable. Traditionally, these calculations were too slow to run in real-time because they were trying to do everything with just one "chef."

The Solution: A "Real-Time Safe" Team of Chefs

The authors built a new system to turn that single chef into a team of chefs working in perfect sync.

1. The Manager and the Workers: They created a special library (a set of rules) that acts like a Manager. The Manager hands out small, independent tasks to a group of Worker threads (other computer cores).
2. No Chaos, Just Order: In normal computer programs, when you add more workers, they might get confused, wait for each other too long, or crash the system if one makes a mistake. The authors' system is "real-time safe." It's like a military unit where every soldier knows exactly when to move and when to stop. They use a special "handshake" (atomic variables) to say, "I'm ready," "I'm done," and "Let's start the next round."
3. Deterministic Timing: The most important part is that this system guarantees it will finish its work within a strict time limit. It doesn't matter if the computer is busy with other things; this team is isolated and will always finish on time. This is crucial because if the computer takes too long, the plasma might already have crashed.

The Two Main Recipes They Cooked

The team used this new "multi-chef" system to speed up two specific physics codes:

1. TORBEAM: The Laser Beam Guide

• What it does: Imagine trying to hit a tiny, moving target inside a dark room with a laser. The plasma is the room, and the "laser" is a beam of energy (Electron Cyclotron Heating) used to control the plasma's stability.
• The Challenge: The computer has to calculate the exact path the laser beam will take through the plasma to hit the right spot.
• The Result: Because each laser beam (from different machines called gyrotrons) travels independently, the new system let the "workers" calculate the paths for all beams at the same time.
• The Speed: They got the calculation done in under 20 milliseconds. This is fast enough to steer the lasers in real-time, keeping the plasma stable.

2. STRIDE: The Stability Checker

• What it does: Imagine a tightrope walker. STRIDE is the safety inspector who constantly checks if the walker is about to fall. It calculates a "stability score" to see if the plasma is about to become unstable and crash.
• The Challenge: This calculation is very heavy and usually takes too long for real-time use.
• The Trick: The authors realized they could break the safety check into many small, independent pieces (like checking different sections of the tightrope). They sent these pieces to the "workers" to solve simultaneously, then combined the answers.
• The Speed: They reduced the calculation time to about 100 milliseconds. This is fast enough to warn the control system before a disaster happens.

The Bottom Line

The paper demonstrates that by building a specialized, highly disciplined team of computer processors (a "real-time safe multi-threading library"), they can run complex physics simulations fast enough to actually control a fusion reactor while it is running.

• TORBEAM (Laser steering) runs in ~20ms.
• STRIDE (Stability checking) runs in ~100ms.

Without this new "teamwork" system, these calculations would be too slow to be useful for keeping a fusion reactor safe and stable. This work proves that we can make the "pilot" of a fusion reactor smart enough to handle the extreme speed of the plasma storm.

Technical Summary

Technical Summary: Parallelized Real-time Physics Codes for Plasma Control on DIII-D

Problem Statement
Real-time plasma control systems (PCS) for tokamaks, such as those planned for future fusion reactors like ITER and SPARC, require physics-based codes to operate with deterministic latency and high robustness. While offline physics codes exist for electron cyclotron heating (ECH) ray tracing (TORBEAM) and ideal magnetohydrodynamic (MHD) stability analysis (STRIDE), their real-time implementations often face performance bottlenecks on single CPU cores. Existing multi-threading solutions, such as OpenMP, are unsuitable for fusion PCS environments because they lack hard real-time guarantees, rely on implementation-defined behaviors, and introduce non-deterministic latency jitter due to kernel interactions. Furthermore, GPU acceleration is often impractical for these specific codes due to high startup costs and insufficient parallelism granularity, necessitating a robust CPU-based parallelization strategy that ensures microsecond-scale determinism.

Methodology
The authors developed and implemented a custom, real-time-safe multi-threading library on the DIII-D PCS to enable parallel execution of the TORBEAM and STRIDE codes.

• Library Architecture: The library utilizes a "Manager-Worker" pattern implemented in C11. A single Manager thread (coinciding with the main PCS thread) assigns work units to multiple Worker threads. Communication is achieved exclusively through low-level atomic variables using the C11 acquire-release memory model, avoiding higher-level abstractions like semaphores that interact with the kernel.
• Synchronization Mechanism: The system employs a bidirectional synchronization protocol inspired by the Ada "rendezvous" feature. The Manager and Workers independently notify each other of status via atomic flags (Ready and Complete). This ensures that all threads reach synchronization points at the start and end of critical sections without undefined behavior. The design prioritizes determinism over maximum throughput, accepting that threads may spin idly while waiting for others to catch up.
• Hardware Environment: The implementation ran on systems with isolated CPU cores to prevent interference from other processes and kernel interrupts. The STRIDE implementation utilized a commercial real-time OS (RedHawk) to minimize latency jitter, while TORBEAM ran on a standard CentOS 7 environment with core isolation.
• Code-Specific Parallelization:
  • rt-TORBEAM: The ray tracing calculations for independent gyrotrons were parallelized. The code was simplified from a Gaussian beam model to a single-ray trace to calculate the maximum absorption location $(R, Z)$, which is then converted to a normalized flux coordinate ($\rho$) for mirror steering.
  • rt-STRIDE: The ideal stability problem was reformulated as a Riccati equation rather than a Newcomb equation. This allows the State Transition Matrix (STM) to be calculated via a "shooting" method where the integration domain is split into intervals. These intervals are distributed among Worker threads based on a "stiffness" metric (higher step density near rational surfaces), and the resulting STMs are combined by the Manager to compute the final stability parameter ($\delta W$).

Key Contributions

1. Real-Time Safe Multi-threading Library: The development of a deterministic, kernel-free threading library that enables safe parallelization on fusion PCS hardware, addressing the limitations of standard libraries like OpenMP.
2. Parallelized rt-TORBEAM: Successful deployment of a parallelized ray-tracing code for ECH mirror steering, capable of handling multiple gyrotrons simultaneously.
3. Parallelized rt-STRIDE: Implementation of a parallelized ideal MHD stability code using a Riccati formulation and interval splitting, enabling real-time calculation of stability limits.
4. Experimental Validation: Demonstration of these codes operating within the strict cycle time constraints of the DIII-D PCS during actual plasma shots.

Results

• rt-TORBEAM Performance: The code consistently executed in under 20 ms for four steered gyrotrons. The system demonstrated reasonable tracking of target $\rho$ values ($|\rho_{target} - \rho_{torbeam}| \le 0.05$). The cycle time remained constant as additional gyrotrons were added, provided a free worker thread existed for each. A spike in cycle time to ~40 ms was observed in specific edge cases, attributed to the physics code rather than the threading library.
• rt-STRIDE Performance: The code consistently calculated the $\delta W$ stability parameter in approximately 100 ms on a 72-core system. Profiling revealed that the STM integration itself took only ~20 ms, with the majority of the time consumed by serial data pre-processing (equilibrium to flux coordinate conversion).
• Tracking and Control: Experimental results showed that while the physics calculations were fast enough for control, the physical mirror hardware (taking ~0.5 s to move) and the speed of the real-time equilibria reconstruction limited the overall tracking performance, particularly for rapid changes in target values.

Significance and Claims
The paper claims that the developed multi-threading library is a critical enabler for next-generation fusion devices, allowing physics-based codes to meet the execution time requirements for real-time control. The authors emphasize that unlike machine learning surrogate models, which may lack robustness outside their training data, these physics-based codes offer "robust performance guarantees on day one of operation." The work demonstrates that with proper parallelization and deterministic threading, complex physics codes like TORBEAM and STRIDE can be integrated into the PCS to provide essential information on ECH deposition and ideal stability limits. The authors suggest this library can be applied to other real-time codes, including vertical stability analysis and profile prediction, and note that future efforts should focus on parallelizing data pre-processing steps to further reduce latency.