🔬 physics

OpenMP GPU Acceleration and Portability of TRIMEG-C1 for Electromagnetic Gyrokinetic Simulations in Tokamak Plasmas

This paper presents a portable OpenMP-based GPU acceleration of the TRIMEG-C1 electromagnetic gyrokinetic code for NVIDIA and AMD architectures, demonstrating a ninefold speedup on AMD MI300A APUs while verifying the correctness of the implementation through Ion Temperature Gradient mode simulations.

Original authors: Giorgio Daneri, Zhixin Lu, Matthias Hoelzl, Luca Venerando Greco, Edoardo Carrà

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giorgio Daneri, Zhixin Lu, Matthias Hoelzl, Luca Venerando Greco, Edoardo Carrà

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a tokamak (a doughnut-shaped nuclear fusion reactor) as a giant, chaotic kitchen where super-hot plasma is being cooked. To understand how this plasma behaves without melting the pot, scientists use a complex computer program called TRIMEG-C1. This program simulates billions of tiny, invisible "particles" (like microscopic chefs) moving around, bumping into each other, and creating waves of energy.

For a long time, this simulation ran on standard computer processors (CPUs). It was accurate, but it was also incredibly slow—like trying to cook a feast for a million people using only a single, very slow spoon.

This paper is about giving that spoon a superpower upgrade: Graphics Processing Units (GPUs). GPUs are the same chips found in video game consoles, but they are built to do thousands of calculations at once, making them perfect for simulating millions of particles simultaneously.

Here is the story of how the researchers upgraded the code, the hurdles they faced, and the results they achieved.

1. The Challenge: Speaking Two Languages at Once

The researchers wanted to make their code run on two different types of "super-chefs" (GPUs):

NVIDIA: The dominant player in the market (like a famous, expensive brand of kitchen equipment).
AMD: A rising competitor that is cheaper and increasingly common in supercomputers.

The problem? The code was written in Fortran, an old but powerful language for science. Most tools for making Fortran code run on GPUs were designed specifically for NVIDIA. If they used those tools, the code would break on AMD machines. If they wrote separate versions for each, they would have to maintain two different codebases, which is a nightmare for software developers.

The Solution: They chose a tool called OpenMP. Think of OpenMP as a universal translator. It allows the scientists to write one set of instructions that says, "Do this calculation on the GPU," and the computer figure out how to translate that into the specific language of either an NVIDIA or AMD chip.

2. The Hurdles: Bumpy Roads on the Highway

While OpenMP was the right choice for portability, the road wasn't smooth. The researchers encountered several "potholes" caused by the fact that the compilers (the programs that translate code into machine language) for these GPUs were still in their "early childhood" stages.

The "Black Box" Problem: The code relied on a library (a pre-made toolbox) for doing complex math called "B-spline interpolation." This library used advanced features that the GPU compilers didn't fully understand yet. The researchers had to manually rewrite parts of this toolbox to make it compatible, essentially rebuilding the engine while the car was still moving.
The Memory Leak: On one type of GPU (NVIDIA), the code would suddenly freeze after running for a while. It turned out to be a "race condition"—imagine two chefs trying to grab the same ingredient at the exact same time, causing a deadlock. The researchers had to find and fix this invisible bug.
The "Overcrowded" Kitchen: When they tried to run too many simulations at once on a single GPU, the memory ran out. They had to reorganize how data was stored (flattening complex structures into simple lists) so the GPU didn't get overwhelmed.

3. The Results: A Speed Demon

Once the bugs were fixed and the code was optimized, the results were impressive.

The Speed Boost: On a specific AMD supercomputer (the "Viper" cluster), the new GPU version of the particle simulation was 9 times faster than the old CPU version. On a high-end NVIDIA machine (the "Pitagora" cluster), it was also significantly faster.
The "Oversubscription" Test: Usually, you want one GPU to handle one task. But in real supercomputing, resources are scarce. The researchers tested what happens if they force multiple tasks to share one GPU. Surprisingly, the code held up well, showing that it could still be efficient even when the GPU was busy juggling multiple jobs.
Accuracy Check: Speed is useless if the results are wrong. To prove the GPU version was trustworthy, they ran two famous test cases:
1. The Cyclone Case: A simplified model of plasma instability. The GPU results matched the CPU results almost perfectly (within a tiny margin of error, mostly due to the random nature of the simulation).
2. The TCV Case: A more realistic, complex model of a real-world fusion reactor. Again, the GPU version reproduced the physics correctly, capturing the growth of energy waves and the shape of the plasma.

4. The Takeaway

The paper concludes that they successfully built a portable version of a complex physics code. It's like creating a universal remote control that works on both Samsung and LG TVs without needing to buy two different remotes.

What they achieved: They made a slow, CPU-based simulation run 9 times faster on modern GPUs, and it works on both major hardware brands (NVIDIA and AMD).
What they didn't do: They didn't invent new physics or solve the energy crisis yet. They simply proved that the tool used to study fusion is now much faster and more flexible.

In short, the researchers took a heavy, slow-moving scientific simulation, gave it a turbocharger (GPUs), and ensured that the turbocharger works on any brand of engine, paving the way for faster and more detailed studies of how we might one day harness the power of the stars.

Technical Summary: OpenMP GPU Acceleration and Portability of TRIMEG-C1 for Electromagnetic Gyrokinetic Simulations

Problem Statement
The TRIMEG-C1 code is a triangular mesh-based gyrokinetic particle-in-cell (PIC) solver designed to simulate electromagnetic instabilities in tokamak plasmas. It utilizes a high-order $C^1$ finite element method to capture accurate physics with lower grid resolution compared to $C^0$ methods. However, as modern supercomputers increasingly rely on heterogeneous architectures, the code faces the challenge of scaling to multi-scale simulations involving small electron scales and transport time scales. While GPU offloading offers significant acceleration potential for PIC codes due to their high parallelism, existing approaches often lack portability. OpenACC, used in codes like GEM and ORB5, is well-supported on NVIDIA architectures but lacks robust support for AMD GPUs, which are increasingly prevalent in High-Performance Computing (HPC) clusters (e.g., the Max Planck Computing and Data Facility's Viper cluster). Conversely, CUDA or HIP require significant code rewriting or manual kernel translation. The primary problem addressed is achieving a portable, accelerated implementation of TRIMEG-C1 on both NVIDIA and AMD GPU architectures without compromising the existing Fortran codebase structure or requiring extensive code duplication.

Methodology
The authors selected the OpenMP offloading framework as the acceleration strategy. This choice was driven by its ability to support both NVIDIA and AMD GPUs within a unified Fortran codebase, minimizing structural changes and code divergence. The porting process focused on the most computationally intensive components: the particle pusher kernel and grid-to-particle (G2P) operations (including pullback treatment, density calculation, and distribution function interpolation).

Key methodological steps included:

Kernel Adaptation: The particle pusher, which evolves marker particles independently, was wrapped in a single large OpenMP target region to maximize parallelism. This required refactoring the code to ensure all called functions, including those from an external B-spline interpolation library, were device-callable.
Compiler Limitation Workarounds: Significant effort was dedicated to overcoming limitations in amdflang and nvfortran compilers regarding OpenMP offloading. Specific challenges included:
- Polymorphism and Type-Bound Procedures: Neither compiler fully supports these features in device code. The authors modified the code to use non-polymorphic type declarations instead of class where necessary and bypassed type-bound procedure bindings in the B-spline library.
- Memory Management: To handle dynamic arrays mimicking C++ std::vector, the authors flattened Array-of-Structures (AoS) data layouts into one-dimensional arrays to avoid complex runtime pointer mapping issues and Out-Of-Memory (OOM) errors on AMD MI300A APUs.
- Race Conditions: A race condition in the B-spline library's use of shared temporary arrays was resolved by switching to locally declared automatic arrays.
- Performance Optimization: On NVIDIA GPUs, the kernel initially suffered from performance degradation due to runtime allocation of temporary arrays. This was mitigated by switching to compile-time fixed-size arrays for specific functions, a change applied conditionally based on the target architecture.
Hybrid Parallelization: The study assessed a hybrid MPI-OpenMP offloading strategy, evaluating performance under resource oversubscription (running multiple MPI processes per GPU) to determine feasibility for large-scale simulations where node allocation is limited.

Key Contributions

Portable GPU Implementation: The paper presents the first successful port of the TRIMEG-C1 code to both NVIDIA and AMD GPU architectures using OpenMP, enabling execution on diverse HPC systems (e.g., Viper, Raven, Pitagora).
Resolution of Compiler-Specific Challenges: The work details specific workarounds for Fortran compiler limitations in OpenMP offloading, particularly regarding polymorphism, dynamic memory allocation, and external library integration.
Performance Benchmarking: Comprehensive performance evaluations were conducted across multiple clusters (AMD MI300A, NVIDIA A100, NVIDIA H100) and problem sizes, analyzing both individual kernel speedups and strong scalability.
Validation: The correctness of the GPU implementation was rigorously verified against CPU results using standard benchmark cases, ensuring that the porting process did not introduce physical inaccuracies.

Results

Speedup: For a realistic problem size of $32 \times 10^6$ marker particles on a full node, the particle pusher kernel achieved a speedup of approximately 9x on both the AMD MI300A (Viper) and NVIDIA H100 (Pitagora) compared to the fastest CPU-only implementation (AMD EPYC 9754).
Oversubscription: The study found that while oversubscribing GPU resources (increasing MPI processes per node) reduces the speedup of individual kernels due to serialization and context switching, the hybrid approach remains viable. However, the overall code speedup is heavily dependent on the workload size; for smaller workloads ( $10^6$ particles), the advantage diminishes significantly when comparing against a fully utilized CPU node due to data transfer overhead and limited arithmetic intensity.
Scalability: Strong scaling analysis demonstrated that the GPU-accelerated code scales well across multiple nodes, though the overall performance gain is marginal when the particle-to-grid (P2G) phase dominates, as this phase benefits significantly from the high core count of CPU-only nodes.
Correctness: Comparisons of the Ion Temperature Gradient (ITG) mode simulations for both the Cyclone base case and the TCV-X21 case showed good agreement between CPU and GPU versions. Energy growth rates differed by less than 3% for the Cyclone case and generally below 10% for the TCV-X21 case (with higher deviations at higher toroidal mode numbers). Two-dimensional mode structures were also preserved.
Profiling: Profiling on AMD GPUs revealed high register pressure limiting active Compute Units (CUs) and low utilization of Local Data Share (LDS), attributed to the OpenMP API's inability to explicitly manage shared memory.

Significance and Claims
The paper claims that the successful porting of TRIMEG-C1 to GPU architectures using OpenMP provides a portable computational foundation for future large-scale physics studies. The authors emphasize that while the implementation required significant code restructuring to navigate compiler limitations, the resulting code enables the simulation of electromagnetic instabilities on diverse hardware platforms without maintaining separate codebases for different vendors.

The work establishes that the particle pusher is the primary candidate for GPU acceleration, yielding substantial speedups for large particle counts. However, the authors modestly note that the overall code performance is currently bottlenecked by other phases (such as field equation solving and P2G operations) which are less amenable to GPU acceleration or benefit more from CPU parallelism. Consequently, the full potential of the GPU acceleration is realized in simulations with large particle counts where the pusher dominates the runtime.

Future work, as outlined by the authors, includes integrating GPU-enabled solvers via PETSc, exploring a C++ port of TRIMEG to improve offloading efficiency and reduce porting effort, and implementing GPU offloading for particle-to-grid operations. The current GPU-accelerated TRIMEG-C1 is positioned as a stepping stone toward multi-scale turbulence studies and investigations of plasma transport in realistic diverted tokamak configurations.

1. The Challenge: Speaking Two Languages at Once

2. The Hurdles: Bumpy Roads on the Highway

3. The Results: A Speed Demon

4. The Takeaway

More like this