OpenGadget3 GPU solver tests

This paper presents a comprehensive evaluation of the OpenGadget3 GPU port, demonstrating excellent accuracy compared to its CPU counterpart across various cosmological and hydrodynamic tests while achieving chip-to-chip speedups of approximately 2–5 on four different supercomputers.

The Gist

Imagine you are trying to simulate the entire history of the universe, from the Big Bang to the formation of galaxies, inside a computer. This is a massive job. It involves tracking billions of tiny particles (representing dark matter and gas) and calculating how they pull on each other with gravity and how they crash into each other like a cosmic fluid.

For decades, scientists have used powerful computer processors called CPUs (the "brains" of a standard computer) to do this work. But recently, supercomputers have started using GPUs (Graphics Processing Units). Originally designed to make video games look pretty, GPUs are actually thousands of tiny workers that can do simple math tasks all at once, making them incredibly fast for this kind of simulation.

This paper is like a quality control report for a new version of a famous universe-simulation code called OpenGadget3. The team has rewritten parts of this code so it can run on these fast GPU workers instead of just the slow CPU brains. Their goal was to answer two questions:

1. Is it fast? (Does it actually save time?)
2. Is it accurate? (Does it give the same correct answer as the old, trusted version?)

Here is a breakdown of what they found, using some everyday analogies:

1. The "Double-Check" Test (Accuracy)

The scientists didn't just assume the new code worked; they ran it side-by-side with the old code on four different supercomputers around the world. They tested four different scenarios, ranging from simple to complex:

• The "Ghost" Test (Dark Matter Only): Imagine a room full of invisible ghosts pulling on each other. They ran this on the GPU and the CPU.
  • The Result: The maps of where the ghosts ended up were identical. If you looked at the "energy" of the simulation, the difference was less than 1%—basically invisible to the naked eye.
• The "Shock Wave" Test: Imagine a tube of gas where a sudden shockwave travels through it. This is a classic physics test.
  • The Result: The GPU and CPU produced almost the exact same wave pattern. The difference was so small (0.001%) it's like measuring the thickness of a human hair against the length of a football field.
• The "Galaxy Cluster" Test: They simulated a massive cluster of galaxies, first without stars (just gas and gravity) and then with "full physics" (including stars, black holes, and cooling gas).
  • The Result: Even with the complex chaos of stars forming and black holes growing, the GPU simulation matched the CPU simulation. The only tiny differences appeared in the very center of the clusters, which is expected because that area is so dense that even tiny timing differences can cause small ripples.

The Verdict: The GPU version is a perfect twin of the CPU version. It didn't introduce any "hallucinations" or errors.

2. The "Speed Demon" Test (Performance)

Now, let's talk about speed. The researchers measured how much faster the GPU was compared to the CPU.

• The "Specialist" Speedup: When they looked at just the hardest part of the math (calculating gravity between particles), the GPU was 3 to 5 times faster.
  • Analogy: Imagine a single chef (CPU) chopping vegetables. Now imagine a team of 50 chefs (GPU) chopping the same vegetables simultaneously. The job gets done much faster.
• The "Real World" Speedup: When they ran the full, complex simulation with all the extra physics (stars, gas, etc.), the total speedup was 2 to 3 times faster.
  • Why the drop? Even with a fast team of chefs, you still have to spend time passing ingredients back and forth, organizing the kitchen, and waiting for orders. In computing, this is called "overhead." The GPU is fast at the math, but the computer still has to manage the data, which slows the total time down a bit.

3. The "Scaling" Test

They also tested what happens when you add more computers to the job.

• Strong Scaling: If you have a fixed amount of work (a specific universe size) and you throw more GPUs at it, the job finishes faster. They found that even with huge resources, the system stayed efficient (over 80% efficiency).
• Weak Scaling: If you make the universe bigger (more particles) but also add more GPUs to handle it, the time to finish stays roughly the same. This is crucial for simulating the whole universe without waiting years for a result.

4. The "Future Roadmap"

The paper concludes that the current GPU version is a success, but there is room to get even better.

• More Tools: They plan to move more parts of the simulation (like how stars cool down or form) onto the GPU.
• Better Organization: Currently, the data is organized in a way that is easy for the old code to understand. In the future, they want to reorganize the data (like switching from a messy toolbox to a perfectly sorted parts bin) to make the GPU work even more efficiently.

Summary

Think of this paper as a mechanic saying: "We took a classic car (OpenGadget3), installed a brand-new, high-performance engine (the GPU porting), and test-drove it on four different tracks. The car drives exactly the same as before (accurate), but it gets to the finish line 2 to 3 times faster (speedup). We are ready to race."

The paper does not claim this will change how we treat diseases or build bridges; it strictly focuses on ensuring that our digital models of the universe are both fast and trustworthy.

Technical Summary

Technical Summary: OpenGadget3 GPU Solver Tests

Problem Statement
Contemporary High-Performance Computing (HPC) facilities are increasingly transitioning to architectures dominated by Graphics Processing Units (GPUs), with nine of the top ten Top500 systems utilizing accelerators. This shift necessitates the optimization of scientific codes to exploit GPU-based architectures. While previous work (Ragagnin et al., 2020) detailed the technical strategy for porting the N-body code OpenGadget3 to GPUs, a comprehensive evaluation of the accuracy of these ported modules was lacking. Specifically, there was a need to verify that the GPU implementation of short-range gravity, hydrodynamics, and thermal conduction produced results consistent with the classical CPU version across various cosmological and hydrodynamic scales.

Methodology
The authors conducted a series of tests to evaluate the accuracy and performance of the OpenGadget3 GPU port. The implementation supports both OpenACC and OpenMP-target directives, though the OpenMP-target version was primarily tested for Barnes-Hut interactions at the time of writing. The testing strategy involved a progressive inclusion of physical modules:

1. Dark Matter-Only (DMO) Simulation: A cosmological box simulation (Magneticum Box1a/mr) using only gravity to test the Barnes-Hut tree solver. This utilized the OpenMP-target porting on SuperMUC-NG2.
2. Hydrodynamic Shock Tube: A Ryu-Jones magneto-hydrodynamic (MHD) shock tube test to validate the Smoothed Particle Hydrodynamics (SPH) solver.
3. Non-Radiative Galaxy Cluster: A zoom-in simulation of a galaxy cluster including gravity, hydrodynamics, and thermal conduction, but excluding radiative cooling and star formation.
4. Full-Physics Galaxy Cluster: A zoom-in simulation incorporating the full suite of physics: gravity, hydrodynamics, thermal conduction, radiative cooling, star formation, stellar evolution, and supermassive black hole growth.

These tests were executed on four distinct supercomputing systems: Leonardo Booster (CINECA), MareNostrum-V (BSC), SuperMUC-NG2 (LRZ), and the LMU CIP cluster. The authors compared the GPU results against the standard CPU version, analyzing mass density maps, power spectra, thermodynamic profiles, and stellar mass fractions.

Key Contributions and Technical Adjustments
The paper details specific architectural changes made to adapt OpenGadget3 for GPU execution while maintaining scientific fidelity:

• Asynchronous Task Management: The primary phase of particle interaction is split into two asynchronous tasks. The CPU assembles target particle lists and communicates tree nodes, while the GPU executes local interactions. Guest particles are exchanged via CPU while the GPU processes local data.
• Drift Strategy Modification: To avoid data races inherent in the adaptive drift scheme of the CPU version (which uses OpenMP critical regions), the GPU version drifts all particles (active and non-active) at the onset of a timestep. This eliminates critical regions but introduces minor discrepancies in time-stepping.
• Neighbour Processing: To better exploit Single Instruction Multiple Threads (SIMT) parallelism, neighbour processing is performed in chunks of 32 particles rather than individually, leveraging existing infrastructure for combining guest-particle results.
• Memory Optimization: The port minimizes memory transfer by sending only read-required properties and downloading only computed properties, and by offloading the tree structure at each timestep.
• Thresholding: Timesteps with fewer than 1,000 active particles are executed entirely on the CPU to avoid suboptimal GPU performance in low-load scenarios.

Results

• Accuracy: The GPU implementation demonstrates excellent agreement with the CPU version across all test cases.
  • In the DMO simulation, mass content errors were approximately 0.02%, and power spectrum differences were sub-percent (<0.01%) down to the resolution limit (~200 ckpc).
  • In the shock tube test, relative density deviations were generally less than 0.001%, reaching 0.01% only near discontinuities.
  • In cluster simulations, gas density and temperature profiles matched within a few percent at large radii. Divergences in the cluster core were observed but attributed to the expected chaotic behavior of high-density regions and small timestep differences, consistent with intrinsic variability.
• Performance (Speedup):
  • Module-Level: Individual physical modules (e.g., Barnes-Hut gravity, SPH) achieved chip-to-chip speedups ranging from approximately 3 to 5.
  • System-Level: For complex setups involving multiple physical processes and communication overheads, the total chip-to-chip speedup (comparing GPU nodes to CPU nodes with the same core count) ranged from 2 to 3.
  • Scaling: Weak scaling tests on SuperMUC-NG2 and strong scaling tests on MareNostrum-V showed consistent behavior, with the GPU version maintaining over 80% efficiency up to a 32x resource increase.

Significance and Future Prospects
The paper establishes that the GPU porting of OpenGadget3 is scientifically valid, producing results indistinguishable from the CPU version for cosmological and hydrodynamic simulations, with negligible differences only on very small scales or in chaotic core regions. The implementation successfully bridges the gap between legacy CPU codes and modern accelerator-heavy HPC environments.

The authors note that while the current speedups are significant (2x–5x), further optimization is required. Future work includes:

• Porting additional modules (cooling, star formation, feedback) to the GPU.
• Expanding OpenMP-target support, particularly for non-NVIDIA GPUs.
• Refactoring data layouts from Array-of-Structures (AoS) to Structure-of-Arrays (SoA) or SoA-of-Structures (SoAoS) to improve vectorization.
• Optimizing tree-walk algorithms and increasing data persistence in GPU memory to reduce host-device transfer overheads.

The public release of OpenGadget3 is scheduled for the end of 2026 as a milestone for the European Centre of Excellence SPACE.