🔭 astrophysics

gevolution 2.0: GPU-accelerated relativistic N-body simulations for cosmology

This paper presents gevolution 2.0, a major overhaul of the relativistic N-body code that leverages a three-layer parallelization strategy (MPI, OpenMP, and CUDA) to fully exploit GPU acceleration for high-performance cosmological simulations, accompanied by a GPU-ready release of the underlying LATfield2 library.

Original authors: Julian Adamek, Øyvind Christiansen

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Julian Adamek, Øyvind Christiansen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the universe as a giant, invisible ocean. For centuries, scientists have tried to map the currents and waves in this ocean to understand how it formed and where it's going. Most of the time, they've used a simplified map called "Newtonian physics," which works great for everyday things like apples falling or planets orbiting. But when you zoom out to the entire universe, or look at the very large-scale structure of the cosmos, that simple map starts to blur. You need the full, complex rulebook of Einstein's General Relativity to see the true shape of space and time. The problem is, Einstein's math is incredibly heavy; it's like trying to solve a million-piece puzzle where every piece is constantly changing shape. To do this, scientists use supercomputers to run "simulations," which are basically virtual universes where they can watch galaxies form and interact. But these simulations have been slow and limited, often forcing scientists to choose between accuracy and speed.

Enter gevolution, a special computer program designed to simulate the universe using Einstein's full rules. Until now, it was like a high-end sports car that could only run on a very specific, slow track. The new paper introduces gevolution 2.0, a massive upgrade that turns this car into a rocket ship. The authors, Julian Adamek and Øyvind Christiansen, have completely rebuilt the engine to run on GPUs (Graphics Processing Units). You might know GPUs as the powerful chips inside gaming computers that make video games look amazing. In the world of science, these chips are like a swarm of thousands of tiny, super-fast workers who can all do math at the exact same time. By teaching gevolution to use these GPU workers, the team has made it possible to run these incredibly complex, Einstein-level simulations much faster than ever before, opening the door to understanding the "dark sector" of the universe—mysterious stuff like dark energy and dark matter that we can't see but definitely feel.

The Great Makeover: From One Worker to a Swarm

Think of the old version of gevolution as a construction crew where every single worker (a computer processor) had to wait for the person next to them to finish a task before they could start. It was organized, but slow. The new version, gevolution 2.0, is like hiring a massive army of specialized workers. The authors didn't just add a few more workers; they reorganized the entire job site. They now use three layers of teamwork:

MPI: This is the project manager that splits the universe into chunks and hands them out to different computers in a giant network.
OpenMP: Inside each computer, this is a foreman who makes sure all the CPU cores (the standard brain power) are working together efficiently.
CUDA: This is the real game-changer. It offloads the heavy lifting to the GPUs. Imagine the CPU cores as the brain that plans the route, while the GPUs are a fleet of race cars that actually drive the route at lightning speed.

The team had to rewrite the part of the code that handles "particles" (the virtual stars and galaxies). In the old version, the particles were like a messy pile of marbles that had to be sorted one by one. In the new version, they are organized into neat rows, allowing the GPU to grab and move thousands of them at once. It's the difference between a librarian sorting books one by one and a robot arm that can grab a whole shelf at a time.

The Race: How Fast is Fast?

To prove their new engine works, the authors put gevolution 2.0 through some serious tests on the "Alps" supercomputer in Switzerland. This machine is a beast, packed with Nvidia Grace Hopper 200 superchips (each with 72 CPU cores and a powerful GPU).

They ran a "strong scaling" test, which is like asking: "If we give this same-sized puzzle to more and more workers, how much faster do we finish?" The results were impressive. When they increased the number of computers, the simulation speed went up almost perfectly. The parts of the code that move particles around (the "kick" and "drift" steps) were so fast they barely slowed down, even with thousands of workers. However, they found a bottleneck: writing the results to the hard drive. It's like having a race car that can go 200 mph, but the finish line is a narrow gate that only lets one car through at a time. Even with the speed, the team had to wait for the data to be saved, which took up a lot of time.

They also compared gevolution 2.0 to two other famous simulation codes: PKDGRAV3 and Gadget-4.

PKDGRAV3 is like a master carpenter who uses a special tool to measure every tiny detail of a wood grain (adaptive resolution). It's very accurate for small, clumpy things but takes a long time.
Gadget-4 is a reliable, old-school truck that runs on standard fuel (CPUs only). It's sturdy but much slower than the new race cars.
gevolution 2.0 is the new electric race car. It uses a fixed grid (like a chessboard) rather than measuring every tiny grain.

The results showed that gevolution 2.0 was about 4.5 times faster than the old CPU-only version of itself. Compared to the other codes, it was roughly 20 times faster than PKDGRAV3 and nearly 200 times faster than Gadget-4 (when accounting for the number of computers used).

But speed isn't everything. The authors checked if the race car was accurate. They compared the "power spectrum" (a graph showing how clumpy the universe is) from their simulation against the others.

On large scales (big chunks of the universe), gevolution matched the others almost perfectly.
On medium scales, it was still incredibly close, differing by only about one part in a thousand.
On the very smallest scales (the tiniest details), gevolution started to lose a bit of accuracy because it uses a fixed grid. It's like trying to draw a circle with a ruler; you get a good shape, but the edges are a little jagged compared to a freehand drawing. The paper notes that for most scientific questions, this small loss of detail is a fair trade for the massive gain in speed.

Why This Matters

The main finding of this paper is that gevolution 2.0 is a fully functional, GPU-accelerated tool that can simulate the universe using Einstein's full laws of gravity at a speed previously thought impossible for such complex calculations. The authors show that while you can achieve massive speed gains, there is a trade-off: the fixed grid means you lose some accuracy on the very smallest scales compared to methods that adapt their resolution. However, for many scientific questions, this trade-off is acceptable, allowing researchers to run relativistic simulations that were previously too slow to be practical.

They are very sure about the performance numbers because they measured them directly on a real supercomputer. They are also confident that the code works for standard cosmology, but they suggest that this speed-up will be particularly useful for studying "exotic" models of the universe, like those involving new types of dark energy or fields that behave in weird ways. The paper doesn't claim to have solved the mystery of dark matter, but it provides a much faster, more powerful microscope to look for clues.

In short, the authors have taken a heavy, slow, and complex tool and turned it into a nimble, high-speed instrument. By doing so, they allow scientists to run more detailed, more accurate simulations of our universe, helping us get closer to understanding the invisible forces that shape everything we see.

Technical Summary: gevolution 2.0: GPU-accelerated relativistic 𝑵-body simulations for cosmology

Problem Statement
The paper addresses the increasing dominance of Graphics Processing Units (GPUs) in high-performance computing (HPC) and the need to adapt relativistic cosmological simulation codes to leverage this hardware. The original gevolution code, a relativistic particle-mesh $N$ -body solver, was designed for CPU-only execution using MPI for distributed memory parallelism. As modern HPC systems (e.g., the TOP500 list as of June 2026) are increasingly dominated by GPU-accelerated architectures, the lack of GPU support in gevolution limited its scalability and performance. Furthermore, solving equations for additional relativistic fields (such as scalar fields in extensions beyond $\Lambda$ CDM) is expected to scale well on GPUs, necessitating a code overhaul to utilize the most powerful available infrastructure.

Methodology
The authors present gevolution version 2.0, a major overhaul of the code's parallelization strategy to support three layers of parallelism:

MPI: For scalability across distributed memory systems.
OpenMP: For shared-memory parallelization on each MPI rank to utilize all available CPU cores.
CUDA: For offloading compute-intensive tasks to GPUs.

Key technical implementations include:

Hybrid CPU/GPU Memory Management: The code adopts a unified memory model where particle-mesh operations (projection, kick, drift) are executed on the GPU device using High-Bandwidth Memory (HBM). Fields defined in Fourier space remain in CPU host memory (DRAM) to optimize the use of limited HBM. Bulk data movement is restricted to initialization and specific stages of the 3D Fast Fourier Transform (FFT).
Particle Handler Rewrite: The particle handler was completely rewritten from scratch. The previous implementation used doubly-linked lists, which caused irregular memory access and were unsuitable for parallelization. The new handler bins particles into mesh rows, sorts them by coordinate, and processes them using CUDA thread blocks. It utilizes parallel sorting algorithms (cub::DeviceRadixSort) and templated callable structs to replace function pointers, allowing the compiler to generate specific kernels for different particle-mesh operations.
Mesh Updates and FFTs: The underlying LATfield2 library was updated with a generic lattice_for_each CUDA kernel for mesh operations. The 3D FFT implementation was modified to handle the new parallel layers. Most 1D FFTs are executed on the device using the cuFFT library. However, due to alignment requirements in cuFFT that cannot always be guaranteed with halo cells and multi-component fields, the code dynamically falls back to FFTW3 for specific 1D transforms where necessary.
Equations of Motion: The code solves the weak-field description of General Relativity in the Poisson gauge. It evolves metric perturbations ( $\psi, \phi, B_i, h_{ij}$ ) and particle trajectories using a staggered leap-frog scheme. The Hamiltonian and momentum constraints are solved via a predictor-corrector loop (often requiring zero iterations due to slow evolution), and the system supports an $N$ -body gauge option to recover Newtonian limits.

Key Contributions

GPU-Accelerated Relativistic Solver: The primary contribution is the first GPU-accelerated version of a fully relativistic $N$ -body code capable of handling scalar-vector-tensor decompositions of the Einstein equations.
LATfield2 GPU Backend: The authors release a GPU-ready version of the LATfield2 library, which provides the parallelization backend for gevolution.
Performance Benchmarks: The paper provides extensive benchmarks on the 'Alps' HPC platform (HPE Cray EX254n with Nvidia Grace Hopper 200 superchips), demonstrating strong and weak scaling capabilities.
Comparative Analysis: The code is benchmarked against Gadget-4 (CPU-only, tree-PM) and PKDGRAV3 (GPU-accelerated, tree code) in a pure Newtonian gravity context to establish performance baselines.

Results

Scaling Performance:
- Strong Scaling: For a simulation with $1080^3$ particles, the code achieves near-perfect scaling for particle updates (drift and kick). The distributed 3D FFT shows excellent scaling, though communication overhead becomes noticeable. Total performance is limited by I/O, particularly lightcone output, which shows minimal speed-up.
- Weak Scaling: The code demonstrates near-perfect scaling for particle updates and initial data generation. Metric solvers scale well, though I/O saturation degrades performance at the largest scales (e.g., $2400^3$ particles writing 4 TB of data).
Speed-up: Compared to a CPU-only run of gevolution 1.3 on the same hardware, the GPU implementation (2.0) achieves a performance gain of approximately a factor of 4.5 in terms of total node-hours consumed.
Comparison with Other Codes:
- gevolution 2.0 is approximately 20 times faster than PKDGRAV3 for the same Newtonian problem, attributed to PKDGRAV3's adaptive force resolution and time-stepping overhead.
- gevolution 2.0 is roughly an order of magnitude faster than Gadget-4 (which lacks GPU support).
- Accuracy: In terms of matter power spectra, gevolution agrees with Gadget-4 at large scales. On intermediate scales, gevolution and PKDGRAV3 agree within $\sim 0.1\%$ , while Gadget-4 deviates by up to 2% (attributed to tree-PM transition errors). At small scales, gevolution shows power suppression as it approaches the Nyquist limit of its uniform mesh, a known limitation of fixed-resolution particle-mesh schemes compared to adaptive tree methods.

Significance and Claims
The authors claim that gevolution 2.0 provides a fast and scalable computational tool for relativistic $N$ -body simulations, making it competitive with state-of-the-art $N$ -body methods. The paper emphasizes that the framework allows for the consistent treatment of arbitrary relativistic sources of stress-energy, which is crucial for modeling dynamical dark energy and other scenarios beyond the standard $\Lambda$ CDM model.

The authors modestly note that the "performance" advantage of the particle-mesh approach comes at the cost of fixed force resolution, which may be less accurate for resolving the internal structure of collapsed halos compared to tree codes. However, they argue that for scientific questions where the stated accuracy (within 2% up to specific $k$ -modes) is sufficient, the new GPU implementation offers a highly performant alternative. The ultimate goal highlighted is the ability to include nonlinear relativistic dynamics in the forward modeling of cosmological observables to better understand the dark sector.

The Great Makeover: From One Worker to a Swarm

The Race: How Fast is Fast?

Why This Matters

More like this