CUBE2: A Parallel $N$-Body Simulation Code for Scalability, Accuracy, and Memory Efficiency

The paper introduces CUBE2, an open-source, memory-efficient, and scalable cosmological $N$-body simulation code that combines a multi-level Particle-Mesh method with optimized Green's functions and Information Optimized Storage to achieve high precision and performance, validated by successfully running $6144^3$ particle simulations on the ACECS supercomputer.

The Gist

Imagine trying to predict how a massive crowd of people will move through a city over 13 billion years. Now, imagine that instead of people, it's trillions of invisible particles (dark matter) interacting only through gravity, and instead of a city, it's the entire Universe.

This is the job of an N-body simulation. It's a super-computer game that tries to recreate the history of the cosmos, from the Big Bang to the formation of galaxies we see today.

The problem? The math is incredibly hard. If you have $N$ particles, calculating the gravitational pull between every single pair of them is like trying to count every handshake in a stadium of a billion people. It takes too long and requires too much memory.

Enter CUBE2. Think of CUBE2 not just as a piece of software, but as a super-smart, ultra-efficient traffic controller for the Universe. Here is how it works, broken down into simple concepts:

1. The "Zoom Lens" Strategy (Multi-Level PM)

In the old days, simulators tried to calculate the gravity between every single pair of particles everywhere. That's like trying to measure the distance between every person in a stadium with a ruler. It's slow.

CUBE2 uses a clever trick called a "Multi-Level Particle-Mesh". Imagine looking at the universe through a set of zoom lenses:

• The Wide Lens (PM1): For particles far apart, CUBE2 doesn't look at individuals. It looks at the "cloud" of matter as a whole, like seeing a city from a satellite. It uses a grid (a mesh) to smooth things out. This is fast and handles the big picture.
• The Medium Lens (PM2): As you zoom in, the cloud gets clumpier. CUBE2 switches to a finer grid to handle local neighborhoods.
• The Micro Lens (PM3 & PP): When particles get very close (like neighbors in the same house), the grid isn't enough. CUBE2 zooms in all the way and calculates the exact pull between those specific particles.

The Analogy: It's like navigating a city. You use a highway map for the long drive (Wide Lens), a street map for the neighborhood (Medium Lens), and you look out the window to avoid hitting a specific pothole (Micro Lens). CUBE2 seamlessly switches between these views so it never wastes time looking at a pothole when you're on a highway, or vice versa.

2. The "Memory Saver" (Information Optimized Storage)

Supercomputers have limited memory (RAM). Storing the position and speed of a trillion particles usually requires a massive amount of data, like trying to carry a library in your backpack.

CUBE2 uses a technique called Information Optimized Storage (IOS).

• The Analogy: Imagine you need to describe the location of a million people in a room. Instead of writing down their exact coordinates to the millimeter (which takes a lot of paper), you just write down which "tile" of the floor they are standing on, and then a tiny number for how far they are from the center of that tile.
• By using this "grid-relative" method, CUBE2 shrinks the memory needed by a factor of 4. This allows it to run simulations with trillions of particles on standard supercomputers that would otherwise crash.

3. The "Teamwork" Strategy (Scalability)

To run these simulations, you need thousands of computer processors working together. The challenge is making sure they all stay busy and don't wait for each other.

• Weak Scaling (Adding more people to a bigger job): If you double the size of the universe you are simulating, CUBE2 can double the number of computers and finish in the same amount of time. It's like having a construction crew where adding more workers to a bigger building doesn't slow down the project.
• Strong Scaling (Making a fixed job faster): If you have a specific simulation to run, CUBE2 can split the work so finely that adding more processors makes it finish much faster.
• The Analogy: Imagine a team of chefs in a kitchen. If one chef is stuck chopping onions (a heavy task) while others wait, the whole kitchen slows down. CUBE2 uses a smart "load balancing" system. It sorts the tasks by difficulty and assigns the heavy chopping to the chefs who are free, ensuring everyone finishes their shift at the same time.

4. The "Perfect Match" (Accuracy)

When you switch between your Wide Lens, Medium Lens, and Micro Lens, you have to make sure the numbers add up perfectly. If the math is slightly off at the boundaries, the simulation creates "ghost forces" that ruin the results.

CUBE2 uses a mathematical "glue" (an optimized Green's function) that ensures the force calculated by the grid matches the force calculated by the individual particles perfectly. It's like a tailor ensuring that the sleeve of a jacket fits perfectly into the body of the coat, with no gaps or bunching.

Why Does This Matter?

The authors tested CUBE2 by running two massive simulations (one with over 200 billion particles!) on a Chinese supercomputer. They checked the results against known theories and found that CUBE2 is:

1. Faster: It gets the job done in weeks instead of years.
2. Smaller: It fits in the memory of machines that couldn't handle the old codes.
3. More Accurate: The resulting "Universe" looks exactly like the real one, with the right distribution of galaxies and dark matter.

In a nutshell: CUBE2 is a new, open-source tool that lets scientists simulate the entire history of the Universe with incredible detail, using less computer power and time than ever before. It's the ultimate "cosmic time machine" for understanding how we got here.

Technical Summary

Here is a detailed technical summary of the paper "CUBE2: A Parallel N-Body Simulation Code for Scalability, Accuracy, and Memory Efficiency."

1. Problem Statement

Cosmological N-body simulations are essential for modeling the evolution of the Universe and studying large-scale structures (LSS). However, next-generation galaxy surveys (e.g., DESI, LSST, CSST) require simulations with massive particle counts ($N$) to achieve high mass resolution and large volume coverage. This presents four critical challenges:

• Memory Efficiency: Storing 6D phase-space coordinates for trillions of particles is a bottleneck.
• Scalability: Simulations must scale efficiently (both weakly and strongly) across thousands of computing cores.
• Accuracy: Approximations used to reduce computational complexity (from $O(N^2)$ to $O(N \log N)$) must not compromise the precision of gravitational forces.
• Portability: The code must run efficiently on diverse high-performance computing (HPC) architectures, including domestic Chinese supercomputers.

2. Methodology

CUBE2 is an open-source N-body code written in Coarray FORTRAN (CAF) with OpenMP directives, eliminating the need for explicit MPI communication. Its core methodology involves three main pillars:

A. Adaptive Multi-Level Force Calculation

CUBE2 decomposes the total gravitational force into a sum of a local Particle-Particle (PP) term and multiple Particle-Mesh (PM) terms ($M=3$ layers).

• Force Decomposition: The total reference force $R(r, b_{PP})$ is split into:
  • $R_{PP}$: Short-range forces calculated via direct PP interaction.
  • $R_1, R_2, R_3$: Long-range forces calculated via PM on global coarse, local fixed, and adaptive fine meshes, respectively.
• Optimized Green's Functions: To ensure seamless bridging between PM and PP calculations, CUBE2 derives optimized Green's functions ($G_\alpha$) in Fourier space. These minimize force errors caused by discretization, mass assignment (using Triangular Shape Cloud, TSC), and finite differencing.
• Softening: A specific softening kernel is used to suppress non-physical scattering at small scales, with the force vanishing below a specific radius.

B. Spatial Decomposition and Parallel Scaling

• Cubic Decomposition: The simulation volume is divided into a hierarchy of cubes (Nodes $\to$ Tiles $\to$ Subtiles). This minimizes the surface-area-to-volume ratio, reducing communication overhead.
• Global Scaling (Weak Scaling): The global PM1 calculation (the only $O(N \log N)$ part) uses a pencil decomposition for Fast Fourier Transforms (FFTs). Data is transposed from cubic to pencil format to allow local 1D FFTs, minimizing global communication.
• Local Scaling (Strong Scaling):
  • Nested OpenMP: The code uses a two-level parallelization strategy. Outer loops distribute subtiles among thread teams, while inner loops handle mass assignment, FFTs, and linked-list iterations within each team.
  • Load Balancing: Subtiles are sorted by local matter density ($\rho_{max}$) before assignment. High-density tasks are distributed first to ensure all threads finish simultaneously, preventing idle time.

C. Memory Optimization (IOS)

CUBE2 employs Information Optimized Storage (IOS) to drastically reduce memory footprint:

• Integer-based Coordinates: Particle positions and velocities are stored relative to a grid using 1-byte or 2-byte integers, rather than standard 4-byte floats.
• Efficiency: This reduces memory consumption from ~24 bytes per particle (standard) to 6–12 bytes per particle.
• Linked Lists: The ordering of particles in memory inherently serves as a linked list structure, saving additional memory usually required for neighbor searching.

3. Key Contributions

• Algorithmic Innovation: Introduction of a three-level adaptive PM scheme with optimized Green's functions, significantly improving force matching accuracy compared to previous two-level schemes.
• Architecture Agnostic Design: By using Coarray FORTRAN and OpenMP, CUBE2 achieves high portability across different HPC platforms (including Chinese self-developed chips) without relying on MPI.
• Extreme Memory Efficiency: The IOS technique enables the simulation of trillion-particle systems on modest hardware, a feat previously limited by memory constraints.
• Superior Scalability: Implementation of nested parallelization and density-based load balancing achieves near-perfect strong scaling (88% efficiency at 32 cores) and high weak scaling (94% efficiency at 512 nodes).

4. Results

The authors validated CUBE2 using two high-resolution cosmological simulations ($N = 6144^3$ particles) on the Advanced Computing East China Sub-center (ACECS):

• Force Accuracy: The total force error was reduced to a root-mean-square of 2% (maximum relative error ~7%), a two-order-of-magnitude improvement over its predecessor (CUBE). The force accurately recovers the reference potential across all scales.
• Cosmological Statistics:
  • Matter Power Spectrum: Results at $z=0$ match linear theory (CAMB) and nonlinear predictions (HMCode-2020) within cosmic variance.
  • Halo Mass Function (HMF): The HMF agrees with the fitting function by Warren et al. (2006) with residuals of only ~1% across orders of magnitude.
• Performance Metrics:
  • Weak Scaling: Increasing from 1 to 512 nodes (scaling $N$ proportionally) resulted in only a 6.8% increase in time per timestep (94% efficiency).
  • Strong Scaling: Increasing cores from 1 to 32 on a fixed problem size resulted in a 28.1x speedup (88% efficiency).
  • Runtime: The $6144^3$ simulations completed in 23.8 and 40.1 days, respectively.

5. Significance

CUBE2 represents a significant leap forward in cosmological simulation capabilities. By solving the trade-off between memory efficiency, computational speed, and physical accuracy, it enables:

1. Trillion-Particle Simulations: Making it feasible to run ultra-high-resolution simulations required for upcoming surveys like the China Space Survey Telescope (CSST).
2. Resource Efficiency: Allowing researchers to utilize modest or domestic computing platforms to compete with global supercomputing efforts.
3. Precision Cosmology: Providing a reliable tool for calibrating redshift-space distortions, reconstructing initial conditions, and interpreting cosmological parameters (e.g., neutrino mass, dark energy) with minimal systematic bias.

The code is open-source and available on GitHub, serving as a foundational tool for the next generation of large-scale structure studies.