SCALE-TRACK: Asynchronous Euler-Lagrange particle tracking on heterogeneous computing architecture

The paper introduces SCALE-TRACK, an open-source, asynchronous Euler-Lagrange particle tracking algorithm designed for heterogeneous exascale architectures that achieves unprecedented scalability by successfully simulating up to 256 billion particles across 256 GPUs while maintaining accuracy and efficiency.

The Gist

Imagine you are trying to predict how a massive cloud of smoke behaves inside a giant room, or how billions of raindrops form and move inside a storm. To do this, scientists use a method called Euler-Lagrange simulation.

Think of it like this:

• The Eulerian part (The Room): Imagine the air in the room is a giant, invisible grid. We calculate how the wind, temperature, and pressure change at every single point in that grid. This is done by a team of workers (CPUs) standing at the grid points.
• The Lagrangian part (The Particles): Now imagine billions of tiny, individual raindrops or smoke particles flying through that room. Each one has its own path, speed, and temperature. Tracking every single one of these billions of particles is like having a separate worker follow every single drop.

The Problem:
In the past, doing this was incredibly slow and expensive.

1. The Bottleneck: The "room" workers (CPUs) and the "particle" trackers often had to wait for each other. The room workers would finish their calculations, shout the results to the particle trackers, wait for the trackers to finish, and then shout back. This waiting time wasted a lot of power.
2. The Traffic Jam: If you tried to do this on a supercomputer with thousands of processors, the communication between them became a traffic jam. Moving data back and forth slowed everything down.
3. The Limit: Most simulations could only handle about 1 billion particles before the computer gave up.

The Solution: SCALE-TRACK
The authors of this paper created a new software called SCALE-TRACK. Think of it as a highly efficient, asynchronous traffic control system for a massive city.

Here is how it works, using simple analogies:

1. The Asynchronous Dance (No More Waiting)

In old systems, the CPU and the GPU (a super-fast graphics card used for heavy math) were like two dancers who had to hold hands and move in perfect lockstep. If one stopped to tie a shoe, the other had to stop too.

SCALE-TRACK lets them dance independently.

• The CPU (the brain) calculates the wind and temperature of the room.
• The GPU (the muscle) tracks the billions of particles.
• Instead of waiting, the CPU says, "Here is the wind data from 5 seconds ago, go!" while the GPU keeps moving. The CPU then says, "Okay, here is the new wind data," and the GPU adjusts.
• The Magic Trick: To make sure the particles don't get confused by the "old" wind data, the software uses a clever predictor-corrector method. It's like a GPS that guesses where you are going based on your last speed, and then corrects the route once it gets the real traffic update. This keeps the simulation accurate even though the two parts aren't moving in perfect sync.

2. The Smart Neighborhoods (Chunking)

Imagine you have a billion particles. If you just throw them randomly into a room, some workers will have a million particles to track, while others have none. This is unfair and slow.

SCALE-TRACK uses a "Chunking" strategy.

• It groups particles into "chunks" (like neighborhoods).
• These neighborhoods aren't fixed. If a crowd of particles moves to the left, the neighborhood boundary moves with them.
• The Overlap: Sometimes, a neighborhood might overlap with another. This sounds messy, but it's actually smart. It means a particle doesn't have to be "handed off" to a new worker every time it crosses a line. It stays in its current "neighborhood" longer, reducing the number of times data has to be shipped across the network.

3. The Exascale Achievement

The team tested this on a local workstation (a powerful desktop computer) and a massive supercomputer called MareNostrum5.

• On the Desktop: They tracked 1.4 billion particles on a single graphics card. Before this, that would have required a massive supercomputer. It's like fitting a whole city's traffic into a single garage.
• On the Supercomputer: They scaled it up to 256 billion particles using 256 GPUs. That is 256 times more than the previous world records.

Why Does This Matter?

This isn't just about math; it's about real-world problems.

• Clouds: We can now simulate clouds with incredible detail to better predict weather and climate change.
• Engines: We can design cleaner, more efficient engines by seeing exactly how fuel droplets burn.
• Medicine: We can model how aerosol particles (like from an inhaler) travel through human lungs.

In a Nutshell:
The authors built a software bridge that allows the "brain" (CPU) and the "muscle" (GPU) of a computer to work together without stopping to chat constantly. By being smart about how they group particles and how they predict data, they unlocked the ability to simulate hundreds of billions of particles, turning what used to be a supercomputer-only task into something that can run on a powerful desktop, and pushing the limits of what is possible on the world's fastest machines.

Technical Summary

1. Problem Statement

Euler-Lagrange (EL) simulations are the standard for modeling disperse multiphase flows (e.g., clouds, sprays, sediment transport) where a continuous fluid phase interacts with a dispersed particle phase. While effective, these simulations are computationally expensive, particularly when two-way coupling is required (where particles influence the fluid via momentum, heat, and mass transfer).

Existing implementations on High-Performance Computing (HPC) systems face several scalability bottlenecks:

• Underutilization of Heterogeneous Architectures: Many codes either run both phases on CPUs (ignoring GPU power) or port both to GPUs (leaving CPUs idle).
• Synchronization Barriers: Traditional approaches often force the Eulerian (fluid) and Lagrangian (particle) solvers to wait for each other, causing significant idle time.
• Load Imbalance: As particles move freely, they can accumulate in specific regions, causing some processors to be overloaded while others remain idle.
• Communication Overhead: Transferring data between thousands of devices (CPUs and GPUs) for exascale problems creates massive communication bottlenecks.
• Scalability Limits: Current methods struggle to track more than $\approx 10^9$ particles, whereas exascale systems could theoretically handle orders of magnitude more.

2. Methodology

The authors propose SCALE-TRACK, a novel algorithm designed for heterogeneous CPU-GPU environments that addresses the above limitations through four key innovations:

A. Asynchronous Two-Way Coupling

Instead of synchronizing the fluid and particle solvers at every time step, SCALE-TRACK runs them asynchronously:

• CPU-GPU Offloading: The Eulerian fluid phase is solved on CPUs (using OpenFOAM), while the Lagrangian particle tracking is offloaded to GPUs.
• Non-Blocking Execution: While the CPU solves the fluid equations, the GPU simultaneously tracks particles using the most recent available fluid data.
• Extrapolator-Corrector Method: To handle the time lag between the fluid solver and particle tracker, the algorithm uses an extrapolation scheme.
  • It predicts the source term (force/heat/mass from particles) for the current time step based on previous steps.
  • Once the actual source term is calculated by the GPU, it is used to correct the Eulerian field.
  • The authors tested zero, constant, and linear extrapolators, finding that the constant extrapolator (using the last known true value) offers the best balance of accuracy and stability, reducing errors to the level of conventional synchronous methods.

B. Independent and Overlapping Domain Decomposition

SCALE-TRACK decouples the spatial partitioning of the fluid and particle domains:

• Chunk-Based Partitioning: Particles are grouped into "chunks" (Lagrangian partitions) that are independent of the Eulerian grid partitions.
• Dynamic Overlap: Unlike traditional methods where partitions are fixed, Lagrangian chunks can grow, shrink, and overlap. If a particle crosses a boundary, the chunk expands to include the new region rather than immediately transferring the particle to a neighbor. This drastically reduces particle-to-particle communication.
• Hilbert Curve Initialization: Particles are initialized using a Hilbert space-filling curve to ensure compact grouping, minimizing the number of Eulerian partitions a single Lagrangian chunk needs to query.

C. Cache-Friendly Data Structures

• Particles are stored in a Structure of Arrays (SoA) format rather than Arrays of Structures (AoS). This layout optimizes memory access patterns for GPUs, facilitating vectorization and coalesced memory reads.
• Bounding Boxes: Rectangular bounding boxes are generated around particle chunks to identify exactly which Eulerian partitions are needed, avoiding the transfer of unnecessary field data.

D. Implementation

• Language: Written entirely in Julia, leveraging its concurrency and parallel task capabilities (coroutines and threads).
• Solver Agnostic: While coupled with OpenFOAM for validation, the architecture allows low-effort coupling to any CFD solver.
• Hardware: Designed for nodes containing both CPUs and GPUs (e.g., NVIDIA H100s).

3. Key Contributions

1. Scalability Breakthrough: Demonstrated the ability to track up to 256 billion particles ($2.56 \times 10^{11}$) on 256 GPUs, a scale previously unattainable for two-way coupled EL simulations.
2. Asynchronous Algorithm: Proved that asynchronous coupling, when paired with a correction scheme, does not compromise accuracy compared to synchronous methods.
3. Local Workstation Capability: Showed that a single workstation GPU can simulate 1.4 billion particles (using 1.4 billion parcels), a scale typically requiring large HPC clusters.
4. Open Source: The code is released as open source, enabling broader adoption and further development.

4. Results

The paper presents validation and scaling results from tests on a local workstation and the MareNostrum5 HPC cluster:

• Accuracy Validation:

  • Compared against an analytical solution for momentum transfer, the asynchronous method with a constant extrapolator achieved relative errors comparable to conventional synchronous EL methods ($\approx 0.04\%$).
  • In a cloud chamber simulation (convection, heat, and mass transfer), SCALE-TRACK produced results nearly identical to OpenFOAM's built-in tracking, despite tracking 10 times more particles (1.4 billion vs. 0.14 billion).
  • Performance: SCALE-TRACK was 2.7x faster and 2.5x more energy-efficient than OpenFOAM for the same physical time, even with the higher particle count.

• Scaling Performance:

  • Strong Scaling: On a fixed problem size (24 million cells, 8 billion particles), the Lagrangian part scaled almost ideally up to 256 GPUs. The total time deviated from ideal scaling only when the Eulerian part (CPU-bound) became the bottleneck due to communication overhead at very high core counts.
  • Weak Scaling ("Semi"-Weak): With a fixed number of particles per GPU (1 billion), the system scaled nearly ideally up to 256 GPUs, successfully tracking 256 billion particles.
  • Efficiency: The asynchronous nature allowed the Eulerian and Lagrangian computations to overlap significantly, minimizing idle time.

5. Significance

SCALE-TRACK represents a paradigm shift in multiphase flow simulation:

• Exascale Readiness: It demonstrates that heterogeneous architectures (CPU+GPU) can be fully exploited for two-way coupled problems, overcoming the synchronization and load-balancing issues that have limited EL simulations for decades.
• High-Fidelity Simulations: By enabling the simulation of hundreds of billions of particles, it allows for high-fidelity modeling of complex natural phenomena (e.g., cloud microphysics, aerosol propagation) and industrial processes (e.g., spray combustion) that were previously too computationally expensive.
• Accessibility: The ability to run billion-particle simulations on a single workstation democratizes high-fidelity multiphase flow research, reducing the barrier to entry for researchers without access to massive supercomputers.
• Future Applications: The framework is agnostic to the fluid solver and can be extended to include collision models, unstructured grids, and dynamic load balancing, making it a versatile tool for future exascale applications.