LEDDS: Portable LBM-DEM simulations on GPUs

The paper introduces LEDDS, an open-source framework that achieves high-performance, portable LBM-DEM simulations on GPUs by expressing complex fluid-particle interactions entirely through algorithmic primitives, delivering performance comparable to hand-tuned CUDA solvers while maintaining code clarity and broad hardware compatibility.

The Gist

Imagine you are trying to simulate a massive, chaotic dance party where thousands of tiny dancers (particles) are moving through a thick, invisible crowd (fluid). Some dancers bump into each other, some slide along the walls, and the fluid pushes them around while they push back.

Simulating this on a computer is incredibly hard. Usually, to make it fast enough to run in a reasonable time, scientists have to write very specific, complex code for the computer's graphics card (GPU). It's like hiring a different, specialized orchestra for every single instrument in the room. If you want to play on a different type of computer, you have to rewrite the entire sheet music.

Enter LEDDS.

The paper introduces LEDDS (LBM-Enhanced Device-independent DEM Solver), a new way to run these simulations. Think of LEDDS not as a custom-built orchestra, but as a set of universal building blocks (like LEGO bricks or standard kitchen tools) that can be used to build the simulation on any computer, whether it's a standard laptop or a super-powerful graphics card.

Here is a breakdown of how it works, using simple analogies:

1. The Two Main Characters

To understand the simulation, you need to know the two "actors" involved:

• The Dancers (DEM - Discrete Element Method): These are the solid particles (like sand grains or marbles). They bounce, roll, and collide.
• The Crowd (LBM - Lattice Boltzmann Method): This is the fluid (like water or air) flowing around the dancers.

In the past, making these two interact on a fast computer required writing "low-level" code—essentially speaking the computer's native, difficult language. LEDDS changes the game by using High-Level Primitives.

2. The "Recipe" Approach (Algorithmic Primitives)

Instead of writing a custom recipe for every single step, the authors decided to describe the whole simulation using a few standard, high-level instructions, like:

• Map: "Do this specific thing to every single dancer at the same time."
• Sort: "Line everyone up from shortest to tallest."
• Reduce: "Add up all the scores to get a total."
• Unique: "Remove the duplicate names from the list."

The Analogy: Imagine you are organizing a massive school event.

• Old Way: You hire a different team of people to handle the lunch, the seating, and the music, and each team speaks a different language. If you change the venue, you have to hire new teams and translate everything.
• LEDDS Way: You give everyone the same standard checklist (Map, Sort, Reduce). You tell the "Map" team, "Check every student's ID." You tell the "Sort" team, "Arrange them by grade." Because everyone follows the same standard checklist, you can swap out the teams (or the computers) without changing the instructions. The result is the same, but it's much easier to manage and works on almost any machine.

3. The "Partially Saturated" Trick

One of the hardest parts of this simulation is figuring out where the solid dancers end and the fluid crowd begins.

• The Problem: If a marble is half-in and half-out of a water cell, is it water or rock?
• The LEDDS Solution: They use a method called Partially Saturated Cells (PSM). Imagine a grid of boxes. If a marble covers 30% of a box, that box is 30% "rock" and 70% "water." The computer doesn't need to redraw the walls of the box every time a marble moves; it just adjusts the "mix" inside the box. This keeps the simulation smooth and fast.

4. Why is this a Big Deal?

The authors tested LEDDS against the "gold standard" of these simulations (a program called waLBerla that uses highly specialized, hand-tuned code).

• The Result: LEDDS was almost as fast as the specialized code (within a factor of two), but it was portable.
• The Metaphor: It's like driving a car that runs on any fuel (gas, diesel, electric) and can be driven on any road, yet it still goes just as fast as a Formula 1 car that only works on a specific track with a specific fuel.

5. What Can It Do?

The paper shows LEDDS handling:

• Bouncing Balls: Simulating perfectly elastic collisions where energy is conserved.
• Sand Piles: Watching how sand made of weirdly shaped grains (ellipsoids) forms a pile and calculating the "angle of repose" (how steep the pile can get before it slides).
• Red Blood Cells: Simulating how non-spherical particles (like red blood cells) spin and move in flowing blood (Jeffery orbits).

The Bottom Line

LEDDS proves that you don't need to be a "code wizard" writing complex, machine-specific instructions to get super-fast results. By breaking the problem down into simple, standard building blocks (primitives), you can create a simulation that is:

1. Fast: It runs efficiently on powerful GPUs.
2. Portable: It works on different types of computers without rewriting the code.
3. Readable: It's much easier for humans to understand and fix.

It's a blueprint for the future of scientific computing: building complex physics simulations with simple, universal tools rather than custom, fragile machinery.

Technical Summary

1. Problem Statement

Simulating granular flows and fluid-particle interactions (CFD-DEM) is computationally intensive, particularly when high resolution is required to resolve individual particle contacts and fluid-solid interfaces. While Graphics Processing Units (GPUs) offer significant acceleration for these data-parallel tasks, traditional GPU programming relies on low-level, device-specific languages (e.g., CUDA, HIP) and hand-crafted kernels. This approach creates several challenges:

• Lack of Portability: Code written for one GPU architecture often cannot run on others without significant rewriting.
• Maintainability: Complex, low-level kernels are difficult to read, debug, and extend.
• Concurrency Risks: Manual management of thread synchronization increases the risk of race conditions.

The authors aim to address these issues by developing a fully coupled Lattice Boltzmann Method (LBM) and Discrete Element Method (DEM) solver that achieves high performance on GPUs while maintaining portability, readability, and generality through the use of high-level algorithmic primitives rather than device-specific code.

2. Methodology

The proposed framework, LEDDS (LBM-Enhanced Device-independent DEM Solver), is built on the philosophy that complex physics simulations can be decomposed into compositions of well-defined parallel primitives (e.g., map, reduce, sort, scan).

A. Algorithmic Paradigm

• Primitives: The entire workflow is expressed using C++ Standard Library (STL) parallel algorithms and the Thrust library. Key primitives include:
  • Map: Element-wise operations (e.g., force calculation, particle integration).
  • Sort/Unique: Used for collision detection (organizing particle pairs) and force reduction.
  • Reduce_by_key: Used to aggregate forces and torques from collision pairs to individual particles.
• Portability: By avoiding explicit kernel launches and relying on compiler-supported parallelism, the code runs efficiently on both multi-core CPUs and single GPUs with minimal changes (typically just compiler flags).
• Data Structure: The framework uses a Structure-of-Arrays (SoA) layout to ensure coalesced memory access, which is critical for GPU performance.

B. Physics Models

1. Discrete Element Method (DEM):
   • Models particles as spheres or ellipsoids.
   • Uses a spring-dashpot model for contact forces (normal and tangential) and friction.
   • Implements a Velocity-Verlet integration scheme for translational and rotational motion.
   • Collision Detection: Utilizes a uniform grid with a cell-linked list to avoid $O(N^2)$ neighbor searches. The list construction and collision pair identification are performed entirely via map, sort, and unique primitives.
2. Lattice Boltzmann Method (LBM):
   • Uses the D3Q19 lattice model with a Two-Relaxation-Time (TRT) collision operator for stability and accuracy.
   • Handles fluid-solid coupling via the Partially Saturated Method (PSM). Instead of sharp boundaries, cells intersected by particles are assigned a "solid fraction" ($\beta$). The collision operator is modified to blend fluid and solid behaviors, enforcing the no-slip condition smoothly.
   • Fluid-Particle Coupling: Hydrodynamic forces and torques are calculated based on the solid fraction and the deviation of the local fluid velocity from the particle surface velocity. These forces are then reduced to per-particle values using sort and reduce_by_key.

C. Implementation Details

• Language: C++ (leveraging C++17/20 standards).
• Hardware: Primarily targets NVIDIA GPUs (using Thrust for optimized sorting/reduction) but is designed to be hardware-agnostic.
• Precision: Supports both Single (FP32) and Double (FP64) precision.

3. Key Contributions

1. First Fully Coupled LBM-DEM Solver via Primitives: LEDDS is the first framework to express a complete, fully coupled LBM-DEM simulation (including complex ellipsoidal particles and fluid-solid coupling) exclusively using high-level algorithmic primitives.
2. Portability without Performance Loss: The framework demonstrates that high-level abstractions do not necessarily sacrifice performance. It achieves throughput comparable to hand-tuned CUDA solvers.
3. Ellipsoidal Particle Support: Unlike many primitive-based approaches limited to spheres, LEDDS handles ellipsoids, requiring complex contact detection and torque calculations, all implemented via portable primitives.
4. Open-Source Framework: The code is released under an MIT license, providing a blueprint for future portable multiphysics frameworks.

4. Results

The framework was validated through a hierarchy of benchmarks and performance analyses:

A. Accuracy and Validation

• DEM Benchmarks:
  • Elastic Collisions: Verified conservation of momentum and energy for head-on sphere collisions and eccentric ellipsoid collisions.
  • Friction: Accurately reproduced analytical solutions for sphere-wall collisions with friction (sliding vs. sticking regimes).
  • Many-Body Systems: Reproduced Maxwell-Boltzmann velocity distributions for gas-like particle systems and correctly predicted the angle of repose for sand piles of varying aspect ratios.
• LBM-DEM Benchmarks:
  • Single Particle Settling: Matched experimental data for terminal settling velocities across various Reynolds numbers.
  • Jeffery Orbits: Accurately captured the rotational dynamics of an ellipsoid in shear flow, matching analytical Jeffery solutions.
  • Effective Viscosity: Reproduced the Krieger-Dougherty relationship for the effective viscosity of particle suspensions.

B. Performance Analysis

• GPU Speedup: On an NVIDIA A100 GPU, LEDDS achieved speedups of 11x to 22x compared to a 128-core AMD EPYC CPU baseline, depending on the volume fraction and precision.
• Comparison with State-of-the-Art: When compared to waLBerla (a highly optimized, hybrid CUDA-CPU framework), LEDDS achieved performance within a factor of 2x of waLBerla using FP64. As the particle volume fraction increased (denser regimes), the performance gap narrowed, with LEDDS becoming comparable to waLBerla.
• Precision Impact: Single precision (FP32) offered significant speedups (up to 2x) in dilute regimes but provided diminishing returns in dense regimes where integer-based sorting and collision handling dominate the runtime.

5. Significance

This work establishes a new paradigm for High-Performance Computing (HPC) in computational physics:

• Blueprint for Portability: It proves that complex multiphysics solvers can be written in a portable, readable manner using standard C++ algorithms, reducing the barrier to entry for GPU programming and ensuring code longevity across different hardware generations (e.g., NVIDIA, AMD, Intel).
• Scalability: The algorithmic decomposition naturally supports future extensions to distributed-memory systems (MPI) and multi-GPU setups, as the data-dependent operations are already structured as bulk-synchronous primitives.
• Accessibility: By removing the need for low-level kernel programming, LEDDS allows researchers to focus on physics and algorithm design rather than hardware-specific optimization, while still achieving competitive performance.

In conclusion, LEDDS demonstrates that high-level algorithmic programming is sufficient to deliver high-performance, portable, and maintainable solutions for complex fluid-particle interactions, challenging the notion that hand-crafted kernels are strictly necessary for top-tier GPU performance.