Scalability of the asynchronous discontinuous Galerkin method for compressible flow simulations

This paper presents the implementation and evaluation of an asynchronous discontinuous Galerkin method with asynchrony-tolerant fluxes in the deal.II library, demonstrating that this approach recovers high-order accuracy for compressible flow simulations while achieving significant speedups (up to 1.9x) by reducing synchronization overheads in large-scale parallel computing.

The Gist

Imagine you are trying to solve a massive, complex puzzle (simulating how air flows around a plane or a car) using a team of thousands of workers (computers) working in a giant warehouse.

The Problem: The "Stop-and-Go" Traffic Jam

In traditional supercomputing, the workers are organized into small groups. To solve the puzzle, every worker needs to know what their neighbors are doing.

• The Old Way (Synchronous): Every few seconds, the whole team has to stop. Everyone shouts their current progress to their neighbors, waits for everyone else to finish shouting, and then everyone takes the next step together.
• The Bottleneck: As you add more workers to make the team bigger, the time spent shouting and waiting (communication) starts to take up more time than the actual work of solving the puzzle. It's like a traffic jam where everyone is waiting for the light to change, but the light changes so slowly that no one is actually moving forward. This limits how big the team can get before it becomes useless.

The Solution: The "Asynchronous" Team

The authors of this paper proposed a new way to work called the Asynchronous Discontinuous Galerkin (ADG) method.

The Analogy: Instead of stopping the whole team to shout, imagine a "Communication-Avoiding" strategy.

• The New Way: Workers keep moving and solving their part of the puzzle. They only stop to shout to their neighbors every few minutes, not every few seconds.
• The Trick: While they are waiting for the latest shout from a neighbor, they don't just sit idle. They use the last thing they heard (delayed data) to keep working.
• The Risk: If you use old data, you might make a mistake. In math terms, this usually ruins the accuracy of the solution, turning a high-definition movie into a blurry, low-quality video.

The Magic Ingredient: "Asynchrony-Tolerant" Fluxes

This is where the paper's real breakthrough comes in. The authors realized that just using old data makes the math sloppy. So, they invented a special mathematical tool called Asynchrony-Tolerant (AT) Fluxes.

The Analogy: Think of it like a smart chef.

• The Problem: A chef needs fresh ingredients (latest data) to make a perfect dish. If they have to wait for the delivery truck, the food gets cold.
• The Old Fix: Just use the cold food anyway. The taste is bad (low accuracy).
• The AT Flux Fix: The chef has a secret recipe. Even if the fresh tomatoes haven't arrived yet, the chef looks at the tomatoes they used 5 minutes ago, 10 minutes ago, and 15 minutes ago. By mixing these "old" ingredients in a very specific, clever way, the chef can predict what the fresh tomatoes would have tasted like and recreate the perfect dish.
• The Result: The team keeps moving fast (no traffic jams), but the final puzzle solution remains perfectly sharp and accurate, just as if they had waited for the fresh data.

What They Found

The researchers tested this on a massive supercomputer in India with thousands of processors.

1. Accuracy: They proved that if you just use old data without the special "AT Flux" recipe, the math falls apart and becomes very inaccurate. But with the AT Flux, the math stays perfect, even with delays.
2. Speed: Because the workers spend less time shouting and waiting, the whole team gets the job done much faster.
   • In 2D simulations (like a flat map), they were 1.9 times faster.
   • In 3D simulations (like a real-world object), they were 1.6 times faster.

Why This Matters

As computers get bigger and bigger (heading toward "Exascale" systems with billions of cores), the time spent waiting for data is becoming the biggest problem. This paper shows that by letting workers keep moving and using "smart guesses" based on old data, we can build super-fast, highly accurate simulations for things like weather forecasting, airplane design, and climate modeling without getting stuck in traffic jams.

In short: They taught the computer team how to keep running at full speed without stopping to check in, while using a clever mathematical trick to ensure they don't make any mistakes.

Technical Summary

1. Problem Statement

The Discontinuous Galerkin (DG) method is a powerful framework for solving hyperbolic partial differential equations (PDEs), such as the compressible Euler equations, due to its high-order accuracy and locality. However, on modern massively parallel systems (exascale), the scalability of DG solvers is severely constrained by communication and synchronization overheads.

• The Bottleneck: In standard synchronous implementations, every stage of the time-integration scheme (e.g., Runge-Kutta) requires global or neighborhood-level synchronization to exchange "ghost values" (boundary data) between processing elements (PEs).
• The Consequence: As the number of processes increases, the surface-to-volume ratio of subdomains grows. The time spent waiting for communication and synchronization eventually dominates the runtime, causing the solver to deviate from ideal strong scaling.
• The Gap: While "asynchronous" approaches that relax communication requirements have been proposed for finite-difference methods, their implementation, accuracy, and scalability in modern, matrix-free DG solvers for compressible flows had not been systematically investigated.

2. Methodology

The authors implemented an Asynchronous Discontinuous Galerkin (ADG) method within the open-source finite element library deal.II, specifically extending the matrix-free DG solver found in step-76. The methodology consists of three core components:

A. Communication-Avoiding Algorithm (CAA)

Instead of synchronizing at every Runge-Kutta (RK) stage, the CAA allows the solver to proceed for a specific number of time steps ($L$) without exchanging ghost values.

• Mechanism: The solver uses previously communicated data (delayed values) to compute numerical fluxes at PE boundaries.
• Standard Flux Limitation: Using standard numerical fluxes with delayed data degrades the solution accuracy to first-order, regardless of the polynomial degree ($N_p$) of the basis functions.

B. Asynchrony-Tolerant (AT) Fluxes

To recover high-order accuracy while maintaining asynchrony, the authors implemented AT fluxes.

• Concept: AT fluxes reconstruct high-order accurate fluxes at the current time level by taking a linear combination of numerical fluxes from multiple previous time levels ($N_p + 1$ levels).
• Implementation: The method employs a rolling buffer to store historical flux values. When communication is skipped, the solver computes the flux using coefficients derived from a Taylor series expansion to cancel out the error introduced by the time delay.
• Coupling: The authors used a "naive" coupling approach for multi-stage RK schemes, updating the delay parameter at each RK stage to account for fractional time-step advancements.

C. Solver Architecture

• Platform: Implemented in C++ using deal.II on the PARAM Pravega supercomputer (Intel Xeon Cascade Lake).
• Discretization: Matrix-free DG formulation with Low-Storage Explicit Runge-Kutta (LSERK) time integration.
• Execution Model: The solver partitions elements into interior (part-0, part-2) and boundary (part-1) sets. In the asynchronous mode, interior elements are computed independently, while boundary elements utilize either the latest ghost data (during sync phases) or AT fluxes (during async phases).

3. Key Contributions

1. First Large-Scale Implementation: The first implementation of the ADG method with AT fluxes in a modern, matrix-free DG solver (deal.II) for compressible flows.
2. Accuracy Recovery: Demonstrated that while standard asynchronous fluxes degrade accuracy to first-order, AT fluxes successfully recover the formal high-order accuracy ($O(h^{N_p+1})$) of the DG discretization.
3. Scalability Analysis: Conducted extensive strong-scaling studies for 2D and 3D compressible Euler equations, quantifying the performance benefits of CAA over synchronous baselines.
4. Algorithmic Integration: Successfully integrated the CAA with AT fluxes into an existing high-performance solver infrastructure, managing the complex data dependencies of multi-stage RK schemes without sacrificing the matrix-free efficiency.

4. Results

Accuracy Verification

• 2D Isentropic Vortex: The ADG method with standard fluxes showed first-order convergence, confirming theoretical predictions. In contrast, the ADG method with AT fluxes achieved second-order ($N_p=1$) and third-order ($N_p=2$) convergence, matching the accuracy of the synchronous solver.
• 3D Flow Around a Cylinder: Visual and quantitative analysis of pressure fields and density gradients showed that the standard ADG method introduced distortions in shock structures. The AT flux variant eliminated these distortions, producing results nearly identical to the synchronous solver.

Performance and Scalability

• Strong Scaling: The study tested configurations ranging from $10^5$ to $2 \times 10^8$ degrees of freedom across up to 20,400 processes.
• Speedup: The CAA-based ADG solver with AT fluxes significantly outperformed the synchronous baseline:
  • 2D: Up to 1.9× speedup at extreme scales (16,416 processes).
  • 3D: Up to 1.6× speedup at extreme scales (20,400 processes).
• Efficiency: Parallel efficiency for the largest 3D case dropped to 0.13 for the synchronous solver but remained at 0.53 for the asynchronous solver.
• Bottleneck Mitigation: Profiling revealed that the synchronous solver's runtime became dominated by vector_update_ghosts_finish (synchronization) at high process counts. The CAA-AT approach reduced synchronization overhead, keeping computation as the dominant cost for a wider range of process counts.

5. Significance and Conclusion

This work validates that asynchronous computing is a viable and necessary strategy for next-generation exascale DG solvers.

• Overcoming the Communication Wall: By mathematically relaxing synchronization requirements and using AT fluxes to correct the resulting errors, the method effectively mitigates the communication bottleneck that limits current high-order solvers.
• Practical Applicability: The approach maintains the strict locality and conservation properties of DG methods while enabling substantial speedups on modern CPU clusters.
• Future Outlook: The authors note that while stability constraints (CFL limits) are tighter for asynchronous schemes, this is less critical for stiff problems (e.g., reacting flows) where time steps are already limited by chemistry. Future work will focus on higher-order coupling strategies and extending the method to viscous and multi-physics systems.

In summary, the paper demonstrates that the Asynchronous DG method with Asynchrony-Tolerant fluxes offers a promising pathway to achieving efficient, accurate, and scalable simulations of compressible flows on emerging exascale architectures.