High-Performance Resilient Multi-GPU Hybrid Particle-in-Cell Monte Carlo Simulations at Scale

Technical Summary: High-Performance Resilient Multi-GPU Hybrid PIC Monte Carlo Simulations at Scale

Problem Statement
Plasma physics applications, particularly Particle-in-Cell (PIC) Monte Carlo (MC) simulations, face significant challenges when scaling to modern heterogeneous high-performance computing (HPC) systems. These simulations are characterized by dynamic particle distributions, irregular memory access patterns, and the necessity for efficient data movement. As systems transition toward exascale architectures utilizing mixed CPU/GPU environments (Nvidia and AMD), traditional implementations struggle with scalability, load imbalance, and resilience. Specifically, the need for robust checkpoint/restart (C/R) mechanisms and efficient I/O without interrupting long-running simulations has become critical, as system failure rates increase and computational workloads grow.

Methodology
This work extends the Berkeley Innsbruck Tbilisi 1D3V (BIT1) code, a 1D3V electrostatic PIC MC solver, into a portable, resilient hybrid MPI+OpenMP framework. The methodology focuses on three primary technical pillars:

1. Hybrid Architecture and Memory Optimization: The code was refactored to support both Nvidia and AMD accelerators. Profiling using Nvidia Nsight Systems and AMD ROC-Profiler revealed that Unified Memory (UM) introduced latency due to page faults and implicit migrations. To address this, the implementation transitioned to Pinned (Page-Locked) Host Memory (PinM) for critical data structures. This was achieved using OpenMP memory allocators with pinned traits, allowing for selective allocation in particle mover and arranger functions. This approach enables direct device-pointer access via use_device_ptr and is_device_ptr clauses, facilitating asynchronous execution and reducing data-transfer overhead.
2. Scalable Particle Load Balancing (PLB): To mitigate load imbalances caused by non-uniform particle distributions, a PLB mechanism was integrated. In the original implementation, MPI ranks processed fixed cell intervals, leading to uneven workloads. The new approach utilizes the load_balance flag to redistribute particles across ranks. MPI rank 0 determines the optimal particle distribution and cell intervals, which are then communicated to all ranks to update local parameters before memory allocation. This ensures a balanced MPI workload distribution.
3. Standardized Resilient I/O: The framework integrates the openPMD standard and ADIOS2 for I/O operations. It leverages the BP4 engine for high-throughput file-based checkpointing and the SST engine for in-memory data streaming. This layer supports both uniform and non-uniform load conditions, enabling seamless C/R, in-situ analysis, and visualization without halting the simulation.

Key Contributions
The paper presents four specific contributions:

• Portable Hybrid Extension: A resilient hybrid MPI+OpenMP extension of BIT1 designed for modern HPC systems, supporting both strong and weak scaling under uniform and non-uniform load conditions.
• Resilient Execution Mechanisms: The introduction of scalable particle load balancing and checkpoint/restart mechanisms that allow for seamless continuation from checkpoints and efficient management of computational and I/O workloads across Nvidia and AMD GPU architectures.
• Standardized I/O Integration: The integration of openPMD and ADIOS2 (BP4 and SST backends) to enable high-throughput checkpointing and in-memory streaming, ensuring consistent state management and efficient data movement in large-scale multi-GPU executions.
• In-Situ Analysis Tooling: A customized Python script utilizing the openPMD API and ADIOS2 backends to perform real-time checkpoint analysis and visualization of hybrid BIT1 outputs (both file-based and in-memory) without interrupting the simulation.

Performance Results
The framework was evaluated on four major HPC systems: Dardel (Sweden), MareNostrum5 (MN5, Spain), LUMI (Finland), and Frontier (USA).

• Optimization Gains: On MN5, transitioning from a 3D data layout with Unified Memory to a 1D layout with Pinned Memory and OpenMP parallelism resulted in a 29.19× speedup over the original BIT1 version.
• Strong Scaling: Under non-uniform load conditions on 40 nodes (320 GPUs), the optimized hybrid BIT1 achieved a 14.23× speedup over the original CPU-based version. The combination of the SST backend and PLB provided the fastest execution times.
• Large-Scale Scaling: On Frontier, scaling up to 100 nodes (800 GPUs) demonstrated strong scaling improvements. The combination of openPMD (SST backend) and PLB achieved a 9.13× speedup over the serial I/O baseline.
• Weak Scaling: For weak scaling up to 800 GPUs, the original hybrid GPU version maintained 70.8% parallel efficiency. The openPMD-enabled versions with SST and PLB achieved a peak parallel efficiency of 88.0%, demonstrating superior resilience to I/O and C/R overheads.
• In-Situ Visualization: The system successfully performed real-time monitoring of plasma profiles and particle loads during 10,000 timesteps on 320 GPUs, validating the ability to analyze data without simulation interruption.

Significance
The authors claim that this work provides a validated framework for large-scale PIC MC simulations on current and future exascale systems. By combining portability, resilience, and scalable I/O, the hybrid BIT1 implementation addresses critical bottlenecks in heterogeneous computing, specifically load imbalance and I/O overhead. The integration of openPMD and ADIOS2 allows for efficient data movement and continuous execution, while the PLB mechanism ensures resource utilization remains high even under dynamic plasma conditions. The results confirm that the framework is capable of handling the computational and I/O demands of next-generation plasma edge and tokamak divertor physics simulations.