Advancing Quantum Many-Body GW Calculations on Exascale Supercomputing Platforms

This paper presents innovative implementations of the BerkeleyGW package on Frontier and Aurora exascale supercomputers, achieving unprecedented performance portability and scaling for quantum many-body GW calculations on systems with up to 17,574 atoms, thereby enabling high-accuracy predictive simulations for future quantum technologies.

The Gist

Imagine you are trying to predict the weather, but instead of clouds and rain, you are trying to predict how tiny particles inside a computer chip or a new battery will behave. To do this accurately, you can't just look at the average temperature; you have to track the chaotic dance of billions of individual electrons bumping into each other.

This is what scientists call Quantum Many-Body Physics. It's incredibly hard to calculate because the math is so complex that even the world's fastest supercomputers usually crash or take years to finish a single simulation.

This paper is about a team of scientists who built a "super-calculator" (a software package called BerkeleyGW) that finally cracked the code. They managed to run these massive simulations on the world's newest, most powerful supercomputers (Frontier and Aurora), which are so fast they can perform a quintillion calculations per second (an "Exascale" computer).

Here is a breakdown of their breakthrough using simple analogies:

1. The Problem: The "Traffic Jam" of Electrons

Think of a standard computer simulation (like the ones used to design new materials) as a highway.

• Old Method (DFT): This is like driving a car where you only look at the road directly in front of you. It's fast, but you miss the traffic jams, accidents, and interactions happening three lanes over. It's good for simple things, but it fails when you need to know exactly how electrons interact with each other (like a massive traffic jam).
• The "GW" Method: This is the "God's eye view." It tracks every single car, every interaction, and every ripple in the traffic. It gives the perfect answer for how materials will behave, but the math is so heavy that it's like trying to drive a school bus through a narrow alley. It's too slow for big problems.

2. The Solution: Building a "Formula One" Fleet

The team didn't just make the bus faster; they rebuilt the entire vehicle to handle the weight of the math. They did this in three clever ways:

A. The "Universal Adapter" (Portability)

Imagine you have a car engine that works perfectly on American roads (NVIDIA chips), but you need to drive it on European roads (AMD chips) and Asian roads (Intel chips). Usually, you'd have to build three different cars.

• What they did: They built a "universal adapter" (using open programming languages like OpenACC and OpenMP). This allowed their software to run efficiently on any of the world's top supercomputers without needing to be rewritten for each one. It's like having a car that automatically adjusts its suspension and tires the moment it crosses a border.

B. The "Smart Subspace" Trick (Full-Frequency GW)

Calculating how electrons interact usually requires checking a million different "frequencies" (like tuning a radio to every single station to find the right one). This takes forever.

• What they did: They realized that most of the stations are static. They developed a trick called the "Static Subspace Approximation." Imagine instead of listening to 1,000 radio stations, you only listen to the top 20 most important ones, and you mathematically guess the rest. This made the calculation 25 to 100 times faster without losing accuracy.

C. The "Group Dance" (Mixed Stochastic-Deterministic)

When you have a huge system (like a crystal with 17,000 atoms), calculating every single electron interaction is like trying to choreograph a dance for 17,000 people individually.

• What they did: They used a "Stochastic" (random) method. Instead of tracking every single dancer, they grouped them into "pseudobands" (randomized clusters). They treated the group as a whole, which drastically reduced the number of calculations needed, turning a 4th-power math problem into something much more manageable.

3. The Results: Breaking the Speed Limit

The team tested their software on the Frontier (AMD-based) and Aurora (Intel-based) supercomputers.

• The Scale: They simulated a chunk of Lithium Hydride with 17,574 atoms. This is the largest simulation of its kind ever done.
• The Speed: They reached a speed of 1.069 ExaFLOP/s.
  • Analogy: If a regular laptop does 1 calculation per second, this supercomputer did 1,000,000,000,000,000,000 calculations per second.
  • They achieved nearly 60% of the theoretical maximum speed of these machines. In the world of supercomputing, getting 50%+ of the peak speed is like a race car driver hitting the absolute top speed limit of the track without crashing.

4. Why Does This Matter?

Why should a regular person care about simulating 17,000 atoms?

• Better Batteries: We can design batteries that charge faster and hold more energy.
• Quantum Computers: We can understand how to stop "quantum noise" (decoherence) so quantum computers don't make mistakes.
• New Materials: We can design materials that are stronger, lighter, or more efficient for solar panels and electronics before we ever build them in a lab.

The Bottom Line

This paper is a victory lap for the "Exascale Era." The team proved that we can now take the most difficult, "impossible" physics problems and solve them on the world's fastest computers. They didn't just make the math faster; they made it portable (works everywhere) and scalable (works for tiny systems and massive ones alike).

They turned a "school bus" simulation into a "Formula One" machine, opening the door to designing the quantum technologies of the future.

Technical Summary

1. Problem Statement

Quantum materials research is increasingly focused on complex, heterogeneous systems (e.g., solid-state qubits, moiré superlattices) and many-body phenomena (electron-electron, electron-phonon, electron-hole interactions). While Density Functional Theory (DFT) can handle systems with $\sim 10^4$ atoms, it fails to accurately predict excited-state properties like band gaps and electron-phonon coupling strengths, which are critical for designing next-generation quantum and energy devices.

The GW approximation is the state-of-the-art method for these properties but suffers from extreme computational complexity ($O(N^4)$ scaling) and high memory demands. Previous implementations were limited to smaller systems or specific hardware architectures. The challenge is to adapt these methods to exascale supercomputers (Frontier and Aurora) with heterogeneous GPU architectures (AMD and Intel) while maintaining high performance and portability.

2. Methodology and Innovations

The authors present significant algorithmic and HPC optimizations within the BerkeleyGW software package to overcome these bottlenecks.

A. Algorithmic Innovations

• GW Perturbation Theory (GWPT): A new method implemented for the first time to calculate correlated electron-phonon coupling at the many-body level. It extends the GW formalism to compute atom-displacement-perturbed self-energy matrix elements, offering higher accuracy than standard DFT-based DFPT for strongly correlated systems.
• Full-Frequency (FF) GW with Static Subspace Approximation: To avoid the prohibitive cost of full-frequency integration, the authors use a static subspace approximation. The zero-frequency polarizability is calculated in the full plane-wave basis, and a reduced subspace (10–20% of basis vectors) is used to reconstruct non-zero frequencies. This reduces the computational cost from $O(N_\omega N_v N_c N_G^2)$ to $O(N_v N_c N_G^2 + N_\omega N_v N_c N_{Eig}^2)$, yielding a 25–100x speedup.
• Mixed Stochastic-Deterministic Pseudobands: To handle the sum-over-bands bottleneck, the authors employ a stochastic compression of the Lehmann representation. High-energy Kohn-Sham states are replaced by stochastic linear combinations (pseudobands), while low-energy states near the Fermi level remain deterministic. This reduces the effective number of bands ($N_b$) and lowers the scaling of the calculation.

B. HPC and Portability Strategies

• True Performance Portability: The codebase utilizes OpenACC and OpenMP-target directives to support a unified codebase across NVIDIA, AMD, and Intel GPUs.
• Hardware-Optimized Kernels: For peak performance, the core kernels are rewritten in vendor-specific languages: CUDA (NVIDIA), HIP (AMD), and SYCL (Intel).
• Kernel Optimizations:
  • Diagonal Self-Energy (GPP diag.): Optimized for high arithmetic intensity using explicit memory management (coalesced access), loop unrolling to maximize register occupancy, and two-stage reductions.
  • Off-Diagonal Self-Energy (GPP off-diag.): Reformulated as dense matrix-matrix multiplications (ZGEMM). By precomputing frequency-dependent matrices and casting the problem as $N_\Sigma \times N_G \times N_G$ and $N_\Sigma \times N_G \times N_\Sigma$ multiplications, the arithmetic intensity is significantly increased, leveraging GPU matrix cores.

3. Key Results

The work was benchmarked on the Frontier (AMD MI250X) and Aurora (Intel Ponte Vecchio) exascale systems, as well as Perlmutter (NVIDIA A100).

• Scale of Simulation:
  • Successfully simulated a Lithium Hydride (LiH) defect system with 17,574 atoms (surpassing previous records).
  • Simulated a Silicon divacancy with 2,742 atoms.
  • Simulated a Boron Nitride (BN) moiré bilayer with 867 atoms.
• Performance Metrics:
  • Peak Performance: Achieved 1.069 ExaFLOP/s (double precision) on Frontier (9,408 nodes) and 707.52 PetaFLOP/s on Aurora (9,600 nodes) for the off-diagonal GPP kernel.
  • Efficiency: These results correspond to 59.45% of the theoretical peak on Frontier and 48.79% of the attainable peak on Aurora.
  • Scaling: Demonstrated excellent strong scaling up to the full machine size and weak scaling across thousands of nodes.
• Portability: The OpenACC implementation recovered ~90% of the best CUDA performance on NVIDIA GPUs and 60–70% of the best HIP performance on AMD GPUs, proving the viability of directive-based models for portability, though hardware-specific kernels (HIP/SYCL) remain necessary for peak efficiency.

4. Significance

This work represents a breakthrough in quantum materials simulation on exascale platforms:

1. Predictive Capability: It enables the rational design of future quantum technologies (qubits, quantum emitters, optoelectronic devices) by providing accurate predictions of excited-state properties and electron-phonon coupling in systems with $10^3$–$10^4$ atoms.
2. Methodological Advancement: The introduction of GWPT and the mixed stochastic-deterministic approach opens new avenues for studying correlated electron-phonon interactions and reducing computational scaling.
3. HPC Leadership: It demonstrates that complex, memory-bound quantum many-body problems can be solved with high efficiency on heterogeneous exascale architectures, achieving near-peak performance across different GPU vendors. This sets a new standard for "performance portability" in scientific computing.

In summary, the paper establishes BerkeleyGW as a premier tool for first-principles many-body perturbation theory, capable of tackling the most complex materials problems on the world's fastest supercomputers.