Multi-GPU fast Fourier transforms in MATLAB (for large-scale phase-field crystal simulations)

This paper presents a MATLAB-based multi-GPU framework that overcomes memory limitations and achieves significant speedups (up to 60x) for large-scale phase-field crystal simulations by implementing two complementary strategies for accelerating two- and three-dimensional fast Fourier transforms.

The Gist

Imagine you are trying to simulate the growth of a crystal, like watching sugar crystals form in a jar of syrup, but on a microscopic level where you can see every single atom. To do this accurately, you need a supercomputer to crunch billions of numbers every second.

This paper introduces a new "super-tool" built in MATLAB (a popular software for scientists) that makes these simulations run much faster by using multiple graphics cards (GPUs) at the same time.

Here is the breakdown using simple analogies:

1. The Problem: The "Single-Worker" Bottleneck

Think of a GPU (the powerful chip in your computer that usually handles video games) as a super-fast chef.

• The Task: The chef needs to chop a mountain of vegetables (data) to make a soup (the simulation).
• The Limit: Even though the chef is incredibly fast, they only have one cutting board (memory). If the mountain of vegetables is too big, it spills off the board, and the chef has to stop.
• The Old Way: Previously, scientists had to use either:
  • One Chef: Who had to work on a smaller mountain of vegetables (less detail).
  • Hundreds of Slow Cookers (CPUs): Who could handle the big mountain but were so slow at chopping that the soup took days to make.

2. The Solution: Two New Strategies

The authors built a system that lets multiple chefs work together in a kitchen. They came up with two different ways to organize the team, depending on the job:

Strategy A: The "Assembly Line" (For One Giant Problem)

Imagine you have one giant, massive pizza (a single huge simulation) that is too big for one chef to handle.

• How it works: You slice the pizza into long strips (slabs).
• The Team: Chef 1 slices the first strip, Chef 2 slices the second, and so on.
• The Handoff: Once they are done, they pass the strips to each other in a specific order (like a relay race) to finish the toppings.
• The Result: They can cook a pizza that is 6 times bigger than what a single chef could handle, and they finish 6 times faster than a team of slow cookers.

Strategy B: The "Specialized Stations" (For Complex, Multi-Part Problems)

Now imagine the simulation isn't just one thing; it's a complex recipe involving four different ingredients at once (like density, temperature, and velocity).

• The Old Way: One chef tries to juggle all four ingredients, getting confused and dropping things.
• The New Way: You assign one chef to each ingredient.
  • Chef 1 only handles the dough.
  • Chef 2 only handles the sauce.
  • Chef 3 only handles the cheese.
  • Chef 4 only handles the spices.
• The Handoff: After they prep their specific ingredient, they shout out to the others to swap data so everyone stays in sync.
• The Result: Because they aren't juggling, they can handle a massive, complex recipe that was previously impossible. This version is 60 times faster than the old slow-cooker method!

3. Why Does This Matter?

The paper focuses on Phase-Field Crystal (PFC) models.

• What is it? It's a way to simulate how materials behave, like how metals bend, how cracks form, or how crystals grow.
• Why is it hard? These models require looking at the world at the atomic level (tiny details) but over a long period of time (large areas). It's like trying to watch a movie of a whole city, but you need to see every single brick in every building.
• The Breakthrough: Before this, scientists were limited by how much memory their computer had. They couldn't simulate big enough areas to see real-world phenomena.
• The Impact: With this new tool, scientists can now simulate huge, complex materials in a fraction of the time. It's like upgrading from a bicycle to a supersonic jet for materials science.

4. The "Secret Sauce" (MATLAB)

Usually, writing code to make multiple graphics cards talk to each other is like trying to teach a group of people to speak a secret language while they are running a marathon. It's very difficult and usually requires complex, low-level coding (C++ or Fortran).

The authors did something special: They built this in MATLAB.

• Why is this cool? MATLAB is like a "user-friendly" language that many scientists already know. By making this tool in MATLAB, they made it accessible. Now, a researcher doesn't need to be a coding wizard to use super-fast, multi-GPU power; they can just plug it into their existing workflow.

Summary

The authors created a teamwork system for computer chips.

1. They let multiple chips share the load of one giant task.
2. They let multiple chips specialize in different parts of a complex task.
3. They made it easy to use for scientists.

The Bottom Line: They turned a slow, limited process into a fast, massive one, allowing scientists to see the "invisible" world of atoms and crystals in ways that were previously impossible.

Technical Summary

1. Problem Statement

Large-scale numerical simulations using the pseudo-spectral Fourier method, particularly for Phase-Field Crystal (PFC) models, face significant bottlenecks due to:

• Memory Limitations: PFC models require fine spatial resolution over large periodic domains to resolve atomic-scale crystalline order while evolving on diffusive time scales. This leads to massive data sizes that often exceed the memory capacity of a single GPU.
• Computational Bottlenecks: The governing equations (often 10th-order partial differential equations) rely heavily on repeated multidimensional Fast Fourier Transforms (FFTs) at every time step. In single-GPU or CPU-only implementations, these FFTs become the primary performance limiter.
• Software Gaps: While multi-GPU FFTs exist in high-performance computing (HPC) libraries (e.g., FFTW, cuFFT), there was a lack of a dedicated, accessible multi-GPU FFT implementation within MATLAB tailored for large-scale 3D PFC simulations. Existing PFC solvers were limited to single-GPU or CPU-based parallelization.

2. Methodology

The authors developed a MATLAB-based framework implementing two complementary multi-GPU strategies to overcome these limitations. The core mathematical approach relies on the Fourier pseudo-spectral method, where spatial derivatives are computed efficiently in Fourier space, and nonlinear terms are evaluated in physical space.

Strategy A: Distributed Single FFT (Domain Decomposition)

• Concept: This strategy addresses problems where the total data size exceeds a single GPU's memory.
• Implementation:
  • A 3D FFT of size $N_x \times N_y \times N_z$ is decomposed across $G$ GPUs using slab decomposition along the $z$-direction.
  • Workflow:
    1. Each GPU performs local 2D FFTs ($x, y$).
    2. Peer-to-Peer (P2P) communication redistributes data slices between GPUs.
    3. Each GPU performs the final 1D FFT along the remaining dimension ($z$).
  • Result: The transformed Fourier-space array is distributed across the GPUs, allowing the simulation of grid sizes impossible on a single device.

Strategy B: Concurrent Multi-Field FFTs (Field Decomposition)

• Concept: This strategy targets multiphysics extensions (e.g., Hydrodynamic PFC) where multiple coupled fields (density, velocity components, temperature) must be evolved simultaneously.
• Implementation:
  • Different physical fields are assigned to separate GPUs (e.g., density field $\psi$ on GPU 1, velocity components $v_1, v_2, v_3$ on GPUs 2–4).
  • Workflow:
    1. Each GPU computes the FFTs required for its specific field.
    2. Synchronized communication occurs after each time step to exchange necessary data (e.g., sending the density field to velocity solvers and vice versa).
  • Benefit: This approach maximizes throughput by utilizing all GPUs concurrently for different parts of the system, rather than just splitting one large array.

3. Key Contributions

• First MATLAB Multi-GPU FFT: The paper introduces the first implementation of multi-GPU FFTs specifically within the MATLAB environment, making high-performance spectral simulations accessible to a broader scientific community without requiring low-level CUDA/C++ coding.
• Dual-Strategy Framework: The software provides both domain decomposition (for memory scaling) and field decomposition (for multiphysics concurrency), offering flexibility for different simulation types.
• Scalable PFC Solver: The framework successfully solves the 10th-order PFC equations and hydrodynamic PFC extensions in 2D and 3D, handling grid sizes up to $1400^3$.
• Open Source Availability: The code is released under an MIT License on GitHub and Zenodo, ensuring reproducibility and extensibility.

4. Results and Performance

The framework was benchmarked on HPC clusters at TU Dresden using NVIDIA H100 and A100 GPUs, compared against a purely CPU-based implementation (Intel Xeon Platinum, 100 cores).

• Standard PFC Simulations (Strategy A):
  • Achieved a ~6x speedup compared to the 100-core CPU implementation.
  • Successfully handled grid sizes from $750^3$ to $1400^3$, which are infeasible on single GPUs due to memory constraints.
• Multiphysics PFC Simulations (Strategy B):
  • Demonstrated speedups of up to 60x compared to CPU execution for large computational domains.
  • Enabled simulations of hydrodynamic PFC models with coupled density and velocity fields that would otherwise be impossible to run on a single GPU.
• Visual Validation: The framework successfully reproduced complex physical phenomena, including dendritic solidification (2D) and polycrystalline coarsening (3D).

5. Significance

• Enabling Larger Simulations: By overcoming single-GPU memory limits, this work allows researchers to simulate larger physical domains and finer resolutions, crucial for capturing realistic microstructure evolution, defects, and grain boundaries.
• Bridging the Gap: It democratizes access to high-performance computing for spectral methods. MATLAB is widely used in academia for prototyping; this tool allows researchers to scale these prototypes to production-level HPC resources without rewriting code in C++ or Fortran.
• Future-Proofing: The field-decomposition strategy is particularly relevant for future complex models involving tens of coupled variables (e.g., coarse-grained amplitude models), suggesting a scalable path for next-generation materials science simulations.

In summary, this paper presents a robust, accessible, and highly efficient MATLAB framework that leverages multi-GPU architectures to solve previously intractable large-scale phase-field crystal problems, significantly accelerating the discovery of material properties and microstructural evolution.