Co-Design and Evaluation of a CPU-Free MPI GPU Communication Abstraction and Implementation

This paper presents the design, implementation, and evaluation of a CPU-free MPI GPU communication abstraction leveraging HPE Slingshot 11 capabilities, which significantly reduces latency and improves strong scaling performance for GPU-based HPC applications by eliminating CPU involvement in the communication fast path.

The Gist

Imagine you are running a massive, high-speed relay race with thousands of runners (GPUs) spread across a giant stadium. Their job is to pass notes (data) to each other as fast as possible to solve a complex puzzle.

In the old way of doing things, every time a runner wanted to pass a note, they had to stop, run to the "Coach's Booth" (the CPU), get permission, fill out a form, and wait for the Coach to signal the other runner to be ready. Only then could the note be passed. This "Coach" was incredibly busy, and the time spent running back and forth to the booth slowed the whole race down, especially for short, quick notes.

This paper is about building a new system where the runners can pass notes directly to each other without ever bothering the Coach.

Here is a breakdown of how they did it, using simple analogies:

1. The Problem: The "Coach" Bottleneck

In traditional supercomputing, even though the GPUs are super-fast, they are often held back by the CPU.

• The Old Way: To send data, the GPU had to ask the CPU, "Hey, can I send this?" The CPU would check if the receiver was ready, set up the connection, and then say, "Go!"
• The Cost: This "checking" process takes time (microseconds). In a race where you are passing millions of tiny notes, those tiny delays add up to huge slowdowns. It's like a Formula 1 car having to stop at a toll booth for every single mile.

2. The Solution: The "Smart Mailbox" System

The researchers designed a new communication system (an API) that lets the GPUs talk directly to the network cards (the "Mailboxes") without waking up the Coach.

They used a clever trick involving Persistent Requests and Triggers:

• The Setup (The Coach's Job, done once): Before the race starts, the Coach helps the runners set up their "Smart Mailboxes." They agree on who is talking to whom and what the notes will look like. This is called "matching."
• The Race (CPU-Free): Once the race begins, the runners don't ask the Coach for permission. Instead, they have a special trigger. When a runner finishes their part of the puzzle, they simply flip a switch (a counter) on their mailbox.
• The Magic: The mailbox sees the switch flip and automatically shoots the note to the other runner. The other runner's mailbox sees the note arrive and flips their own switch to say, "Got it!"

3. The "Ready" vs. "Not Ready" Trick

There was one tricky part: What if Runner A tries to send a note before Runner B is ready to catch it?

• The Old Problem: In the old system, the CPU would have to step in and say, "Wait, Runner B isn't ready yet!"
• The New Solution: The researchers created a "Ready Signal" system.
  • If Runner B is ready, they flip a "Ready" switch. Runner A sees it and sends the note instantly.
  • If Runner B isn't ready, Runner A's note is held in a special "waiting room" (a buffer) inside the network card. The network card itself watches for the "Ready" switch. As soon as Runner B flips it, the network card automatically delivers the note.
  • Result: The CPU never has to step in to manage the waiting. The network hardware handles the traffic control.

4. The Results: A Faster Race

The team tested this new system on two of the world's fastest supercomputers (Frontier and Tuolumne).

• The Ping-Pong Test: When two GPUs just sent notes back and forth, the new system was 50% faster for medium-sized notes. It was like removing the toll booth entirely.
• The Big Race (Halo Exchange): In a complex simulation where thousands of GPUs had to swap data constantly (like a giant game of "Life"), the new system allowed the supercomputer to scale up to 8,192 GPUs and still run 28% faster than the standard method.

Why This Matters

Think of this like upgrading a city's traffic system.

• Before: Every car had to stop at a traffic light controlled by a human officer at every intersection.
• After: The cars have smart sensors that talk to each other and the road itself. They flow through intersections automatically when it's safe, without waiting for a human.

This research proves that we can make supercomputers much more efficient by letting the "runners" (GPUs) handle their own communication, freeing up the "Coach" (CPU) to focus on the actual math and logic. This is a huge step forward for Artificial Intelligence and scientific simulations that need to process massive amounts of data quickly.

Technical Summary

Here is a detailed technical summary of the paper "Co-Design and Evaluation of a CPU-Free MPI GPU Communication Abstraction and Implementation."

1. Problem Statement

Modern High-Performance Computing (HPC) and Machine Learning (ML) applications increasingly rely on GPU acceleration. However, traditional GPU communication methods suffer from significant latency overheads because they require the CPU to manage the communication "fast path."

Key Overheads Identified:

• Kernel Launch Latency: Launching a kernel from the CPU takes ~1 microsecond.
• Memory Barriers: Synchronizing CPU and GPU memory (stream synchronization) takes ~1 microsecond.
• Messaging Setup: Traditional two-sided MPI communication (e.g., Ready-to-Send/Clear-to-Send protocols) and message matching add several microseconds of overhead.
• Impact: For small-to-medium messages (e.g., 16KB halo exchange edges), these CPU-mediated steps can increase end-to-end latency by 20–80%, severely limiting the strong scaling performance of HPC applications.

Existing solutions either rely on the CPU for communication (e.g., standard GPU-aware MPI, NCCL) or sacrifice expressiveness (e.g., one-sided APIs like NVSHMEM) to achieve CPU-free communication, often removing essential features like message matching.

2. Methodology

The authors propose a co-designed API and implementation that enables CPU-free, two-sided MPI communication on GPUs. The approach leverages specific hardware capabilities of the HPE Slingshot 11 network interface card (NIC) and the libfabric API.

Core Design Principles:

• Persistent Operations: The API builds on existing MPI persistent point-to-point and collective operations. This allows the separation of the expensive "setup/matching" phase from the "execution" phase.
• Stream-Triggered Execution: Communication operations are enqueued to a GPU stream. The GPU itself triggers the data movement without CPU intervention once the stream reaches the operation.
• Hardware Offloading: The implementation utilizes libfabric Deferred Work Queues (DWQs) and Triggering Counters.
  • Counters: GPU kernels write to memory-mapped I/O regions to increment counters on the NIC.
  • DWQs: The NIC executes communication operations (like fi_write or fi_atomic) only when a specific counter value is reached.
• Two-Sided Semantics: Unlike one-sided approaches, this design preserves standard MPI two-sided semantics (message matching) by moving the matching logic to the setup phase (via MPI_Match) rather than the fast path.

Key API Extensions:

1. MPI_Queue: A new object that binds a set of communication tasks to a specific GPU stream, ensuring ordered execution.
2. MPI_Match / MPI_Matchall: Functions to explicitly match persistent requests. This ensures that when a request is enqueued for execution, the remote peer is already determined, eliminating the need for complex negotiation during the fast path.
3. MPI_Enqueue_Start / Wait: Functions to add start and wait operations to the MPI_Queue, which are then executed directly by the GPU stream.

Implementation Strategy (HPE Slingshot 11):

• Send Side: The GPU triggers a deferred fi_write (data movement) and a chained fi_atomic (completion notification) via a counter.
• Receive Side: The receiver uses a counter triggered by a remote write (FI_REMOTE_WRITE) to update a local completion buffer.
• Readiness Handling: To handle the requirement that a receiver buffer must be ready before a send occurs (without CPU intervention), the system uses a "Clear-to-Send" (CTS) mechanism. The receiver triggers an atomic write to a CTS buffer when ready; the sender's deferred work queue is configured to only execute when the CTS counter is incremented.

3. Key Contributions

1. New MPI API Design: A novel set of extensions (MPI_Queue, MPI_Match, MPI_Enqueue) that enable CPU-free, two-sided communication while preserving familiar MPI abstractions.
2. Prototype Implementation: A full implementation for HPE Slingshot 11 systems using libfabric's CXI provider, demonstrating full CPU-free point-to-point communication.
3. Integration with Cabana/Kokkos: The API was integrated into the Cabana performance portability framework to create StreamHalo objects, enabling CPU-free halo exchanges for production HPC codes.
4. Comprehensive Evaluation: Extensive benchmarking on two supercomputers (ORNL Frontier and LLNL Tuolumne) comparing the new API against standard Cray MPICH.

4. Results

The evaluation was conducted on ORNL Frontier (9,408 nodes, AMD MI250X GPUs) and LLNL Tuolumne (1,152 nodes, AMD MI300A APUs).

A. Microbenchmarks (GPU Ping-Pong):

• Latency Reduction: The CPU-free approach reduced latency by 12–51% for message sizes between 32 bytes and 2MB compared to Cray MPICH.
• Peak Improvement: Up to 50% reduction in latency for medium-sized messages (e.g., 4KB–1MB).
• Bandwidth: Improved bandwidth for messages up to 1MB. For very large messages (>8MB), standard MPICH occasionally outperformed due to optimized hardware protocols for large transfers, but the CPU-free approach remained competitive.

B. Strong Scaling (CabanaGhost Halo Exchange):

• Benchmark: An 8-point stencil "Game of Life" simulation using halo exchanges.
• Large Problem (30GB on Frontier): The CPU-free "Stream Ready Send" achieved a 28% higher maximum mean speedup compared to Cray MPICH when scaling to 8,192 GPUs (1,024 nodes).
• Small Problem (2GB): The CPU-free approach maintained better performance scaling up to 256 processes, whereas standard MPICH began to degrade earlier due to communication overheads.
• Tuolumne Results: Similar trends were observed, with the CPU-free approach showing significant advantages when utilizing 4 GPUs per node, though performance varied more on this system due to ongoing driver optimizations.

5. Significance

• Elimination of CPU Bottlenecks: This work proves that complex, two-sided MPI communication (including message matching) can be performed entirely on the GPU without CPU involvement, removing a major bottleneck in GPU-centric HPC.
• Performance Portability: By building on standard MPI concepts (persistent operations) and extending them rather than replacing them, the API offers a path toward performance portability that is easier for developers to adopt than low-level one-sided APIs.
• Scalability: The results demonstrate that removing the CPU from the fast path is critical for strong scaling on exascale systems (thousands of GPUs), where communication latency dominates execution time.
• Future Hardware Enablement: The design leverages features (triggered operations, deferred work queues) that are becoming standard in modern NICs, suggesting a viable future path for CPU-free communication across different architectures (e.g., NVIDIA InfiniBand, though porting is future work).

In conclusion, the paper successfully demonstrates that a co-designed API leveraging modern NIC features can significantly reduce communication latency and improve strong scaling for GPU-based HPC applications, achieving up to a 28% speedup in complex benchmarks.