Streami: An MPI Data-Parallel Library to Compute Field Lines on GPUs

This paper introduces Streami, an open-source, extensible GPU-accelerated library that interfaces with MPI applications to efficiently compute field lines in fluid flows for both post-hoc and in-situ analysis.

The Gist

Imagine you are trying to visualize the invisible currents of a massive, swirling storm inside a supercomputer. In the world of fluid dynamics, scientists use "field lines" (like streamlines) to draw the paths that tiny particles would take as they ride these currents. It's like dropping a million leaves into a river to see where the water is flowing.

The problem is that these simulations are huge. They run on supercomputers that have dozens of powerful graphics cards (GPUs) working together, split across many different machines. Usually, to draw these lines, you'd have to stop the simulation, copy all that massive data to a separate computer, and then try to draw it. But moving that much data is like trying to pour the entire ocean into a teacup; it's slow, expensive, and creates a bottleneck that stops everything.

Enter "Streami."

Think of Streami as a specialized, high-speed courier service that lives inside the supercomputer itself. Instead of moving the data out, Streami moves the "leaves" (the particles) directly between the different graphics cards that are already holding the data.

Here is how it works, broken down into simple concepts:

1. The "In-Situ" Delivery Service

Most visualization tools are like a delivery service that picks up a package, drives it to a warehouse, sorts it, and then ships it out. Streami is different. It's like a teleportation network built right into the factory floor.

• The Setup: The supercomputer is divided into neighborhoods (data partitions), with each neighborhood managed by a specific GPU.
• The Job: Streami lets a particle start in Neighborhood A, move through the flow, and if it crosses the border into Neighborhood B, it instantly "teleports" (via a fast, direct connection) to the GPU managing Neighborhood B.
• The Benefit: No data ever leaves the supercomputer. The simulation and the visualization happen at the same time, on the same machines, without the slow "truck ride" of copying data.

2. The Two Layers of the Library

The paper describes Streami as having two "languages" or layers:

• The Low-Level Layer (The Engine): This is the heavy machinery written in a very fast, technical language (CUDA/C++). It's the part that actually calculates the math for every single particle, checks which neighborhood it's in, and handles the instant teleportation between computers. It's designed to be as fast as physically possible, using "templates" so it can adapt to different types of data grids without slowing down.
• The High-Level Layer (The Dashboard): This is the user-friendly interface (written in C++). It's like the steering wheel and dashboard of a car. Scientists don't need to know how the engine works; they just tell the dashboard, "Draw me a stream of particles starting here," and the dashboard handles the complex math and communication behind the scenes.

3. Handling Different Terrains

Fluid simulations can be messy. Sometimes the data is a neat, uniform grid (like a checkerboard). Other times, it's a chaotic, jumbled mesh of shapes (like a pile of rocks).

• Streami is extensible. It has a "universal translator" that can understand both the neat checkerboard grids and the messy rock piles.
• If a scientist has a new, weird type of data, they can plug it into Streami's low-level engine without having to rebuild the whole system. The library figures out how to navigate the specific terrain of that data.

4. Real-World Testing

The authors tested Streami on a cluster with 16 powerful GPUs. They tracked 100,000 particles moving through a simulated galaxy.

• The Result: The system was incredibly fast, taking only about 1 to 2 milliseconds to move all particles one step forward.
• The Bottleneck: The only thing that slowed it down slightly was the "phone call" between the different computers (MPI communication) to say, "Hey, this particle is now in your neighborhood." Even then, it was very efficient.

Summary

In short, Streami is a tool that allows scientists to draw flow lines (like wind or water currents) directly inside a massive supercomputer while the simulation is running. It avoids the slow, painful process of copying huge amounts of data. Instead, it acts as a seamless bridge, letting particles hop instantly between different graphics cards, making it possible to visualize complex, massive fluid flows in real-time or near real-time.

The authors have made this tool open-source, meaning anyone can use it to build their own "interactive seed point placement" apps (where you can click and drop virtual leaves into a simulation to see where they go) or integrate it into their own scientific workflows.

Technical Summary

Technical Summary: Streami – An MPI Data-Parallel Library for GPU-Accelerated Field Line Computation

Problem Statement

Field lines (e.g., streamlines, streaklines) are essential primitives for analyzing fluid flows, particularly in Computational Fluid Dynamics (CFD). These simulations typically run on High-Performance Computing (HPC) systems utilizing distributed memory and multi-GPU nodes. While individual GPU computations are accelerated via APIs like CUDA or HIP, inter-node communication relies on MPI.

Current visualization workflows face significant bottlenecks:

1. Data Movement: Copying massive, distributed datasets to a separate workstation for post-processing is often infeasible due to volume and latency.
2. Lack of Abstraction: While libraries exist for scalar field visualization (e.g., OSPRay, ANARI) or in-situ orchestration (e.g., Catalyst, Ascent), there is no high-performance, data-parallel abstraction specifically for field line computation that seamlessly integrates with existing MPI domain decompositions.
3. Inflexibility: Existing tools often require resampling or remapping of custom flow fields to fit specific data structures (e.g., voxel grids), leading to quality loss or excessive memory usage. Furthermore, handling load imbalances and particle migration across MPI ranks (the "ping-pong" effect) is often tightly coupled to specific ecosystems like VTK, limiting portability.

Methodology

The authors present Streami, an extensible, open-source library designed to compute field lines directly on multi-GPU clusters using a data-parallel approach. Streami acts as a thin layer between CFD simulations and low-level rendering, supporting both in-situ and in-transit visualization.

Software Architecture

Streami is divided into two primary interfaces:

1. Low-Level Interface (CUDA/C++)
This layer provides the performance-critical building blocks, utilizing compile-time polymorphism to avoid runtime overhead.

• Vector Field Abstraction: The core abstraction is a VecField class with both host and device components. The device class implements two critical intrinsic functions:
  • sample(position): Reconstructs the vector field value at a specific 3D position.
  • destinationID(position): Determines the MPI rank (GPU) holding the data for a given position. This allows the library to handle arbitrary spatial partitions (e.g., uniform grids, tree-based structures) without imposing a specific data layout.
• Particle Advection: Implemented via CUDA kernels (e.g., using Runge-Kutta integration). The kernel samples the field, updates particle positions, and determines the target rank.
• Inter-GPU Communication: Particle exchange between ranks is handled by RaFI, an optimized library originally designed for ray tracing. RaFI treats particles as rays, buffering them and performing collective all-to-all transfers using CUDA-aware MPI. This abstracts the complexity of neighbor queries and halo regions from the user.

2. High-Level Interface (Object-Oriented C++)
This layer provides a Tracer class that orchestrates the visualization algorithms:

• Algorithm Support: Supports Streamlines (steady flow) and Streaklines (unsteady flow).
• Workflow: The Tracer manages the simulation time step, particle generation (seed placement), and the iterative calling of the low-level update() kernel.
• Flexibility: The API allows applications to pass data in its native format (structured or unstructured) without resampling, provided a compatible field type implementation exists.

Field Extensions

Streami supports different field representations through C++ templates. Currently, it supports:

• Structured Fields: Uniform voxel grids stored in global memory.
• Unstructured Fields: Grids containing tetrahedra, pyramids, prisms, and hexahedra, utilizing a GPU-accelerated Bounding Volume Hierarchy (BVH) via cuBQL for spatial queries.

Key Contributions

1. A Drop-in Library: Streami offers a simple API for integrating field line computation into existing HPC pipelines without requiring data copies or context switches.
2. Data-Parallel Efficiency: By leveraging the existing domain decomposition of the CFD application, Streami avoids the load-balancing issues and data transfer costs associated with traditional post-processing.
3. Extensibility: The library allows users to define custom vector field types and spatial partitioning schemes (e.g., tree traversals) via compile-time templates, eliminating the need for resampling.
4. Open Source Release: Streami is released under the Apache 2.0 license, providing a reference implementation for multi-GPU, MPI-based flow visualization.

Results and Performance

The authors evaluated Streami using an astrophysics dataset (1K³ voxels) on up to 16 A100 GPUs (two nodes).

• Performance: Single advection steps for 100,000 particles averaged 1–2 ms.
• Bottlenecks: The results indicate that performance is primarily limited by MPI communication overhead rather than local GPU computation.
• Scalability: The library successfully traced particles across distributed ranks, with the RaFI library handling the particle exchange efficiently.
• Unstructured Data: The library successfully adapted the structured dataset to an unstructured hexahedral format, though memory constraints prevented execution on very small GPU counts (1–4 GPUs) for the unstructured variant due to the overhead of the BVH structure.

Significance and Claims

The paper positions Streami as a necessary abstraction filling a gap in the state-of-the-art for multi-node, multi-GPU flow visualization. Its primary significance lies in its ability to:

• Interoperate with existing HPC simulation applications by reusing their domain decomposition.
• Minimize Overhead by avoiding data copies and context switches, making it suitable for both interactive post-processing and in-situ analysis.
• Provide a Foundation for future extensions to other field line types (e.g., pathlines) and complex field topologies without sacrificing the "speed-of-light" performance required for large-scale HPC visualization.

The authors conclude that while the library is modest in its current feature set (focusing on streamlines and streaklines), its design philosophy—separating the low-level field sampling from the high-level algorithm logic—provides a robust framework for efficient, data-parallel field line computation.