Beyond Exascale: Dataflow Domain Translation on a Cerebras Cluster

This paper introduces the Domain Translation algorithm, which overcomes the limitations of traditional domain decomposition on Exascale systems by achieving unprecedented performance and perfect weak scaling (88% of peak) on a 64-node Cerebras CS-3 cluster to simulate planetary-scale tsunamis at 112 PFLOP/s.

The Gist

The Big Problem: The "Traffic Jam" in Supercomputing

Imagine you are trying to simulate a massive event, like a tsunami hitting a planet, or the weather changing over a year. To do this, supercomputers break the world into a giant grid of tiny squares (like a chessboard). Each square needs to talk to its neighbors to figure out what happens next.

The Old Way (Von Neumann Architecture):
Think of traditional supercomputers like a giant factory with a single, massive central warehouse (memory). All the workers (processors) have to run back and forth to this warehouse to get their instructions and data.

• The Bottleneck: As the factory gets bigger (more processors), the workers spend more time running to the warehouse and less time working. This is called the "Memory Wall."
• The Result: Even with super-fast computers, they spend a lot of time waiting. When you try to simulate a global event, the computers get stuck in traffic jams waiting for data to arrive from other parts of the cluster. They are fast, but they are inefficient.

The New Hardware: The "Wafer-Scale Engine"

The researchers used a special computer made by Cerebras Systems. Instead of a factory with a central warehouse, imagine a giant, flat city where every house (processor) has its own tiny pantry (memory) right in the kitchen.

• No Running: The workers never leave their houses. They just pass ingredients to their immediate neighbors.
• The Scale: This city is built on a single, massive silicon wafer (the size of a dinner plate), containing hundreds of thousands of these "houses."

The New Software: "Domain Translation" (The Moving Sidewalk)

Even with this amazing hardware, there was still a problem when connecting many of these cities together. If City A needs to send a message to City B, there is a tiny delay (latency) while the message travels across the internet.

In traditional computing, if you divide a simulation between City A and City B, the workers at the border have to stop and wait for the message to arrive before they can take their next step. This slows everything down.

The Solution: The Moving Sidewalk
The authors invented a clever trick called Domain Translation.

Imagine a long, moving sidewalk (like at an airport) that carries people from one side of a room to the other.

1. The Old Way: You stand still, and the world moves around you. If you need to talk to someone on the other side, you wait for them to walk over to you.
2. The New Way (Domain Translation): Instead of the data staying still and waiting, the data moves.
   • Imagine the "grid" of the simulation is printed on a giant conveyor belt.
   • As the simulation runs, the entire grid shifts one step to the right every second.
   • The workers (processors) stay in their fixed spots.
   • Because the grid is moving, a worker who was just talking to their neighbor on the left is now talking to a different neighbor on the right.
   • The Magic: The "message" (data) is always moving in the same direction as the conveyor belt. It never has to go "backwards" against the flow.

Why this is genius:
In a traditional setup, a worker at the edge of a computer chip has to wait for a message to travel all the way from the other chip (a delay of 10 microseconds).
With Domain Translation, the worker does 1,000 steps of work while the message is traveling. By the time the message finally arrives, the worker has already finished a huge chunk of work and is ready to use it immediately. The waiting time is completely hidden.

The Results: Breaking the Speed Limit

The researchers tested this on a cluster of 64 of these massive computer chips.

1. Speed: They simulated a tsunami caused by an asteroid hitting the ocean. They achieved 1.6 million time steps per second. To put that in perspective, if you were simulating a year of weather, you could do it in a fraction of a second.
2. Efficiency: They reached 88% of the computer's maximum theoretical speed. Most supercomputers only reach 1-5% of their max speed for these kinds of tasks because they are stuck waiting for data.
3. Power: They did this while using very little electricity compared to other supercomputers. It's like driving a car that gets 100 miles per gallon while going 200 mph.

The Real-World Impact: The Asteroid Tsunami

To prove it worked, they simulated a terrifying scenario: A massive asteroid hitting the ocean.

• They modeled the wave spreading across the entire planet.
• They could see the wave hit San Francisco Bay in their simulation.
• Because the computer was so fast, they could run these simulations in real-time or faster, which is crucial for predicting disasters or understanding climate change.

Summary Analogy

• Old Supercomputers: A relay race where runners have to stop at a central post office to pick up the baton. The post office is far away, so the race is slow.
• Cerebras + Domain Translation: A relay race where the baton is a ball rolling down a long, moving conveyor belt. The runners are standing on the belt. They just grab the ball as it passes them, do their job, and pass it to the next person. The ball never stops, and the runners never wait.

The Bottom Line: This paper shows that by changing how we move data (making the data move with the calculation) and using a new type of computer chip, we can finally unlock the true speed of supercomputers. We can now simulate complex physical events (like tsunamis and weather) with unprecedented speed and efficiency.

Technical Summary

1. Problem Statement

The paper addresses a fundamental bottleneck in high-performance computing (HPC) for physical simulations: the inability of traditional Von Neumann architectures to simultaneously achieve high simulation rates and high utilization in distributed cluster environments.

• The Memory Wall: While Exascale systems have increased spatial resolution (weak scaling), the temporal evolution rate (time steps per second) has stalled due to communication latency. In standard domain decomposition, grid points at subdomain boundaries must wait for data from neighboring nodes at every time step.
• Latency Dominance: In traditional fixed-partitioning methods, network latency ($\tau$) imposes a hard limit on the iteration rate ($1/\tau$). To mitigate this, methods like "ghost zones" (replicating boundary data) are used, but they sacrifice computational efficiency and power efficiency as the time-step rate increases.
• Low Utilization: Typical Earth system models on current supercomputers achieve less than 5% of peak performance, with most large-scale models operating between 1.2 and 8 PFLOP/s.

2. Methodology: Domain Translation

The authors introduce Domain Translation, a novel parallel algorithm designed to completely hide network latency by leveraging the principle of locality inherent in spatial computing architectures (specifically the Cerebras Wafer Scale Engine).

• Core Concept: Instead of keeping grid points static on specific processors, the algorithm shifts the mapping of grid points to processors by a distance equal to the stencil radius ($p$) at every time step.
• Unidirectional Flow: In a ring or torus topology, this shifting creates a unidirectional flow of data. A grid point only experiences network latency after it has traversed the entire width of a node's subdomain.
• Latency Hiding: By ensuring the subdomain size ($n$) is sufficiently large relative to the stencil radius ($p$) and latency ($\tau$), the time required to compute the interior points of a subdomain exceeds the time required for data to travel across the network.
  • Condition for Full Utilization: $n^2 > 2p\lambda/c$ (where $\lambda$ is latency and $c$ is compute time per point).
  • Result: The network latency is amortized over the subdomain width, allowing the system to run at full compute-bound efficiency regardless of inter-node latency.
• Hardware Synergy: The method is implemented on the Cerebras Wafer Scale Engine (WSE), a spatial architecture where memory is distributed alongside processing elements (PEs) on a single wafer. The WSE's flat memory hierarchy and low-latency Network-on-Chip (NoC) allow data to move in parallel with computation without global synchronization barriers.

3. Key Contributions

1. Novel Algorithm: The first distributed PDE solver using the Domain Translation algorithm, which eliminates the additive latency penalty of traditional domain decomposition.
2. Implementation on WSE Cluster: Successful deployment on a cluster of 64 Cerebras CS-3 systems (Wafer Scale Engines), demonstrating the scalability of spatial architectures for scientific computing.
3. Perfect Weak Scaling: The method achieves near-perfect weak scaling (efficiency >98%) across the cluster, maintaining high performance even as the problem size and node count increase.
4. Application to Complex Physics: The algorithm was applied to two distinct systems:
   • Heat Equation: Using 5-point and 9-point stencils to characterize performance limits.
   • Shallow Water Equations (SWE): A complex, non-linear hyperbolic system used to simulate a planetary-scale tsunami caused by an asteroid impact.

4. Key Results

The experiments yielded unprecedented performance metrics for stencil computations:

• Simulation Rate: The cluster achieved 1.6 million time steps per second.
• Peak Performance:
  • Unconstrained Environment: Achieved 112 PFLOP/s (PetaFLOPS) at 88% of peak theoretical performance.
  • Power-Limited Environment: Achieved 84.7 PFLOP/s with a power efficiency of 57 GFLOP/J.
• Efficiency:
  • The 9-point heat equation reached 88% utilization of the system peak, a record for stencil computations.
  • The Shallow Water Equations achieved 53% utilization of peak performance (limited by higher arithmetic complexity and variable counts), yet maintained perfect weak scaling.
• Scalability:
  • Weak Scaling: Efficiency ranged from 98.8% to 99.9998% when scaling from 4 to 60 nodes.
  • Strong Scaling: The system demonstrated strong scaling capabilities, effectively hiding the 10µs interconnect latency by using subdomains large enough to cover the latency with computation.
• Real-World Simulation: The authors simulated a planetary-scale tsunami (460m resolution) resulting from an asteroid impact, visualizing wave propagation across the globe and specific impacts like San Francisco Bay.

5. Significance and Implications

• Redefining Exascale Limits: This work proves that "Beyond Exascale" performance (in terms of time-step rate and utilization) is achievable by changing the algorithmic approach to communication, not just by adding more hardware.
• Latency Agnosticism: The Domain Translation method suggests that clusters in different cities (with millisecond latencies) could potentially be networked together for parallel applications, as the algorithm scales latency tolerance with memory size.
• Earth System Modeling: The results indicate a path toward order-of-magnitude improvements in throughput and power efficiency for critical applications like weather prediction, climate modeling, and uncertainty quantification.
• Architectural Validation: It validates the Wafer Scale Engine as a superior platform for PDE solvers compared to traditional clusters, specifically due to its ability to maintain data locality and eliminate memory hierarchy bottlenecks.

In conclusion, the paper demonstrates that by aligning the computational algorithm with the physical topology of spatial hardware, it is possible to overcome the "memory wall" and achieve unprecedented efficiency in large-scale physical simulations.