🤖 machine learning

Distributed Training using an Intelligent Network

This paper proposes an intelligent network framework for distributed training across wide area networks that combines multicast and in-line FPGA aggregation with optimized synchronization schedules to overcome bandwidth and latency constraints, thereby narrowing the performance gap with colocated training.

Original authors: Nihar Shah, Ben Blier

Published 2026-08-28✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nihar Shah, Ben Blier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The world's most powerful artificial intelligence models are becoming too large to fit inside a single building. Training these systems requires thousands of computers working together, but the physical limits of power and space mean that no single data center can hold them all anymore. Instead, researchers must spread their computing power across multiple locations, sometimes separated by oceans. This creates a difficult problem: computers need to talk to each other constantly to stay in sync, but sending massive amounts of data across long distances is slow and expensive. The connections between these distant sites are often narrow and uneven, causing the computers to sit idle while they wait for information to arrive. If the computers cannot share their progress quickly, the entire training process slows down, wasting valuable time and energy.

A team of researchers from the DoubleZero Foundation has proposed a new way to solve this bottleneck by turning the network itself into an active helper. Instead of treating the internet connection as a passive pipe that simply carries data, they suggest using specialized hardware within the network to manage the flow of information. Their approach combines two existing technologies in a novel way for wide-area connections. First, they use a method called multicast, which allows a single computer to send a message that the network automatically copies and delivers to many different destinations at once, rather than sending separate copies to each one. Second, they place programmable chips, known as FPGAs, at the edge of the network near each computer cluster. These chips act as smart aggregators, gathering incoming streams of data from many different sources and merging them into a single, clean stream before it reaches the local computers. By doing the heavy lifting of copying and combining data inside the network, the system reduces the strain on the limited connections between the distant sites.

To make this system work effectively, the researchers also developed a new mathematical framework to decide exactly how and when the computers should talk to each other. In a traditional setup, every computer might try to talk to every other computer at the same time, which would clog the network. The new framework treats the network like a map with specific roads and traffic rules. It calculates the best way to group the computers into small, rotating circles of communication. In each round, a specific group of computers exchanges information while others wait, and then the groups shift in the next round. The goal is to find the perfect balance between how long the computers wait and how much information they share. The researchers tested this idea using a simulation based on a real, programmable network that spans nine cities across three continents. They modeled the actual travel times and connection speeds between cities like Tokyo, New York, and London to see how different grouping strategies would perform.

The simulations revealed that the best strategy depends heavily on the capabilities of the hardware. When the network chips had very little memory, the most efficient approach was to form small groups of three computers that rotated through different combinations. This allowed data to flow quickly without overwhelming the system's ability to hold information. However, when the researchers gave the chips more memory, the optimal strategy changed completely. With enough memory to handle the delays of long-distance travel, the system could support a strategy where every computer talked to every other computer simultaneously. This "all-to-all" approach, which was previously impossible over long distances, became the fastest method because it allowed information to spread to everyone in the shortest amount of time. The study showed that by combining these smart network technologies with a schedule that adapts to the hardware's limits, it is possible to narrow the gap between training computers spread across the globe and those sitting side-by-side in the same room.

The researchers emphasize that their work is currently a simulation based on a live network that is still being built. While the DoubleZero network exists and carries traffic, it does not yet have enough connected computer clusters to train a massive model in a real-world test. The results presented are a proof of concept, demonstrating that the theoretical gains are real and that the right combination of network hardware and scheduling logic can overcome the traditional barriers of distance. The findings suggest that the future of training giant artificial intelligence models will not just rely on faster computers, but on smarter networks that actively participate in the learning process, turning the vast distances between data centers from a weakness into a manageable part of the system.

Technical Summary: Distributed Training using an Intelligent Network

Problem Statement

Distributed training of large-scale models is increasingly constrained by the limitations of individual data centers due to power, permitting, and data sovereignty regulations. While training within a single data center is feasible due to high-bandwidth, low-latency, and symmetric connectivity, extending this to a Wide Area Network (WAN) introduces severe bottlenecks. The primary challenges include:

Bandwidth Constraints: WAN links, particularly egress/ingress links to islands of compute, are expensive and capacity-limited. A standard synchronous Stochastic Gradient Descent (SGD) approach would require terabits per second to exchange full model updates, which is infeasible.
Latency and Asymmetry: WAN topologies are heterogeneous with high latency (milliseconds vs. microseconds) and asymmetric bandwidth. This causes significant idling of compute resources while waiting for synchronization or results in the use of stale parameters.
Topology Heterogeneity: The varying distances and link capacities between islands make standard synchronization schemes (like ring all-reduce) inefficient or impossible without massive overhead.

Existing literature typically adapts algorithms to the network by reducing communication frequency (e.g., DiLoCo), compressing data (quantization/sparsification), or tolerating staleness. However, these approaches often treat the network as a passive constraint rather than an active participant.

Methodology

The paper proposes a two-pronged approach where the network actively participates in the training process, combining specific system technologies with a novel algorithmic optimization framework.

1. Systems Proposal: Active Network Technologies

The authors propose leveraging two specific technologies to alleviate ingress and egress bottlenecks across the WAN:

Multicast for Egress Optimization: Instead of a sending island transmitting $N-1$ unicast streams (one per receiver), the network uses multicast. The sender transmits a single payload, and the network replicates it at branch points in the distribution tree. This eliminates duplicate traffic on the core network and drastically reduces egress bandwidth usage.
In-Line FPGAs for Ingress Aggregation: To address the ingress bottleneck where an island would otherwise receive $N-1$ $N - 1$ distinct streams, Field-Programmable Gate Arrays (FPGAs) are deployed at the network edge. These FPGAs aggregate inbound streams before they reach the island's access link.
- Memory Management: Unlike data center switches, WAN FPGAs must handle asynchronous arrivals due to varying latencies. The system utilizes High-Bandwidth Memory (HBM) to buffer incoming weights. Aggregation occurs as weights arrive, managed by a hashing scheme (Weight ID) and eviction criteria (completion, timeout, or conflict).
- Loss Recovery: Since multicast precludes TCP, the system handles packet loss by attaching sequence numbers to post-aggregation packets. The FPGA retains a resend buffer for a specific window ( $t_{resend}$ ) to allow the receiving island to request missing packets, protecting the critical final hop.

2. Algorithms Proposal: Optimization Framework

The authors develop an optimization framework to generate "rich synchronization schedules" based on the network's physical topology and technological capabilities.

Rotating Cliques: Instead of all-to-all synchronization, islands are partitioned into disjoint "cliques" (groups) in each round. These cliques rotate over time to ensure global mixing.
Objective Function: The goal is to minimize a cost function representing information staleness.
- The function combines the round time ( $T$ ) and the average lifetime of disagreement ( $A$ ).
- $A$ is derived from "survival curves" that measure how quickly a specific generation of parameter updates diffuses across all islands.
- The objective is to minimize $T(A + 1/2)$ , trading off the time cost of a round against the quality of mixing.
Feasibility Constraints: A schedule is valid only if it fits within the round time $T$ $T$ and the FPGA memory constraints ( $M$ $M$ ).
- Completion Constraint: The time for the last byte to arrive at the destination must be $\le T$ .
- Memory Constraint: The FPGA must hold bytes in memory from the arrival of the first stream to the last. If memory is insufficient, islands must stagger their transmission offsets ( $t_j$ ) to align arrivals, which increases $T$ .
Solution Strategy: The optimal schedule is found by iterating through candidate clique topologies, calculating the minimum feasible $T$ for each (solving a linear program for offsets and stream rates), and selecting the schedule that minimizes the objective function.

Key Contributions

Systems Innovation: The extension of multicast and in-network aggregation (via FPGAs) from data center environments to the WAN. This is the first proposal to utilize these technologies specifically to solve WAN bandwidth asymmetry and latency for distributed training.
Algorithmic Framework: A synchronization scheduling framework that explicitly incorporates network topology, latency, bandwidth, and hardware memory constraints. It moves beyond static or heuristic schedules to an optimization-based approach that generates "rotating cliques" tailored to the specific network.
Integrated Design: The paper demonstrates that the systems and algorithmic components are mutually reinforcing. The algorithm leverages the capabilities of multicast and FPGAs, while the hardware enables the algorithm to achieve mixing properties previously thought impossible over WANs.

Results

The authors evaluate their proposal using a simulation based on the DoubleZero network, a live programmable WAN connecting nine cities across three regions (North America, Europe, Asia) with realistic latencies and 100 Gbps core links.

Schedule Comparison: The study compares four schedules: "Triangles" (regional cliques rotating to cross-region), "Pairs" (linear matching), "All-to-All," and "Regional."
- Without FPGA Memory: The "Triangles" schedule performed best, balancing fast error decay with feasible round times (~565ms). "All-to-All" was infeasible due to the inability to coordinate nine simultaneous senders without memory buffering.
- With FPGA Memory (32 GB): The "All-to-All" schedule became feasible with a round time of ~676ms. Due to its perfect mixing properties, it outperformed all other schedules, demonstrating that hardware capabilities directly dictate the optimal algorithmic strategy.
Comparison to Synchronous SGD:
- Standard Synchronous SGD (Ring All-Reduce) requires GPUs to idle ~80% of the time to wait for global synchronization.
- The proposed system, by tolerating a small amount of parameter staleness (1 second), achieves 5x faster training throughput.
- Even within Synchronous SGD, utilizing the proposed hardware (multicast/FPGAs) significantly reduced idling time compared to standard host-based aggregation.

Significance and Claims

The paper claims that the gap between distributed WAN training and the "gold standard" of colocated training can be narrowed by making the network an active participant.

Novelty: The authors assert that while multicast and FPGAs are known in data centers, their application across a WAN for training is novel. Similarly, the synchronization framework is unique in being deeply embedded in the network's physical properties rather than treating the network as a black box.
Modest Scope: The paper acknowledges that the DoubleZero network is currently a live network for financial markets and does not yet connect enough GPU clusters to train a model "in earnest" for a full empirical comparison against peer systems. Therefore, the results presented are simulations based on a realistic topology.
Future Outlook: The authors position this work as a theoretical and systems foundation for the next generation of models, arguing that private, programmable WANs with bespoke control and ample memory are the necessary infrastructure to overcome the scaling limits of individual data centers. They explicitly reject the notion that such approaches are only viable for limited infrastructure, arguing that the combination of active networking and optimized scheduling is the key to future distributed training.