🤖 AI

Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving

This paper presents a systematic cross-layer analysis demonstrating that lower-cost, switchless network topologies (specifically 3D full-mesh) are significantly more cost-effective than expensive high-bandwidth scale-up networks for Mixture-of-Experts LLM serving, while also revealing that current link bandwidths are often over-provisioned.

Original authors: Junsun Choi, Sam Son, Sunjin Choi, Hansung Kim, Yakun Sophia Shao, Scott Shenker, Sylvia Ratnasamy, Borivoje Nikolic

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junsun Choi, Sam Son, Sunjin Choi, Hansung Kim, Yakun Sophia Shao, Scott Shenker, Sylvia Ratnasamy, Borivoje Nikolic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-speed restaurant (the AI model) where thousands of chefs (GPUs) are working together to cook complex dishes (generating text). In the past, these chefs worked in small, isolated kitchens. But now, the recipes have become so huge that you need a whole city of kitchens working in unison. This is the world of Mixture-of-Experts (MoE) Large Language Models.

The problem? The chefs spend so much time running back and forth to swap ingredients and share notes that they aren't actually cooking. In fact, in some setups, nearly 60% of the time is spent just running around the kitchen, not cooking.

To fix this, the industry has been building "super-highways" (expensive, high-bandwidth networks) between the kitchens so the chefs can swap notes instantly. But this paper asks a simple question: "Are we overpaying for these super-highways?"

Here is the breakdown of their findings, using simple analogies:

1. The "Super-Highway" vs. The "Local Road"

Currently, most companies build Scale-Up networks. Think of this as a massive, multi-lane super-highway where every chef is connected to every other chef with a direct, lightning-fast fiber optic cable. It's incredibly fast, but it requires thousands of expensive traffic switches (routers) and miles of expensive cabling. It's like building a private jet for every chef just to deliver a single spice jar.

The researchers compared this to Switchless Topologies (like a 3D Torus or Full-Mesh).

The Analogy: Imagine a giant cube of chefs. Instead of a super-highway, they just pass notes to their immediate neighbors (up, down, left, right, front, back). If a note needs to go to a chef on the other side of the cube, it hops from neighbor to neighbor.
The Result: This is slower for a single message, but it's drastically cheaper because you don't need the expensive traffic switches.

2. The "Chef's Overlap" Trick (Software Optimization)

The paper highlights a clever software trick called Dual-Batch Overlap (DBO).

The Analogy: Imagine a chef is chopping vegetables (computation) while waiting for a delivery truck (communication). In the old way, the chef stands still and waits. With DBO, the chef starts chopping the next batch of vegetables while the truck is still delivering the current batch.
The Magic: If the chef is chopping fast enough, the delivery time becomes invisible. The chef never stops working. The researchers found that with this trick, the "slow" local roads (switchless networks) can keep up with the "fast" super-highways because the chefs are so busy cooking that they don't notice the traffic.

3. The "Sweet Spot" Discovery

The researchers ran simulations with different scenarios (short conversations vs. long stories, strict time limits vs. relaxed ones). They found:

We are over-provisioned: The current expensive super-highways are wider than necessary. If you cut the bandwidth in half (or even more), you save a fortune on hardware, and the "overlap trick" ensures the chefs still finish on time.
The Winner: The 3D Full-Mesh topology (where every chef in a row is directly connected to their neighbors, forming a mesh) turned out to be the "Pareto-optimal" choice.
- Translation: It offers the best balance. It's not the absolute fastest, but it's so much cheaper that you can build more of these clusters for the same money, which actually gives you more total cooking power.

4. The Future: Will This Change?

The paper looked at the next generation of computer chips (Blackwell and Rubin).

The Good News: Even with faster chips, the "local road" strategy remains the best value. The chips are getting faster at cooking, but the "traffic" (communication) is still the bottleneck. The cheap networks can still keep up.
The Caveat: If the chips get so fast that they can cook a million dishes a second, but the roads between them don't get any wider, the cheap networks might eventually struggle. However, for the foreseeable future, the cheap, switchless networks are the smarter financial choice.

The Bottom Line

The paper concludes that the industry is currently spending a fortune on "Ferrari-level" networks for AI clusters when a "reliable sedan" network would do the job just as well, thanks to smart software tricks.

By switching to these lower-cost, switchless networks, companies could save 20% to 56% on their infrastructure costs while delivering the exact same amount of AI service. It's a classic case of "don't buy a Ferrari if a Honda Civic gets you to the destination on time."

1. Problem Statement

The adoption of Mixture-of-Experts (MoE) architectures in Large Language Models (LLMs) has shifted serving workloads from single-node to cluster-scale. MoE models require Expert Parallelism (EP), necessitating frequent All-to-All (A2A) communication between accelerators (XPUs like GPUs/TPUs) to route tokens to specific experts.

Current Industry Trend: To minimize communication latency, the industry is investing heavily in expensive, high-bandwidth "scale-up" networks (e.g., Clos-based fat-trees with high-radix switches and NVLink-like interconnects).
The Core Question: Is this massive investment in expensive, switch-based infrastructure strictly necessary? Can lower-cost, switchless topologies (like 3D Torus or Full-Mesh) provide sufficient performance, especially when considering the total cost of ownership (TCO)?
The Gap: Existing research often optimizes for collective communication time (CCT) in isolation or focuses on training. Serving has distinct requirements (strict latency SLOs, smaller batch sizes, specific traffic patterns) and allows for aggressive computation-communication overlap, which previous network comparisons often overlook.

2. Methodology

The authors propose a holistic, cross-layer evaluation framework that models the interaction between compute, communication, software optimizations, and hardware costs.

Topologies Evaluated:
1. Scale-up: High-bandwidth, switch-based fat-tree (e.g., NVIDIA DGX clusters).
2. Scale-out: Standard Ethernet/InfiniBand switch-based networks.
3. 3D Torus: Switchless, multi-hop topology (e.g., Google TPU).
4. 3D Full-Mesh: Switchless, direct-link topology (e.g., Huawei UB-Mesh-Pod).
Workload & Metrics:
- Model: DeepSeek-V3 (671B parameters, 256 experts).
- Metric: Throughput per Cost (Tokens/sec per dollar). The primary SLO is Time Per Output Token (TPOT).
- Key Concept: Exposed Communication Time (ECT). The goal is to reduce ECT to near zero via overlap. If ECT $\approx$ 0, additional bandwidth yields diminishing returns.
Modeling Approach:
- Communication: Uses an extended Alpha-Beta (Hockney) model fitted to real NCCL benchmarks on NVIDIA H100s. It decomposes latency into per-message ( $\alpha_0$ ), per-round ( $\alpha_r$ ), and per-destination ( $\alpha_d$ ) costs, plus bandwidth ( $\beta$ ).
- Compute: Uses a profile-based method (extending tools like Vidur) to estimate kernel execution times under different parallelism configurations.
- Software Optimizations: Explicitly models Dual-Batch Overlap (DBO) and Speculative Decoding (SD) to determine how much communication time can be hidden.
- Cost Model: Calculates Total Cost of Ownership (TCO) including CapEx (GPUs, switches, links) and OpEx (power), normalized to a reference unit.

3. Key Contributions

First Systematic Cross-Layer Analysis: The paper is the first to jointly analyze network topology, link bandwidth, software optimizations (DBO/SD), and cluster cost specifically for MoE LLM serving.
Identification of the "Sweet Spot": The authors demonstrate that current industry provisioning (high-bandwidth scale-up) is often over-provisioned. Reducing link bandwidth can improve throughput-per-cost by up to 27%.
Superiority of Switchless Topologies: The study proves that lower-cost, switchless topologies (specifically 3D Full-Mesh) are more cost-effective than traditional scale-up networks across almost all serving scenarios.
Future-Proofing Analysis: A forward-looking analysis using projected specs for NVIDIA Blackwell and Rubin generations suggests the cost advantage of switchless networks will persist, provided interconnect bandwidth scaling keeps pace with compute/memory scaling.

4. Key Results

A. Link Bandwidth Over-Provisioning

Current scale-up systems often use 450 GB/s links. The analysis shows that for many scenarios (especially with DBO and SD), 150–300 GB/s is the "sweet spot."
Reducing bandwidth to this sweet spot improves throughput per cost by 6–27% because the extra bandwidth does not translate to higher throughput (due to compute bottlenecks or ECT already being zero) but significantly increases switch/link costs.

B. Topology Comparison (Cost-Effectiveness)

3D Full-Mesh is Pareto-Optimal: Despite being less widely adopted, the 3D Full-Mesh topology offers the best performance-to-cost tradeoff.
Performance Gains: Switchless topologies (Torus and Full-Mesh) improve cost-effectiveness by 20.6% to 56.2% compared to scale-up networks.
Mechanism:
- DBO Effectiveness: In scenarios with relaxed SLOs (TPOT $\ge$ 40ms) or short contexts, DBO successfully hides communication latency. Once ECT is zero, the lower raw bandwidth of switchless networks does not hurt throughput, but their lower cost drastically improves the metric.
- Long Context/Strict SLO: Even in long-context scenarios (where batch sizes are small and DBO is less effective) or strict SLOs (TPOT = 15ms), switchless networks remain competitive because the small message sizes reduce the impact of the bandwidth term ( $\beta$ ).

C. Future Generations (Blackwell & Rubin)

Scaling Trends: Compute and memory bandwidth are scaling faster than interconnect bandwidth in upcoming generations (e.g., Rubin: 6.57x memory bandwidth vs. 4x link bandwidth).
Implication: If this trend continues, the "sweet spot" bandwidth requirement might rise, potentially narrowing the gap. However, the authors show that even with higher requirements, switchless networks can match scale-up performance if software overhead ( $\alpha$ terms) is reduced or if link bandwidth is moderately increased (e.g., doubling bandwidth for Full-Mesh still costs less than Scale-Up).

5. Significance and Implications

Economic Impact: The paper challenges the prevailing "bigger is better" mindset in AI infrastructure. It suggests that hyperscalers can achieve the same serving capacity at a significantly lower cost by deploying switchless clusters (like Full-Mesh) rather than expensive fat-tree scale-up networks.
Design Guidance: System designers should prioritize cost-effectiveness (throughput per dollar) over raw throughput per cluster when cluster expansion is feasible.
Software-Hardware Co-design: The results highlight that software optimizations (DBO, SD) fundamentally alter the network requirements, making high-bandwidth hardware less critical than previously thought.
Architecture Shift: It validates the viability of alternative topologies (Full-Mesh) that were previously considered niche, potentially reshaping the design of future AI datacenters.

In summary, the paper argues that for MoE LLM serving, lower-cost, switchless networks (specifically 3D Full-Mesh) are the optimal choice, offering superior economic efficiency without sacrificing performance in the vast majority of practical serving scenarios.