💻 computer science

HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference

This paper proposes HetRoute, a collaborative routing framework for distributed edge MoE inference that unifies transmission, computation, and quality costs into a single model to optimize expert placement and online routing, achieving significant reductions in latency and traffic while maintaining quality constraints.

Original authors: Xin Yuan, Ning Li, Wenchao Xu, Athanasios V. Vasilakos, Song Guo, Haijun Zhang

Published 2026-08-04

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Xin Yuan, Ning Li, Wenchao Xu, Athanasios V. Vasilakos, Song Guo, Haijun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant, complex puzzle, but the pieces are scattered across a neighborhood of different houses. Some houses have super-fast computers, others have slow ones, and some houses are connected by lightning-speed fiber optics while others are linked by bumpy, slow dirt roads. In the world of artificial intelligence, this is exactly what happens when we try to run massive "Mixture-of-Experts" (MoE) models. These are huge AI brains that don't use every single part of themselves for every question; instead, they only wake up a few specific "expert" parts to solve a problem. The challenge is figuring out which experts to wake up and where to send the question so the answer comes back fast, without getting stuck in traffic or losing accuracy. If we just send the question to the nearest house, it might be slow because that house's computer is tired or its hard drive is full. If we send it far away, it might get stuck in a traffic jam on a slow road. Scientists have been trying to find the perfect way to route these questions, but most previous methods were like traffic cops who only looked at one car at a time or only cared about how close the house was, ignoring the speed of the road or the condition of the computer inside.

This paper introduces a new, smarter system called HetRoute. Think of HetRoute as a super-organized delivery service that doesn't just look at one house or one road. Instead, it looks at the entire delivery route for a single puzzle piece at once. It considers everything: how fast the roads are between houses, how powerful the computers inside are, whether the computer is currently busy (like a line of people waiting), and even if the computer is using a "compressed" version of the puzzle piece to save space (which might make the answer slightly less perfect). HetRoute makes a unified plan for the whole group of experts needed for a single question, rather than making separate, greedy decisions for each one. By doing this, it found that it can make AI answers arrive up to 59.0% faster on average and reduce the worst-case delays by 58.0%. It also cuts down on the amount of data traveling between houses by 72.1%, all while keeping the quality of the answers almost as good as the original, uncompressed version.

The Problem: The "Smart" AI That Gets Lost

To understand why HetRoute is a big deal, we first need to understand the "Mixture-of-Experts" (MoE) model. Imagine a massive library where every book is an "expert" on a specific topic. When you ask a question, the library doesn't read every single book; it only pulls out the top few books (the "Top-k" experts) that are most relevant. This is efficient because you don't waste time reading books about cooking when you are asking about math.

However, in the real world, these libraries are often split up across many different servers (computers) located in different places, like edge servers near you. When a question comes in, the "Top-k" experts needed might be scattered across three different servers. The old way of handling this was like asking a friend to run to three different houses to grab three different books. If the friend runs to the nearest house first, they might find the book is locked in a basement (stored on a slow CPU) and has to wait for the key. Or, they might run to a far-away house that has the book on a high-speed shelf (in fast GPU memory), but the road there is clogged with traffic.

Previous methods tried to solve this by either:

Staying Local: Always trying to use the experts on the nearest server, even if that server is slow or busy.
Greedy Selection: Picking the "best" server for each expert individually, without realizing that picking the best for Expert A might force Expert B onto a terrible path, slowing down the whole group.

The paper argues that these old methods are flawed because they treat the experts as independent travelers. In reality, they are a team. If one team member is slow, the whole team is slow.

The Solution: HetRoute's "Team Captain"

HetRoute acts like a brilliant team captain who plans the whole mission before anyone leaves the starting line. It uses a "unified cost model," which is a fancy way of saying it has a single scorecard that weighs four different things at once:

Transmission Cost: How long it takes to send the question over the internet to a server.
Loading Cost: How long it takes to move the expert from a slow hard drive (CPU) to a fast memory bank (GPU) if it's not already there.
Computation & Queueing: How fast the server can think, and how long the question has to wait in line behind other questions.
Quality Penalty: If the server uses a "compressed" version of the expert to save space, how much does the answer suffer?

HetRoute works in two stages: Offline and Online.

The Offline Stage (The Map Maker):
Before any questions are asked, HetRoute looks at the network and decides where to put copies of the experts. It's not just about putting them on the closest server. It asks: "If we put a copy of this expert on Server B, will it save time later?" It also decides which experts should live in the fast "GPU" memory and which can stay in the slow "CPU" memory. Crucially, it creates "redundant" copies. Just like having a spare tire in your car, HetRoute puts extra copies of popular experts on different servers. This ensures that if one server is busy or broken, the team captain has other options.

The Online Stage (The Real-Time Navigator):
When a real question arrives, HetRoute doesn't just pick the nearest server. It looks at the entire group of experts needed for that question. It asks: "If we send Expert A to Server X and Expert B to Server Y, what is the total time?" It calculates the "bottleneck"—the slowest part of the team. If Server X is fast but Server Y is stuck in a traffic jam, HetRoute might decide to send both experts to Server Z, even if Server Z is a bit further away, because the whole team will finish faster together.

It uses a clever trick called "beam search" (like a flashlight scanning a few best paths at once) to find the perfect combination of servers without getting stuck in a maze of possibilities.

The Results: Faster, Smarter, and Safer

The authors tested HetRoute on a simulated network of 10 different edge servers with varying speeds and connections. They used three different large AI models to see how it performed.

The results were impressive:

Speed: HetRoute reduced the average time it took to get an answer by 59.0% compared to the best existing methods. It also cut the "tail latency" (the worst-case delays that happen when things go wrong) by 58.0%.
Traffic: It reduced the amount of data traveling between servers by 72.1%. This is huge because sending data over the internet is slow and expensive.
Throughput: The system could handle 2.13 times more questions per second than the other methods.
Quality: Despite being faster, the quality of the answers stayed very high. The "quality degradation" (how much the answer got worse) was kept within a tiny, pre-set budget of 2%.

The paper also proved mathematically that their system is "quality-safe." Even if the network gets super busy and the normal fast paths are blocked, HetRoute has a "fallback" plan. It will always route the question to a "full-precision" expert (the highest quality version) that is guaranteed to exist somewhere, ensuring the answer is never bad, even if it takes a little longer.

Why This Matters

This paper shows that we don't have to choose between speed and quality, or between local and remote computing. By treating the AI experts as a coordinated team rather than individual runners, and by planning the whole route based on real-time traffic and computer health, we can make powerful AI run smoothly even on the "edge" of the network (like on your phone or a local server). HetRoute suggests that the future of AI isn't just about building bigger models, but about being smarter about how we move them around. It turns a chaotic, traffic-jammed network into a well-oiled machine where every expert knows exactly where to go to get the job done fastest.

Technical Summary: HetRoute: Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference

Problem Statement
The deployment of Mixture-of-Experts (MoE) models over geo-distributed, heterogeneous edge servers presents a complex routing challenge that existing distributed inference methods fail to address holistically. While MoE architectures offer efficiency through sparse activation (Top-k experts), deploying them across edge networks introduces four interdependent cost factors:

Cross-server transmission: Heterogeneous Internet links with varying bandwidth and latency.
GPU-CPU offloading: The delay incurred when "cold" experts reside in CPU memory and must be loaded into GPU memory before execution.
Heterogeneous computation: Varying GPU capabilities and instantaneous queueing backlogs across servers.
Quantization quality loss: The trade-off between memory footprint (using lower precision) and inference quality.

Current approaches often treat these factors separately or rely on "local-first" heuristics. However, a locally available CPU-resident expert may be slower than a remote GPU-resident expert due to loading delays, and selecting the individually best server for each expert independently can lead to globally inefficient collaboration paths due to fan-out/fan-in coupling. The paper formulates this as a joint optimization problem of expert deployment, residency (GPU vs. CPU), and online routing under strict quality budgets.

Methodology
The authors propose HetRoute, a two-stage framework comprising an offline deployment phase and an online collaborative routing phase, unified by a per-assignment cost model.

Unified Heterogeneous Cost Model:
The framework defines a unified cost for assigning a target expert to a specific replica on a server. This cost explicitly combines transmission delay, GPU-CPU loading delay, computation time (including queueing), and quantization/substitution quality penalties. The layer-level latency is modeled as a bottleneck (maximum) of parallel branches, not a sum, capturing the synchronization constraints of Top-k routing.
Offline Stage: Routing-Cost-Coupled Deployment:
Instead of static placement, Algorithm 1 determines expert server placement, GPU/CPU residency, and replica precision based on the expected online unified cost.
- GPU Residency: Replicas are promoted to GPU memory based on their routing probability (how often they are actually selected online) rather than raw activation frequency, weighted by the offload latency saved.
- Redundant Replication: New replicas are added only if they reduce the expected online unified cost, balancing memory usage against the benefit of expanding the low-cost collaboration domain.
- Precision Selection: Replicas may be stored at different quantization levels to fit memory budgets while staying within quality constraints.
- Invariant: The offline stage guarantees that every expert has at least one full-precision exact replica on some server to serve as a quality-safe fallback.
Online Stage: Set-Level Collaborative Routing:
Algorithm 2 routes tokens layer-by-layer.
- Exact-First Policy: The router first attempts to use exact replicas. If none are admissible (due to quality or stability guards), it considers feasible substitute replicas.
- Set-Level Optimization: Rather than greedily selecting the best server for each of the $k$ experts independently, HetRoute treats the Top-k set as a whole. It minimizes the bottleneck layer cost (fan-out + parallel execution + fan-in) using exact enumeration for small candidate domains or beam search with 1-exchange refinement for larger domains.
- Emergency Fallback: If no admissible assignment exists under normal stability guards, the system routes the expert to the guaranteed full-precision exact replica. This relaxes the stability constraint (allowing queueing delay) but strictly maintains the quality budget.
- Adaptive Feedback: Unit costs are refined online based on realized system bottlenecks (e.g., if transmission becomes the bottleneck, its weight increases).

Key Contributions

Unified Modeling: The paper is the first to formulate distributed edge MoE inference as a heterogeneous cost-aware Top-k collaborative routing problem, jointly modeling transmission, GPU-CPU residency, queueing, and quantization quality.
Set-Level Routing Algorithm: A novel algorithm that routes the Top-k activated expert set as a complete collaboration group, explicitly capturing the coupling between fan-out, parallel execution, and fan-in, avoiding the sub-optimality of per-expert greedy selection.
Offline-Online Co-Design: A deployment strategy where offline decisions (placement, residency, precision) are optimized specifically to support the online routing objective, rather than being static or based solely on activation frequency.
Theoretical Guarantees: The framework provides proofs for fallback feasibility (quality safety), a bound on participating servers ( $\le k$ ), per-layer optimality for small domains, and polynomial online complexity.

Experimental Results
Evaluated on a trace-driven heterogeneous 10-server edge testbed using three MoE models (Switch-Base-8E, Qwen-MoE-A 2.7B, and Mixtral-8x7B), HetRoute demonstrates significant improvements over baselines (Prism, EdgeShard, MoE-Infinity, Petals):

Latency: Reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%.
Traffic: Cuts cross-server traffic by up to 72.1%.
Throughput: Achieves a 2.13x improvement in throughput compared to the best baseline.
Quality: Maintains inference quality degradation within the configured budget (e.g., <2%), with measured degradation on Mixtral-8x7B being minimal (e.g., perplexity increase from 5.32 to 5.39).
Ablation: Removing set-level routing or GPU-CPU residency optimization leads to significant latency increases, confirming the necessity of both components.

Significance and Claims
The paper claims that HetRoute fills a critical gap in edge intelligence by addressing the joint problem of heterogeneous edge MoE inference. It argues that existing methods, which focus on dense model partitioning, expert caching, or single-node memory offloading, cannot fully exploit the sparse activation property of MoE models in distributed edge environments. By breaking the "local-first" assumption and recognizing that a remote GPU-resident expert can be faster than a local CPU-resident one, HetRoute provides a unified framework that dynamically balances transmission, computation, offloading, and quality costs. The authors emphasize that their approach ensures quality safety through a rigorous fallback mechanism while significantly improving latency and throughput in resource-constrained, heterogeneous edge networks.

The Problem: The "Smart" AI That Gets Lost

The Solution: HetRoute's "Team Captain"

The Results: Faster, Smarter, and Safer

Why This Matters

More like this