🤖 machine learning

OrderMoE: An expert similarity driven distributed edge MoE inference

OrderMoE is a novel framework that accelerates distributed edge MoE inference by leveraging expert similarity to enable local substitution for remote experts, thereby significantly reducing latency and communication overhead while maintaining controllable inference quality.

Original authors: Xin Yuan, Ning Li, Quan Chen, Wenchao Xu, Athanasios V. Vasilakos, Song Guo, Haijun Zhang

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xin Yuan, Ning Li, Quan Chen, Wenchao Xu, Athanasios V. Vasilakos, Song Guo, Haijun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling city where super-smart computers (called Large Language Models) live. These computers are like brilliant librarians who can answer any question, write stories, or solve math problems. However, these librarians are so huge and heavy that they can't fit inside your phone or your local coffee shop's computer. Usually, to get an answer, your phone has to shout a question across the city to a massive data center, wait for the giant librarian to think, and then wait for the answer to shout back. This takes time and uses up a lot of the city's "road space" (bandwidth).

To make things faster, scientists have invented a new kind of librarian called a "Mixture of Experts" (MoE). Instead of one giant brain, this librarian is actually a team of many smaller specialists. When you ask a question, a smart manager (called a router) quickly picks just the two or three specialists needed for that specific task and ignores the rest. This makes the librarian much faster and lighter. But here's the tricky part: in a distributed edge system, these specialists are scattered across different local servers (like different coffee shops or neighborhood hubs) instead of being in one big building. If the router picks a specialist who is working in a coffee shop three towns away, your phone has to send the question all the way there and wait for the answer to come back. In a city with narrow, crowded roads, this travel time can be just as slow as talking to the giant data center.

This is where a new study called OrderMoE comes in. The researchers, working with a real testbed of eight physical servers, asked a simple but clever question: What if, instead of always sending a question to the exact specialist the router picked, we could use a different specialist who is right next door and does almost the same job?

The paper suggests that many of these specialists, even though they have different names, actually think in very similar ways. The authors found that by measuring how similar their "thought patterns" are (using something called router-induced logits), they could group these specialists into families. If the router picks a specialist who is far away, OrderMoE checks if there is a "cousin" specialist nearby who is similar enough to do the job. If there is, the question is handled locally, saving the long, slow trip across the city.

However, the researchers knew this was a delicate balancing act. Using a cousin instead of the exact specialist might save time, but it could also mean the answer isn't quite as perfect. To solve this, they built a smart traffic controller that decides, for every single question, whether it's safe to use a local cousin or if it's worth the wait to send the question to the exact specialist far away. This controller also looks ahead, thinking about where the next question might need to go, so it doesn't accidentally send a question to a server that will be a bad place for the next step.

When they tested OrderMoE on a real network of eight servers with different speeds and memory sizes, the results were impressive. The system managed to cut the average wait time (latency) by about 40% to 51% compared to other methods, and it slashed the amount of data traveling between servers by a huge margin. Perhaps most importantly, the quality of the answers barely dropped at all—only a tiny, almost unnoticeable amount. The study shows that by letting local servers swap in similar experts, we can make AI feel much faster and more responsive without needing to build bigger, more expensive data centers. It's a bit like realizing you don't need to drive to the next town to get a specific type of sandwich; the place down the street has a very similar one, and it's ready in seconds.

Technical Summary of "OrderMoE: An Expert Similarity Driven Distributed Edge MoE Inference"

Problem Statement
While Mixture-of-Experts (MoE) models offer a favorable trade-off between model capacity and computation cost for Large Language Models (LLMs), deploying them on resource-constrained and bandwidth-limited edge infrastructures remains challenging. Existing distributed MoE serving methods primarily rely on exact expert placement, caching, or communication scheduling. These approaches treat the activated expert for a token as a fixed computation unit; if the routed expert is not local, the token must be transmitted to a remote server for exact execution. This paradigm ignores the functional similarity among experts within the same MoE layer. In edge environments characterized by heterogeneous resources, dynamic workloads, and ordinary Internet-based interconnections (as opposed to high-speed data center interconnects), frequent cross-server token transmission introduces significant latency and unstable performance. The core problem addressed is how to reduce cross-server token transmission and keep token computation local without incurring unacceptable inference quality degradation.

Methodology
The paper proposes OrderMoE, a similarity-aware expert grouping and distributed deployment framework. The methodology consists of three main components:

Expert Similarity and Grouping Model:
- OrderMoE constructs an expert similarity model based on router-induced logits representations. It measures the functional similarity between same-layer experts using the cosine similarity of their routing response vectors over calibration tokens.
- It recognizes that expert similarity is layer-dependent: deeper layers tend to have higher similarity, while shallower layers exhibit stronger specialization.
- A layer-aware grouping strategy partitions experts into similarity groups. Deeper layers allow larger groups with looser similarity thresholds, while shallower layers use stricter thresholds. Each group contains a dominant expert (based on activation frequency) and non-dominant experts that can serve as feasible substitutes.
Similarity-Aware Distributed Deployment:
- The framework employs an importance-aware deployment strategy to distribute experts across heterogeneous edge servers under memory constraints.
- Representative Coverage: Experts from the same similarity group are deliberately dispersed across different servers to maximize the probability that any edge server hosts a feasible substitute for a given group.
- Redundant Deployment: Using remaining memory, the system redundantly deploys replicas of important experts (high activation frequency and high representativeness) to cover groups that are not yet locally available, prioritizing busy servers.
Token Trajectory-Aware Runtime Selection:
- A quality-aware and trajectory-aware online algorithm decides, for each token at each MoE layer, whether to invoke the remote target expert or use a feasible local substitute.
- Look-Ahead Mechanism: To avoid myopic decisions that might force future cross-server transmissions, the algorithm uses a depth- $H$ recursion based on expert transition statistics from a calibration set. It estimates the future cost-to-go of the token trajectory.
- Quality Budget: The algorithm enforces a per-user quality budget. It constructs a feasible candidate set that includes the exact target expert and similar substitutes, ensuring that the accumulated substitution quality loss does not exceed the allocated budget.
- Weight Adaptation: The trade-off weights between inference delay and quality loss are dynamically adapted based on runtime observations of network conditions and workload.

Key Contributions

Problem Formulation: The paper formulates the distributed edge MoE inference problem by jointly considering expert deployment, expert similarity, substitute execution, and token trajectory. Unlike existing methods, it allows tokens to use quality-feasible substitute experts and dynamically updates token locations.
Similarity-Aware Strategy: It proposes a novel expert grouping and deployment strategy that leverages functional similarity to improve local substitutability across edge servers, reducing the probability of remote expert invocation.
Trajectory-Aware Algorithm: It designs an online server-expert selection algorithm that jointly considers transmission delay, computation delay, substitution quality loss, server workload, and future token movement costs.
Implementation and Evaluation: OrderMoE is implemented on a real distributed edge testbed with 8 heterogeneous servers.

Experimental Results
Experiments were conducted on three MoE models (Switch-Base-8E, Qwen-MoE-A2.7B, and Mixtral-8x7B) using a real edge testbed. The results indicate that OrderMoE:

Latency: Significantly reduces average latency (e.g., 187ms for Mixtral-8x7B, a 40.1% reduction compared to MoE²) and tail latency.
Communication: Drastically reduces cross-server traffic (e.g., 1.5 GB per 1,000 tokens) and remote expert invocation ratios.
Throughput: Achieves higher inference throughput compared to baselines.
Quality: Introduces only small and controllable inference quality degradation (e.g., perplexity increase of only 0.3 on WikiText-103), maintaining quality retention above 99.3% across models.
Scalability: Demonstrates robustness to increasing numbers of edge servers, whereas exact-execution baselines suffer from increased latency as the probability of remote target location grows.

Significance and Claims
The paper claims that OrderMoE is the first work to exploit same-layer expert similarity for distributed collaborative deployment across multiple edge servers. Its significance lies in transforming the distributed deployment problem from "exact remote expert access" to "similarity-aware local substitute execution." By acknowledging that experts in the same layer are not isolated, OrderMoE offers a new paradigm for edge intelligence that reduces reliance on high-bandwidth interconnects. The framework successfully balances the conflicting objectives of minimizing communication overhead and preserving inference quality, making it particularly suitable for bandwidth-limited and dynamic edge environments where traditional cloud-based distributed deployment strategies are ineffective.

More like this