Routing-Free Mixture-of-Experts

1. Problem Statement

Standard Mixture-of-Experts (MoE) models, while scalable, rely on centralized routing mechanisms that introduce rigid inductive biases and structural limitations:

• Router Bottleneck: Standard MoEs use small, external routers (gating networks) to assign tokens to experts. These routers lack sufficient capacity to store expert capabilities, forcing them to learn preferences via indirect trial-and-error, leading to suboptimal routing and unstable early training.
• Rigid Sparsity (Top-K): Conventional MoEs enforce a fixed Top-K selection, imposing uniform sparsity regardless of input complexity or expert capability. This prevents adaptive activation patterns where some inputs might benefit from more or fewer experts.
• Information Loss (Softmax): The Softmax operation forces a competitive probability distribution, discarding the absolute magnitude of expert activation scores. This suppresses the contribution of highly suitable experts if others happen to score slightly higher.
• Balancing Conflicts: Existing load-balancing strategies (Token-Choice vs. Expert-Choice) often treat balancing objectives as mutually exclusive, forcing predetermined activation patterns that may not adapt well to dynamic workloads.

2. Methodology: Routing-Free MoE

The authors propose Routing-Free MoE, a bottom-up architecture that eliminates all centralized routing components (routers, Softmax, Top-K, and hard-coded load balancing).

Core Architecture

• Decentralized Activation: Each expert independently determines its own activation based on its internal state.
• Internal Scoring: Instead of an external router, the model adopts the Autonomy-of-Experts (AoE) design. Each expert projects the input hidden state $x$ into a low-rank space ($r \ll D$) using a learnable matrix $A_{gate}$ and computes an internal preference score based on the norm of this projection: $\|x A_{gate}\|^2$.
• Learnable Bias & Thresholding:
  • A learnable per-expert bias $b_i$ is introduced to adjust the activation threshold for each expert individually.
  • The activation score is computed as $G_i(x) = \text{ReLU}(\|x A_{gate,i}\|^2 - b_i)$.
  • A global post-activation threshold $\theta$ is applied to make a binary decision: $f_i(x) = \mathbb{1}\{G_i(x) - \theta \geq 0\}$.
• Result: Experts activate purely based on their internal confidence scores. If the score exceeds the threshold, the expert is activated; otherwise, it remains inactive. This creates a fully decentralized system where the global activation pattern emerges from collective self-adjustment.

Unified Adaptive Load-Balancing Framework

To address the lack of centralized control, the authors introduce a unified framework that jointly optimizes Token-Balancing (uniform expert usage per token) and Expert-Balancing (uniform token distribution across experts) without hard constraints.

• Differentiable Proxy: Since the binary activation $f_i(x)$ is non-differentiable, the pre-threshold score $G_i(x)$ is used as a differentiable proxy for gradient flow.
• Composite Loss: The load-balancing loss $L_{LB}$ is a configurable interpolation of two terms:
  • $L_{EB}$ (Expert-Balancing): Penalizes experts receiving too many or too few tokens.
  • $L_{TB}$ (Token-Balancing): Penalizes tokens activating too many or too few experts.
  • Formula: $L_{LB} = \mu L_{EB} + (1-\mu) L_{TB}$, where $\mu \in [0, 1]$ is a tunable parameter.
• Adaptive Coefficient: The weight $\lambda_t$ of the load-balancing loss is dynamically adjusted during training based on the deviation of the current activation density from the target density $\rho_\infty$. This ensures the model converges to the desired sparsity without manual hyperparameter tuning.

3. Key Contributions

1. Routing-Free Architecture: A novel MoE design that completely removes external routers, Softmax, Top-K, and hard-coded balancing, allowing experts to self-activate based on internal confidence.
2. Unified Load-Balancing: A flexible framework that interpolates between token-balancing and expert-balancing objectives, allowing customizable resource allocation and adaptive optimization during training.
3. Empirical Superiority: Extensive experiments demonstrating consistent outperformance over standard MoE, AoE, and ReMoE baselines across multiple scales and benchmarks.
4. Scalability & Robustness: The method shows improved training stability, better tolerance to learning rate variations, and superior scalability compared to traditional routing mechanisms.

4. Experimental Results

The authors evaluated Routing-Free MoE on three model scales (Small, Medium, Large, up to ~0.8B parameters) trained on OpenWebText.

• Performance:
  • Language Modeling: Routing-Free MoE consistently achieved lower Perplexity (PPL) than standard MoE, AoE, and ReMoE across all scales (e.g., at Scale L, PPL dropped from 24.58 to 19.97).
  • Downstream Tasks: It outperformed baselines on 9 benchmarks (including PIQA, HellaSwag, ARC, and GLUE tasks), with a statistically significant average improvement of +0.77 percentage points.
• Training Dynamics:
  • Stability: Routing-Free MoE remained stable at higher learning rates where standard MoE collapsed (e.g., at Scale S, MoE failed at $\alpha=2\times10^{-3}$ while RFMoE remained stable).
  • Emergent Structure: Removing per-layer density constraints allowed the model to self-organize, with early layers showing high activation variance and late layers showing sharp rises, aligning with functional roles in LLMs.
• Efficiency & Deployment:
  • Expert Parallelism: In multi-device settings, Routing-Free MoE eliminates the blocking All-to-All communication bottleneck of standard MoE. It achieves up to 1.19x higher prefill throughput and 1.12x higher decode throughput at 4 devices.
  • Threshold Adaptation: The global threshold $\theta$ allows for inference-time trade-offs between compute and accuracy. The model remains robust even with extreme sparsity (down to 4% activation density) with minimal performance degradation.

5. Significance

This work fundamentally shifts the paradigm of MoE design from centralized orchestration to decentralized autonomy.

• Theoretical Insight: It demonstrates that expert specialization and load balancing can emerge naturally from local optimization without global coordination, challenging the necessity of complex routing mechanisms.
• Practical Impact: The elimination of routing bottlenecks significantly improves inference latency and throughput, particularly in distributed settings. The adaptive load-balancing framework offers a more flexible approach to resource allocation than static Top-K strategies.
• Future Direction: The paper suggests that future MoE designs should focus on enhancing expert autonomy and local decision-making rather than improving global routing algorithms, opening new paths for more efficient and scalable large language models.