💬 NLP

MobileMoE: Scaling On-Device Mixture of Experts

The paper introduces MobileMoE, a family of sub-billion parameter on-device Mixture-of-Experts language models that optimize architecture and training to achieve superior performance and efficiency compared to existing dense and MoE baselines, enabling fast inference on commodity smartphones.

Original authors: Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Bringing "Super-Brains" to Your Pocket

Imagine you have a massive library of knowledge (a Large Language Model, or LLM). Usually, to get the best answers, you need to send your questions to a giant, cloud-based server farm. But what if you could carry that brain in your pocket, on your phone, without needing the internet?

For a long time, the only way to fit a smart brain on a phone was to make it "dense"—like a single, super-hardworking student who tries to remember everything and do everything for every single question. This is slow and takes up a lot of space.

MobileMoE is a new family of AI models from Meta that changes the game. Instead of one hardworking student, they use a team of specialists. This approach, called Mixture of Experts (MoE), allows the phone to run a much smarter brain that is faster, uses less battery, and fits in the memory of modern smartphones.

1. The Problem: The "One-Size-Fits-All" Bottleneck

Think of a standard phone AI (like the ones currently on the market) as a generalist.

The Analogy: Imagine a single chef in a tiny kitchen. If you ask for sushi, they chop the fish. If you ask for a cake, they bake it. If you ask for a steak, they grill it. They have to be good at everything, so they carry every tool in the kitchen with them.
The Issue: This chef is slow because they have to switch tools for every task, and the kitchen (your phone's memory) gets crowded with all the tools they might need, even if they aren't using them right now.

2. The Solution: The "Specialist Team" (MoE)

MobileMoE replaces the single chef with a team of experts.

The Analogy: Now, imagine a kitchen with a Router (a manager) and 64 different Specialist Chefs.
- One chef only knows how to bake cakes.
- One only knows how to grill steaks.
- One only knows how to make sushi.
How it works: When you ask a question, the Router quickly looks at it and says, "Oh, this is a math question! Send it to the Math Chef." The Math Chef does the work, and the other 63 chefs take a break.
The Benefit: Even though the whole team is huge (lots of total knowledge), only a tiny fraction of them are actually working at any given moment. This means the phone doesn't have to do as much heavy lifting, saving battery and time.

3. The "Sweet Spot": Finding the Perfect Team Size

The researchers didn't just guess; they used a Scaling Law (a mathematical recipe) to find the perfect team size for a phone.

The Discovery: They found that for a phone, you don't want a team that is too small (not smart enough) or too big (too heavy).
The Result: They found a "sweet spot" with 8 main experts, where each expert is actually made of 8 tiny sub-experts (fine-grained), plus one "Generalist" expert who is always on call to handle anything the specialists miss.
Why it matters: This specific mix allows the model to be incredibly smart while staying small enough to fit on a phone with 3–5 GB of memory (which is standard on phones like the iPhone 16 Pro or Samsung S25).

4. The Training: A Four-Stage Boot Camp

To make this team work perfectly, they trained them in four specific stages, like a rigorous boot camp:

Pre-training: The team reads a massive library of books (6 trillion words) to learn general language.
Mid-training: They focus on specific, high-quality topics like math, code, and science to sharpen their skills.
Instruction Fine-Tuning: They learn how to follow human instructions (like "write a poem" or "solve this equation").
Quantization (The Compression): This is the magic trick. They shrink the model's memory footprint by 4x (turning it into "INT4" format) without losing much smarts, so it fits easily on a phone.

5. The Results: Faster and Smarter on Real Phones

The team tested MobileMoE on real flagship phones (Samsung Galaxy S25 and iPhone 16 Pro) and compared it to the best existing phone AI (MobileLLM-Pro).

Speed: MobileMoE is 2 to 4 times faster at generating text than the dense models.
- Analogy: If the old model was a delivery truck stuck in traffic, MobileMoE is a motorcycle weaving through.
Smarts: It matches or beats the performance of much larger models. For example, the "Medium" version of MobileMoE is smarter than models that are 3–4 times bigger.
Efficiency: It uses less battery and memory because it only "wakes up" the specific experts needed for the task.

6. The "Last Mile": Making it Run on Your Phone

The paper highlights a major hurdle: most phones don't have built-in software to run this "team of experts" efficiently.

The Fix: The researchers built a custom engine (a special software kernel) that acts like a traffic controller. It organizes the data so the phone's processor can handle the specialists efficiently.
The Proof: They proved this works on real hardware. On an iPhone 16 Pro, the new model generated text 3.4 times faster than the old dense model, while using less memory.

Summary

MobileMoE proves that you don't need a giant cloud server to have a smart AI on your phone. By using a team of specialists instead of a single generalist, and by carefully tuning the team size and training process, they created an AI that is:

Smarter than current phone models.
Faster (up to 4x speedup).
Efficient (fits in your phone's memory and saves battery).

This opens the door for powerful, private, and instant AI assistants that live entirely on your device, without needing to connect to the internet.

Technical Summary: MobileMoE: Scaling On-Device Mixture of Experts

Problem Statement

While Mixture-of-Experts (MoE) architectures have become the standard for scaling large language models (LLMs) to hundreds of billions of parameters on cloud servers, their potential at sub-billion scales for on-device deployment remains largely unexplored. Existing on-device LLMs (e.g., MobileLLM, MobileLLM-Pro) are overwhelmingly dense, requiring increased active parameter counts and inference compute to improve quality. This creates a bottleneck for edge AI, where devices face strict constraints on memory (DRAM) and compute (battery life, latency). Although recent smartphone DRAM capacities have increased (e.g., 12–16 GB), a principled methodology for scaling MoE architectures under these specific mobile constraints—balancing total model size, active parameters, and inference cost—has not been established.

Methodology

1. On-Device MoE Scaling Law

The authors formulate a generalized on-device MoE scaling law to jointly optimize architecture under memory and compute constraints. The loss function $L$ is modeled as a function of active parameters ( $N_{act}$ ), training data size ( $D$ ), a monotonic transformation of the expert count ( $\hat{E}$ ), and architectural choices ( $x$ , including expert granularity $g$ and shared experts $s$ ).

The optimization objective minimizes loss subject to:

Compute Constraints: Training FLOPs ( $F_{train} = 6N_{act}D$ ) and per-token inference FLOPs ( $F_{inf} = 2N_{act}$ ).
Memory Constraints: Total memory footprint $M(N_{total}, T)$ , which includes quantized weights ( $b_w$ -bit) and KV cache ( $b_{kv}$ -bit) at context length $T$ .

2. Architectural Design via Divide-and-Conquer

Using the scaling law, the authors identify an optimal "sweet spot" for on-device MoE through three controlled ablation studies:

Number of Experts ( $E$ ): Moderate sparsity is optimal. Increasing $E$ reduces loss but with diminishing returns beyond $E=8$ . $E=8$ is selected as the practical sweet spot for memory-constrained devices.
Expert Granularity ( $g$ ): Fine-grained experts (splitting each expert into $g$ sub-experts) allow for more diverse routing paths. The study finds $g=8$ to be the compute-optimal granularity, offering significant loss reduction over coarse-grained experts without incurring excessive overhead.
Shared Expert ( $s$ ): Incorporating a shared, always-on expert (a dense pathway) complements the routed specialists. The configuration with a shared expert ( $s=\checkmark$ ) achieves lower loss than the routed-only counterpart under fixed compute.

Resulting Architecture: The derived MobileMoE family consists of three scales (S, M, L) with active parameters of 0.3B, 0.5B, and 0.9B, and total parameters of 1.3B, 2.8B, and 5.3B, respectively. The design uses $E=8$ , $g=8$ (64 fine-grained routed experts), and a shared expert with top-4 routing.

3. Four-Stage Training Recipe

To realize the architectural advantages, a comprehensive training pipeline is employed on open-source datasets:

Pre-training: ~6T tokens with a web-heavy mixture to ensure broad linguistic coverage and expert specialization. Techniques include auxiliary-loss-free balancing, router z-loss, and grouped MLP (GMM) kernels for efficiency.
Mid-training: ~500B tokens focusing on high-quality domain-specific data (math, code, knowledge) to extend context to 8k and sharpen expert specialization.
Instruction Fine-tuning (SFT): ~80M samples across diverse domains using dropless token dispatching to preserve structured instruction signals.
Quantization-Aware Training (QAT): INT4 weight quantization (group size 32) with INT8 dynamic activations. Crucially, the router weights are kept in FP32 to maintain routing stability, adding negligible memory overhead.

4. Efficient On-Device Inference

To bridge the gap to deployment, the authors implement a custom fused MoE kernel in ExecuTorch. This kernel:

Converts sparse expert dispatch into dense grouped GEMMs by reordering tokens.
Fuses all sub-operations (routing, dispatch, projections, activation, scatter) into a single operator to amortize kernel launch and quantization overhead.
Enables efficient inference on commodity smartphone CPUs (Snapdragon 8 Elite, Apple A18 Pro) and GPUs.

Key Results

Benchmark Performance

Across 14 foundational benchmarks (commonsense, knowledge, science, reading, reasoning), MobileMoE establishes a new Pareto frontier:

Efficiency vs. Dense Models: MobileMoE-S/M match or exceed leading dense on-device LLMs (e.g., MobileLLM-Pro, Llama 3.2 1B) using 2–4× fewer inference FLOPs at comparable memory footprints.
Efficiency vs. MoE Models: MobileMoE-M matches the accuracy of the state-of-the-art OLMoE-1B-7B with ~60% fewer active and total parameters. MobileMoE-L surpasses OLMoE-1B-7B with 30% fewer active parameters and a 23% smaller memory footprint.
Token Efficiency: MobileMoE-L achieves superior performance with only ~6T pre-training tokens, outperforming dense baselines trained on 1.5–2× more tokens (e.g., Llama 3.2 1B at 9T).

On-Device Runtime Profiling

Deployed on Samsung Galaxy S25 and iPhone 16 Pro with INT4 weights:

Latency: MobileMoE-S achieves 1.8–3.8× faster prefill and 2.2–3.4× faster decode compared to the dense MobileLLM-Pro baseline at comparable INT4 weight memory.
Memory: At 8k context, MobileMoE-S consumes up to 22% less peak RAM than the dense baseline. MobileMoE-L fits within 5 GB of DRAM at 8k context, confirming feasibility for modern smartphones.
Accuracy: MobileMoE-L delivers substantially higher accuracy (Avg 57.8 vs. 45.5 for MobileLLM-Pro) with only a moderate runtime cost, which diminishes as context length increases.

Significance and Claims

The paper claims to present the first sub-billion-active MoE family optimized specifically for edge deployment, derived from a novel on-device scaling law. Its significance lies in three main contributions:

Theoretical Foundation: It provides a principled framework for MoE design under joint memory and compute constraints, identifying moderate sparsity ( $E=8$ ), fine-grained experts ( $g=8$ ), and shared experts as the optimal configuration for mobile devices.
Training Efficiency: It demonstrates that MoE architectures can achieve state-of-the-art on-device performance with significantly fewer training tokens (~6T) compared to dense counterparts, validating the learning efficiency of MoE at sub-billion scales.
Practical Deployment: It bridges the "last mile" to mobile deployment by providing the first efficient MoE inference on commodity smartphones. The results confirm that MoE is not just a cloud-scale architecture but a practical, efficient path for next-generation edge AI, enabling low-latency, privacy-preserving, and cost-effective applications on smartphones and wearables.

The authors conclude that while existing on-device models remain predominantly dense, MobileMoE demonstrates that MoE offers a superior alternative for the edge, establishing a new Pareto frontier that balances accuracy, memory, and latency.