💬 NLP

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Marco-MoE is a fully open, highly sparse multilingual Mixture-of-Experts model suite that leverages efficient upcycling to achieve state-of-the-art performance-to-compute ratios and superior scalability across languages compared to denser competitors.

Original authors: Fan Jiang, Yu Zhao, Chenyang Lyu, Tianqi Shi, Yichao Du, Feihu Jiang, Longyue Wang, Weihua Luo

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fan Jiang, Yu Zhao, Chenyang Lyu, Tianqi Shi, Yichao Du, Feihu Jiang, Longyue Wang, Weihua Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a massive library that contains books in 64 different languages. The old way of building this library was to hire one giant, super-smart librarian who had to memorize everything in every single language. The problem? This librarian got overwhelmed. When they tried to learn a new language, they started forgetting the old ones, or they became a "jack of all trades, master of none." This is what researchers call the "curse of multilinguality."

Marco-MoE is a new, smarter way to build this library. Instead of one giant librarian, the authors built a team of specialized experts who work together, but only a few of them show up to work for any specific task.

Here is how they did it, broken down into simple concepts:

1. The "Specialized Team" (Mixture-of-Experts)

Think of the Marco-MoE model not as a single brain, but as a gym with 200 different personal trainers (experts).

The Old Way (Dense Models): Every time you ask a question, all 200 trainers try to answer at once. It's loud, expensive, and inefficient.
The Marco Way (Sparse MoE): When you ask a question in Spanish, the system only wakes up 8 specific trainers who are experts in Spanish. When you ask in Japanese, a different set of 8 trainers wakes up.
The Result: The library is huge (it has a lot of total knowledge), but for any single question, it only uses a tiny fraction of its brain power (about 5%). This makes it incredibly fast and cheap to run, even though it knows a lot.

2. The "Renovation" (Efficient Upcycling)

Usually, building a team of 200 trainers from scratch takes years and millions of dollars. Marco-MoE used a clever trick called "Upcycling."

Imagine they took an existing, well-trained "generalist" librarian (a dense model called Qwen3) who was already good at many things.
Instead of firing them and starting over, they renovated the office. They took the generalist's knowledge and sliced it up into smaller, specialized pieces.
They then gave these pieces a little "shake-up" (adding random noise) so they wouldn't all think the same way. This allowed them to turn one big, slow librarian into a fast, specialized team without having to retrain everything from zero.

3. The "Curriculum" (How They Learned)

The team didn't just read random books. They followed a strict, four-stage study plan:

Stage 1: They started with high-quality English and reasoning books to build a strong foundation.
Stage 2: They added more math and science problems to sharpen their logic.
Stage 3: They introduced 9 new languages (like Urdu and Kazakh) that were harder to learn, ensuring the team didn't ignore the "long-tail" languages.
Stage 4: They focused heavily on culture. They didn't just learn the words; they learned the context. They studied local news, cultural stories, and regional history so the model understands why people say things, not just what they say.

4. The "Practice Run" (Post-Training)

After learning the books, the team needed to learn how to talk to humans.

Step 1 (SFT): They practiced answering specific questions and following instructions.
Step 2 (On-Policy Distillation): This is like a student learning from a master. The model generated its own answers, and a "super-teacher" (a much larger AI) corrected them. The model then learned from its own mistakes, refining its style to be more helpful and accurate.

The Results: Small but Mighty

The paper claims that this approach created two main models:

Marco-Nano: A tiny model that activates only 0.6 billion parameters. It punches way above its weight, performing better than much larger models that activate 3 to 14 times more brain power.
Marco-Mini: A slightly larger model (0.86 billion activated parameters) that beats competitors with 3 to 14 times more active parameters.

The Key Takeaway:
Marco-MoE proves you don't need a massive, expensive brain to speak 64 languages well. By using a "specialized team" approach and a smart renovation strategy, they created models that are faster, cheaper, and better at handling diverse cultures than the current giants, all while being fully open for anyone to use and study.

They also found that the model naturally learned to group languages together (like grouping Spanish, French, and Italian) just like a human linguist would, showing that it truly understands the relationships between languages rather than just memorizing them.

1. Problem Statement

The paper addresses the "curse of multilinguality" in Large Language Models (LLMs). This phenomenon describes the trade-off where expanding a model's language coverage within a fixed parameter budget often degrades performance on individual languages due to capacity bottlenecks and cross-lingual interference.

Limitations of Dense Models: Existing small-scale multilingual models typically use dense architectures that struggle to balance broad linguistic breadth with deep task proficiency. They lack the architectural flexibility to scale across diverse language families without sacrificing performance.
Limitations of Existing MoE Upcycling: While Mixture-of-Experts (MoE) architectures offer a solution via conditional computation, current upcycling methods (converting dense models to sparse MoE) rely on coarse-grained expert replication. This involves duplicating entire Feed-Forward Network (FFN) blocks, which leads to redundant initializations, hinders expert specialization, and fails to capture the nuanced representations required for multilingual reasoning.

2. Methodology

The authors propose Marco-MoE, a family of compact, highly sparse multilingual MoE models designed to break traditional trade-offs. The methodology consists of three core pillars:

A. Fine-Grained Expert Upcycling (Drop-Upcycling)

Instead of replicating entire FFN blocks, Marco-MoE employs a sub-matrix splitting technique combined with Drop-Upcycling:

Sub-matrix Partitioning: The weights of the pre-trained dense model's FFNs are partitioned into $N$ slices along the intermediate dimension to create a "pseudo-MoE" framework.
Scaling Adjustment: To maintain mathematical equivalence with the original dense model, the sliced weights are scaled by a factor of $\lambda = N^{1/3}$ to prevent signal magnitude reduction during the transition from dense summation to sparse gating.
Drop-Upcycling: To break the symmetry of identical initializations, the authors randomly sample indices along the intermediate dimensions and drop the corresponding weights, re-initializing them with a Gaussian distribution derived from the statistics of the dropped weights. This promotes expert diversification and specialization from the start of training.

B. Four-Stage Pre-training Curriculum

The models are pre-trained on 5.1 trillion tokens covering 29 languages (later expanded to 64) using a progressive data mixture strategy:

Stage 1: High-quality English, reasoning, and instruction data mixed with multilingual web/QA data (19 languages).
Stage 2: Upsampling of reasoning corpora and Chinese data; downsampling of general English web data.
Stage 3: Introduction of 9 new low-resource languages (e.g., Bengali, Kazakh, Urdu) by reducing weights for English/reasoning domains.
Stage 4: Integration of curated synthetic cultural and regional data (including MCQs and cultural knowledge) to boost localized reasoning and cultural understanding.

Data Synthesis: The authors utilize advanced pipelines to generate high-quality synthetic data for low-resource languages, including:

Multilingual QA: Translating diverse English QA datasets using Qwen3-30B-A3B-Instruct with strict filtering.
STEM Data: Projecting English STEM knowledge into target languages.
Cultural Data: Using LLM evaluators to filter web-crawled documents for cultural density and generating synthetic regional MCQs via hierarchical topic expansion.

C. Post-Training with On-Policy Distillation (OPD)

To create the Instruct variants, the authors employ a two-stage pipeline:

Supervised Fine-Tuning (SFT): Using high-quality instruction-response pairs (Dolci-Instruct, scientific data, and culturally grounded data).
On-Policy Distillation (OPD): A cascaded distillation strategy where the student model generates its own trajectories but learns from dense per-token supervision provided by stronger teacher models (first Qwen3-30B-A3B, then Qwen3-Next-80B-A3B). This bridges the gap between off-policy efficiency and on-policy robustness.

3. Key Contributions

First Sparse Multilingual Upcycling: The first work to leverage MoE upcycling specifically to optimize multilingual performance in compact model sizes, moving away from coarse-grained replication to fine-grained expert specialization.
Open Transparency: Unlike many proprietary models, Marco-MoE is fully open, disclosing all pre-training datasets, data synthesis methodologies, and the complete four-stage training recipe.
Scalable Language Expansion: Demonstrated the ability to scale from 29 to 64 languages without the performance degradation typical of dense models, maintaining high proficiency in both high-resource and long-tail languages.
Structured Expert Activation: Analysis reveals that the model naturally learns structured expert activation patterns that mirror linguistic taxonomies (e.g., Romance, Slavic, East Asian languages share expert pools), while isolating linguistically unique languages (e.g., Thai, Arabic) to specialized experts.

4. Key Results

The paper evaluates Marco-Nano (0.6B activated params, 8B total) and Marco-Mini (0.86B activated params, 17.3B total) against state-of-the-art dense and MoE baselines.

Performance-to-Compute Ratio:
- Marco-Mini-Base achieves the best average performance on English (63.7) and Multilingual General (50.9) benchmarks, outperforming Qwen3-4B (which has 4x more parameters and significantly higher training compute) and Trinity-Mini (which has ~4.5x more activated parameters).
- Marco-Nano-Base (0.6B active) outperforms larger dense models like Llama3.2-3B and Gemma3-4B on multilingual benchmarks despite having fewer than 1/5th the activated parameters.
Multilingual & Cultural Proficiency:
- Marco-Mini-Instruct achieves the highest scores in Multilingual Cultural & Regional benchmarks (71.0), surpassing Qwen3-4B-Instruct (69.1) and Gemma3-12B-Instruct (67.7).
- The models show significant gains in low-resource languages (e.g., Urdu, Bengali, Kazakh), where the performance gap over dense counterparts widens significantly.
Efficiency:
- Marco-Mini-Instruct outperforms LFM2-24B-A2B (a 24B parameter MoE model) by a large margin while activating only 0.86B parameters (a 38% reduction in total parameters and 57% reduction in activated parameters compared to LFM2).
Expert Analysis:
- Pearson correlation heatmaps of expert activation patterns show that related languages (e.g., Spanish, French, Italian) share high activation correlations, while isolated languages (e.g., Thai, Hebrew) exhibit low correlations, confirming the model's ability to learn linguistically meaningful routing.

5. Significance

Redefining Efficiency: Marco-MoE establishes a new state-of-the-art for the performance-to-compute ratio in multilingual LLMs, proving that sparse architectures with efficient upcycling can outperform much larger dense and MoE models.
Democratizing Multilingual AI: By open-sourcing the full training recipe and datasets, the authors provide a transparent foundation for the community to develop high-performance, culturally aware models without the prohibitive costs of training from scratch.
Solving the "Curse": The work demonstrates that the "curse of multilinguality" can be mitigated not just by scaling data, but by architectural innovation (fine-grained MoE) that allows for specialized expert utilization for isolated languages and shared utilization for related languages, enabling scalable expansion to 64+ languages without interference.
Future Direction: The paper highlights the potential for modular, incremental language expansion and the necessity of high-quality synthetic data pipelines to support low-resource language development.