💬 NLP

Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models

This paper introduces Expert Tying, a method that shares expert parameters across consecutive transformer layers in Mixture-of-Experts models to nearly halve memory requirements with negligible impact on performance, thereby significantly improving the compute-to-memory trade-off for training and scaling large language models.

Original authors: Martin Jaggi

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Martin Jaggi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Huge Library" vs. The "Small Reader"

Imagine you are building a giant, super-smart robot librarian (a Large Language Model) to help people write stories or solve problems.

To make this librarian incredibly smart, you give them access to a massive library with billions of books (these are the "parameters" or "experts"). However, to keep the librarian fast and efficient, you decide that for every single sentence they read, they only open one or two specific books to find the answer. This is called a Mixture-of-Experts (MoE) model.

The Catch: Even though the librarian only reads a tiny fraction of the books at any given moment, you still have to keep the entire library physically present in the room (in the computer's memory). If the library has 1,000 books but the librarian only opens 10 at a time, you still need a room big enough to hold all 1,000 books. This makes the system very expensive and slow because the "room" (memory) is huge, even though the "reading" (computation) is small.

The Solution: "Tying the Loop"

The authors propose a clever trick called Expert Tying.

Imagine the librarian works in a long hallway with 32 rooms (layers). In a standard setup, every room has its own unique set of 1,000 books.

Old Way: Room 1 has Books A–Z. Room 2 has Books A–Z (but a different copy). Room 3 has Books A–Z (another copy). You need 32 sets of books.
New Way (Expert Tying): You realize that the books in Room 1, Room 2, and Room 3 are actually doing very similar work. So, you tie them together. You put a single set of books on a rolling cart that moves from Room 1 to Room 2 to Room 3. The librarian uses the same physical books in all three rooms.

The Result: You don't need 32 sets of books anymore; you only need 1 set for every 3 rooms. This cuts the size of the "library" (memory) by nearly half, but the librarian still reads just as fast because they are still only opening one book at a time.

The Secret Sauce: What to Tie and What to Keep Separate

The researchers didn't just tie everything together blindly. They ran experiments to figure out exactly what could be shared without making the librarian stupid.

Think of a room in the library as having four parts:

The Books (Experts): The actual knowledge.
The Librarian's Glasses (Attention): How they look at the words.
The Index Card (Router): Which book to pick.
The Desk (Normalization): How they organize the notes.

The Discovery:

Sharing the Books is Fine: You can share the same books across multiple rooms. The librarian can still do great work because the Glasses (Attention) are different in every room. The glasses change how the librarian looks at the words, which makes each room feel unique even if the books are the same.
Don't Share the Glasses: If you share the glasses too, the librarian gets confused and the quality drops.
The "Prelude and Coda" Rule: The very first room (where the librarian gets the book) and the very last room (where they write the answer) need their own unique books. If you force the librarian to use the shared cart for the start and end, the quality suffers. So, keep the first two and last two rooms unique, and tie the middle ones together.

The Results: Faster, Smaller, and Just as Smart

The paper tested this on three famous AI models (OLMoE, Qwen3, and DeepSeek). Here is what happened:

Memory Savings: By tying the experts in groups of 4, they reduced the total memory needed by 40% to 50%. It's like cutting the size of the library in half.
No Brain Damage: The "smartness" (perplexity and accuracy) barely dropped. It was almost as good as the original, huge model.
Speed Boost: Because the computer didn't have to load as many books into memory, the training process got 23% faster.
The "Reinvestment" Trick: Since they saved so much space by sharing books, they took that saved space and added more books to the shared cart. This created a model that was actually better than the original, even though it used the same total amount of memory.

Summary Analogy

Imagine a construction crew building a skyscraper.

Old Way: Every floor has its own unique, fully stocked toolbox. The truck carrying all the toolboxes is huge and expensive.
New Way (Tying the Loop): Floors 3 through 28 all share the same toolbox that gets passed up the elevator. Floors 1, 2, 29, and 30 keep their own special toolboxes.
Outcome: The truck is now half the size (saving money and space), the crew moves faster because they aren't hauling extra weight, and the building is just as strong. In fact, because they saved money on the truck, they bought better tools for the shared toolbox, making the building even stronger.

The Bottom Line: You can make AI models much smaller and faster by reusing the same "knowledge" across several layers of the model, as long as you keep the "glasses" (attention) unique for each layer and leave the very first and last layers alone.

Technical Summary: Tying the Loop - Tied Expert Layers in Mixture-of-Experts Language Models

Problem Statement

Mixture-of-Experts (MoE) architectures have become the standard for scaling Large Language Models (LLMs) by decoupling total parameter count from per-token compute. By activating only a small subset of expert feed-forward networks (FFNs) per token, MoE models achieve high capacity with efficient inference. However, a critical bottleneck remains: the memory footprint. Even though only a fraction of parameters are active during any forward pass, the full parameter set (dominated by the expert weights) must reside in training and inference memory.

This creates a tension with the emerging trend of "parameter efficiency" in reasoning and looped-depth models, which aim to extract more compute from each unique parameter. Standard MoE architectures are viewed as inefficient in this regard because they inflate memory with parameters that sit idle in any given pass. The paper seeks to reconcile the low per-token compute of MoE with the goal of higher compute-per-unique-parameter efficiency without sacrificing model quality.

Methodology: Expert Tying

The authors propose Expert Tying, an architectural modification that shares expert FFN parameters across a group of consecutive transformer layers while preserving layer-specific components.

Core Mechanism

Parameter Sharing: For a group of $g$ consecutive layers, the gate, up, and down projection weights of the $N$ experts are aliased (shared). This reduces the unique FFN parameters by a factor of $g$ .
Preserved Independence: Crucially, the router (gating network), self-attention projections (Q, K, V, O), and normalization layers remain independent for each layer.
Implementation: The method is implemented as a simple pointer assignment in the model code (e.g., Hugging Face Transformers). It requires no changes to the training or inference infrastructure other than ensuring the optimizer correctly accumulates gradients from the multiple use sites of the tied parameters.
Learning Rate Scaling: To manage gradient accumulation from $g$ layers, the learning rate for tied expert weights is scaled by $1/\sqrt{g}$ .

Design Rationale

The authors hypothesize that the FFN expert pool can be shared across nearby layers because per-layer attention ensures that each layer's effective operator remains distinct. Even with identical expert weights, attending over different mixtures of token positions produces different transformations at each depth.

Key Contributions

1. Controlled Component Ablation

Before scaling to production models, the authors conducted a controlled ablation study on a smaller architecture to determine which components can be tied without degrading quality.

Finding: Per-layer attention is the primary driver of layer distinctness. Untying attention (keeping it independent) improves loss significantly more than untying the router.
Router Impact: While untying the router increases routing diversity (cross-loop agreement drops), it has negligible impact on final loss if attention is tied.
Conclusion: The optimal strategy is to tie expert weights, keep attention per-layer, and treat the router as tied or untied with minimal cost difference.
Prelude/Coda: The first and last layers of the stack (embedding input and LM-head output) should remain untied. A "2+2" untied prelude and coda yields the largest architectural gain compared to fully recurrent designs.

2. Scaling to Production Architectures

The method was applied to three state-of-the-art MoE architectures: OLMoE, Qwen3-MoE, and DeepSeekMoE.

Configuration: Groups of $g=4$ layers share expert weights, with 2 untied layers at the start and end.
Parameter Reduction: This configuration reduces total parameter counts by 29% to 52% (depending on the architecture and group size) with negligible degradation in perplexity or downstream accuracy.
Throughput: The reduction in unique parameters leads to a 23.7% wall-clock training speed-up due to reduced weight-loading bandwidth and optimizer state updates.

3. Heterogeneous Width Expansion

The paper introduces a novel use of the saved parameters: reinvesting them into the tied middle layers.

Instead of simply compressing the model, the authors widen the tied middle layers (increasing the number of experts) to match the original total parameter count ("iso-parameter").
Result: Models with width-expanded tied layers consistently outperform the standard untied baseline at the same parameter count. This reframes expert tying not just as a compression technique, but as a depth-vs-width design axis.

4. Large-Scale Validation

Experiments were extended to a 7B parameter model (OLMoE-1B-7B).

The $g=4$ tied model achieved equivalent loss and downstream accuracy to the untied baseline while using less than half the total parameters (3.50B vs 7.12B).
The width-expanded variant further improved performance, surpassing the official OLMoE-1B-7B checkpoint trained on double the tokens, despite having half the parameters.

Results Summary

Memory Efficiency: Expert tying reduces the memory footprint by nearly 2× (up to 52% parameter reduction) with virtually no degradation in perplexity.
Quality Preservation: At $g=4$ , the loss penalty is minimal (e.g., +0.05 to +0.07 in validation loss), and downstream accuracy drops by at most 1.9%.
Performance Gains: Training throughput increases by 15.7% to 23.7% due to reduced data movement and optimizer overhead.
Iso-Parameter Superiority: When saved parameters are reinvested to widen the tied layers, the resulting models outperform standard untied MoEs of the same size.

Significance and Claims

The paper claims that Expert Tying successfully reconciles the opposing trends of MoE scaling (low per-token compute) and parameter efficiency (high compute per unique parameter). By exploiting the inherent parameter redundancy in MoE pathways, the method removes the memory penalty of sparsity without removing the sparsity itself.

The authors emphasize that their contribution is not merely the configuration (which shares similarities with prior work like Megrez2) but the controlled study that identifies which components can be shared. They demonstrate that attention, not routing, is the critical component for maintaining layer distinctness, allowing the largest parameter pool (experts) to be shared at the lowest quality cost.

The work positions expert tying as a "drop-in" modification for existing MoE codebases that advances the efficient training and scaling of next-generation LLMs, offering a highly favorable compute-to-memory trade-off. The authors remain modest regarding limitations, noting that experiments are currently limited to 7B parameters and that efficiency gains are lower bounds due to the lack of specialized tied-layer kernels.