💬 NLP

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

ExpertWeaver is a training-free framework that unlocks the inherent Mixture-of-Experts structure within pretrained dense LLMs by analyzing GLU activation patterns to partition neurons into universal and specialized groups, thereby outperforming existing methods for both dynamic structural pruning and MoE initialization.

Original authors: Ziyu Zhao, Tong Zhu, Zhi Zhang, Tiantian Fan, Jinluan Yang, Kun Kuang, Zhongyu Wei, Fei Wu, Yu Cheng

Published 2026-02-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziyu Zhao, Tong Zhu, Zhi Zhang, Tiantian Fan, Jinluan Yang, Kun Kuang, Zhongyu Wei, Fei Wu, Yu Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "All-Hands" Meeting vs. The "Specialized Team"

Imagine you have a massive, incredibly smart library (a Dense LLM). Every time someone asks a question, the librarian doesn't just look up the answer; they wake up every single book in the library to read it simultaneously. This is how current AI models work. They are powerful, but it's like trying to lift a mountain with your bare hands—slow, expensive, and energy-hungry.

To fix this, researchers invented Mixture-of-Experts (MoE). Think of this as hiring a team of specialists. Instead of waking up the whole library, you only wake up the specific experts needed for the job.

If you ask about cooking, you wake up the Chef.
If you ask about coding, you wake up the Programmer.
If you ask about history, you wake up the Historian.

This is much faster. But there's a catch: Training a team of specialists from scratch is incredibly expensive and takes years.

The Old Solutions: The "Brute Force" Approaches

Scientists tried to turn the "All-Hands" library into a "Specialized Team" by taking an existing dense model and cutting it up.

The "Scissors" Approach (Pruning): They tried to randomly cut out 75% of the books. But this is like cutting pages out of a dictionary at random. You lose the context, and the library stops making sense.
The "Copy-Paste" Approach (Upcycling): They tried to take a small team and clone them to make a big team. But if you clone the same person 10 times, you don't get 10 different experts; you get 10 copies of the same person. They all think the same way, which isn't helpful.

Both methods broke the natural flow of how the library works, requiring expensive retraining to fix the mess.

The New Discovery: The "Hidden Blueprint"

The authors of this paper, ExpertWeaver, realized something brilliant. They looked at the "All-Hands" library and noticed that even though all the books are there, they aren't all being read with the same intensity.

Inside the library, there is a Gated Linear Unit (GLU). Think of this as a smart bouncer at the door of every book.

For some questions, the bouncer says, "Open this book!" (High activation).
For others, the bouncer says, "Stay closed!" (Low activation).

The paper discovered that these "bouncers" have a hidden pattern. They naturally divide the books into two groups:

The Universal Books: These are books that get opened for almost everything (like a dictionary or a thesaurus). They are the "Shared Experts."
The Specialized Books: These are books that only get opened for specific topics (like a cookbook or a physics textbook). They are the "Routed Experts."

The Analogy: Imagine a busy restaurant kitchen.

Universal Neurons: The head chef who chops onions for every dish. They are always working.
Specialized Neurons: The sushi chef, the grill master, and the pastry chef. They only jump in when a specific order comes in.

The "All-Hands" model already has this structure built-in! It just hasn't been organized into separate stations yet.

The Solution: ExpertWeaver (The "Master Weaver")

Instead of cutting the library or cloning books, ExpertWeaver acts like a master organizer who simply rearranges the furniture based on how the books are actually used.

Here is how they do it, step-by-step:

The Observation (The "Calibration"):
They ask the library a few thousand different questions (from math to jokes to science). They watch which "bouncers" open which books. They record a "fingerprint" of how every single neuron behaves.
The Sorting (The "Layer-Aware" Strategy):
They realize that the kitchen changes as you go deeper.
- Top Layers (The Front of House): These handle general stuff. They need more "Universal Books" (Shared Experts).
- Middle Layers (The Kitchen): This is where the complex cooking happens. They need more "Specialized Books" (Routed Experts).
- Bottom Layers (The Plating): Back to general finishing touches.
- Old methods treated every layer the same. ExpertWeaver customizes the team for every single layer.
The Construction (The "Weaving"):
- They take the "Universal Books" and bind them into a Shared Expert that is always active.
- They take the "Specialized Books" and group them by topic (using a smart clustering algorithm) into Routed Experts.
- They build a Router (the waiter) that knows exactly which expert to call based on the question, using the original "bouncer" signals.

The Magic: They did all this without training. They didn't teach the model anything new. They just organized what was already there. It's like taking a chaotic group of people and instantly turning them into a perfectly coordinated orchestra just by telling them who plays the violin and who plays the drums.

The Results: Why It Matters

The paper tested this on two scenarios:

Instant Efficiency (Pruning): They turned a dense model into a sparse one immediately. It ran faster and used less energy, but kept 90%+ of its intelligence. It beat all other "cut-and-paste" methods.
The "Downcycling" Superpower: They took a huge, expensive model and turned it into a smaller, highly efficient "Expert Team." When they gave this new team just a tiny bit of extra practice (training), it performed better than models that were trained from scratch with double the data.

The Takeaway

ExpertWeaver is like finding a hidden instruction manual inside a complex machine. Instead of trying to rebuild the machine from scratch or smash parts of it, they simply followed the manual to reorganize the gears.

Old Way: "Let's cut the model in half and hope it works."
ExpertWeaver Way: "Let's look at how the model actually thinks, find its natural teams, and organize them so it can think faster."

This means we can get super-smart, fast AI models without needing to spend billions of dollars training them from scratch. We just need to know how to "weave" the experts that are already there.

1. Problem Statement

The Mixture-of-Experts (MoE) architecture offers a way to scale model capacity while maintaining computational efficiency by activating only a sparse subset of parameters per token. However, training high-quality MoE models from scratch is prohibitively expensive. Consequently, researchers focus on converting pre-trained dense Large Language Models (LLMs) into sparse MoE architectures.

Existing conversion methods generally fall into two categories:

Dynamic Structural Pruning: Converts dense models to MoEs with moderate sparsity (e.g., 25%) to balance performance and efficiency.
Downcycling: Initializes highly sparse MoEs (75%+ sparsity) by splitting dense MLPs into smaller experts, aiming to avoid the cost of training from scratch.

The Core Limitation: Existing methods often break the intrinsic activation patterns within dense models. They typically treat neurons as independent units or use uniform configurations across all layers, failing to capture the fine-grained neural-wise activation patterns revealed by the Gated Linear Unit (GLU) mechanism. This leads to suboptimal expert construction and necessitates additional router training, which can degrade performance.

2. Methodology: ExpertWeaver

The authors propose ExpertWeaver, a training-free framework that converts dense LLMs into MoE architectures by leveraging the inherent structure of GLU activation patterns. The method operates in three key stages:

A. Theoretical Insight: GLU as a Blueprint

The authors analyze the GLU mechanism (specifically SwiGLU), which uses a gating mechanism ( $Swish(xW_{gate})$ ) to modulate neuron activations. They identify four key insights:

Intrinsic Sparsity: GLU gating naturally induces token-level activation sparsity.
Universal Neurons: A core set of neurons is consistently activated across diverse tasks (task-agnostic knowledge).
Specialized Neurons: Other neurons exhibit co-activation patterns specific to task clusters (task-specific knowledge).
Layer Heterogeneity: The ratio of universal to specialized neurons varies systematically across layers (e.g., boundary layers are more universal, middle layers are more specialized).

B. The Three-Stage Process

Capturing Activation Patterns:
- The model is run on a multi-task calibration dataset (e.g., Flan-v2).
- For each neuron, the token-averaged gate activation is recorded across all tasks to form an activation profile.
Layer-Aware Expert Allocation:
- The authors calculate the Coefficient of Variation (CV) for each neuron's activation profile.
  - Low CV: Indicates a "Universal Neuron" (consistent activation).
  - High CV: Indicates a "Specialized Neuron" (selective activation).
- A layer-specific specialization ratio is computed. Layers with high specialization get smaller shared experts, while layers with low specialization get larger shared experts. This avoids a "one-size-fits-all" approach.
Expert Construction:
- Shared Expert: The most consistently active neurons (lowest CV) are grouped to form a single Shared Expert that is always activated.
- Routed Experts: The remaining specialized neurons are clustered using Balanced K-Means based on their activation patterns. Each cluster forms a Routed Expert.
- Router Construction: The MoE router is constructed without training by averaging the original gating weights ( $W_{gate}$ ) of the neurons assigned to each expert cluster. This preserves the pre-trained routing logic.

C. Application Modes

Dynamic Structural Pruning: Operates at low sparsity (e.g., 25%) without further training. It acts as a structured pruning mechanism where the router dynamically selects routed experts, while the shared expert is always active.
Downcycling: Converts dense models to highly sparse MoEs, followed by Continued Pretraining (CPT). In this mode, the router is switched to a standard softmax router to allow for further optimization.

3. Key Contributions

Discovery of Inherent MoE Structure: The paper systematically demonstrates that fine-grained GLU activation patterns in dense LLMs naturally reveal a coarse-grained MoE structure composed of universal and specialized neurons.
ExpertWeaver Framework: A novel, training-free method that partitions neurons based on activation patterns and layer-specific specialization, constructing both shared and routed experts without retraining the router initially.
Superior Performance: The method outperforms existing structural pruning and downcycling baselines in both zero-shot inference efficiency and downstream task performance after continued pretraining.

4. Experimental Results

The authors evaluated ExpertWeaver on models like Qwen2.5-7B and LLaMA3-8B across multiple benchmarks (MMLU, HellaSwag, ARC, etc.).

Dynamic Structural Pruning (25% Sparsity):
- ExpertWeaver significantly outperformed training-free baselines like LLM-Pruner, FLAP, and CMoE.
- On LLaMA3-8B, it achieved a 5.6% relative improvement over CMoE.
- It preserved reasoning capabilities (e.g., on GSM8K and HumanEval) much better than static pruning methods, which often fail at high sparsity.
Model Downcycling:
- When used to initialize a highly sparse MoE (7B parameters, 3.5B active) from Qwen2.5-7B, followed by 200B tokens of CPT, the model achieved an average score of 63.5 on MMLU.
- This outperformed the OLMoE-1B-7B baseline (trained from scratch on 500B tokens) by 4.6 points, despite using significantly less training data.
- The method retained 87.6% of the original dense model's performance while activating only 25% of the MLP parameters.
Efficiency:
- Inference benchmarks on A100 and H100 GPUs showed ExpertWeaver achieved higher throughput (RPS, OTPS) and lower latency compared to the dense baseline.

5. Significance

Cost-Effective Scaling: ExpertWeaver provides a highly efficient pathway to create high-performance MoE models without the prohibitive cost of training from scratch or extensive router training.
Preservation of Knowledge: By respecting the intrinsic activation patterns of the pre-trained dense model, the method avoids the "knowledge loss" often seen in static pruning or naive downcycling.
Layer-Aware Design: The introduction of layer-specific expert configurations acknowledges that different layers in LLMs serve different functional roles, leading to more robust architectures.
Training-Free Initialization: The ability to construct a functional router and expert partition without training makes the method immediately applicable to any pre-trained GLU-based LLM.

In summary, ExpertWeaver effectively "weaves" the neurons of a dense model into a structured MoE architecture by decoding the latent functional specialization already present in the model's GLU gating signals, offering a superior alternative to current dense-to-MoE conversion techniques.