ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression

Technical Summary of ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression

1. Problem Statement

Mixture-of-Experts (MoE) language models achieve scalability by activating only a small subset of experts per token, reducing per-token computation. However, the full routed expert pool must still be stored and served, creating a significant memory bottleneck for deployment. Existing post-training compression methods address this primarily through expert pruning (removing experts based on usage or importance) or expert merging (combining multiple experts into new, fused modules).

The authors argue that these approaches often conflate two distinct decisions:

1. Which expert parameters should be retained?
2. How should the router's original expert references be represented after compression?

Pruning deletes parameters, while merging constructs new weights, often requiring post-compression fine-tuning or complex fusion operators. The paper posits a complementary view: Expert-Pool Consolidation. This approach retains a smaller set of pretrained experts as reusable prototypes and deterministically remaps each original expert reference to one of these selected prototypes. This separates the storage of the reduced pool from the reuse structure, allowing the original router interface to remain unchanged without weight updates.

2. Methodology: ConMoE

The proposed framework, ConMoE, is a train-free, one-shot compression method that operates via prototype selection and deterministic reassignment.

A. Problem Formulation
The method defines a local scope $G$ containing one or more neighboring MoE layers. Given a reduction ratio $\rho$, the goal is to construct a reduced prototype pool $P_G$ of size $K$ from the original expert pool $E_G$. A reassignment map $m_G: E_G \to P_G$ is established, where each original expert $e$ is mapped to a selected prototype $p$.
The ideal objective is to minimize the reassignment cost $L_G(P)$, balancing the importance of retained experts against the error introduced by remapping others.

B. Prototype Scoring and Selection
Instead of solving the combinatorial optimization problem directly, ConMoE uses a heuristic scoring rule to select the top-$K$ experts as prototypes. For each expert $e$, two scores are computed using calibration data:

1. Routing-Conditioned Contribution ($a_e$): Measures the average contribution of an expert when it is selected by the router, weighted by routing probabilities and output norms.
2. Replaceability ($b_e$): Measures the distance to the nearest neighbor within the scope. A larger distance implies the expert is harder to substitute.

The final score is the product of the normalized contribution and replaceability scores ($s_e = \bar{a}_e \bar{b}_e$). The top-$K$ experts are selected as the prototype pool $P_G$.

C. Deterministic Reassignment
Once prototypes are selected, every original expert is assigned to its nearest prototype in $P_G$.

• No Weight Updates: The selected prototypes are reused directly as pretrained experts; no weight fusion or fine-tuning occurs in the default pipeline.
• Router Preservation: The original router remains unchanged. When the router selects a set of experts for a token, the consolidated operator redirects these calls to their assigned prototypes. If multiple routed experts map to the same prototype, their routing weights are aggregated into a single coefficient.

D. Scope and Cross-Layer Reuse
ConMoE supports local cross-layer sharing. A scope can contain multiple neighboring layers, allowing prototypes to be shared across them if they exhibit redundancy. However, the scope is bounded to avoid mismatching depth-specific transformations found in distant layers.

3. Key Contributions

1. Formulation: The paper formulates one-shot MoE compression as expert-pool consolidation, explicitly separating the decision of which parameters to store from how original router slots are represented.
2. Framework (ConMoE): A train-free remapping method that preserves the original router interface while reducing the logical routed-expert pool. It relies on calibration-based contribution and replaceability signals to select prototypes without gradient updates.
3. Empirical Validation: The authors demonstrate that remapping-based consolidation is a viable alternative to pruning and merging across multiple pretrained MoE models (Qwen3, DeepSeekMoE, OLMoE).
4. Ablation Insights:
   • Deterministic reassignment is identified as the most stable component of the framework.
   • Cross-layer sharing is beneficial but strictly local; expanding scopes too broadly (e.g., 8–16 layers) degrades performance due to depth mismatch.
   • Post-hoc weight fusion (e.g., Arcee, weighted averaging) is shown to be less stable than direct prototype reuse in this setting.

4. Experimental Results

Experiments were conducted on three models: Qwen3-30B-A3B, deepseek-moe-16b-base, and OLMoE-1B-7B-0125. Compression was evaluated at 25% and 50% logical routed-expert reduction ratios using six multiple-choice benchmarks (WinoGrande, ARC-C, ARC-E, BoolQ, HellaSwag, PIQA).

• Performance: ConMoE matches or outperforms strong pruning (Frequency, REAP) and merging (M-SMoE, HC-SMoE) baselines.
  • On DeepSeekMoE, ConMoE achieved the best average score at both 25% and 50% reduction.
  • On OLMoE, it performed best at 25% reduction and remained competitive at 50%.
  • On Qwen3, while pruning was strong (especially at 50%), ConMoE remained competitive and preserved strong performance on specific tasks like WinoGrande and BoolQ.
• Ablations:
  • Scope Size: Small scopes (1–4 layers) yielded comparable or best results. Larger scopes (8–16 layers) caused significant performance drops, confirming that expert substitutability is local.
  • Selection Policy: Routing-conditioned contribution was the dominant signal. Distance-only selection performed poorly, and replaceability served best as a regularizer for capacity allocation.
  • Fusion: Direct prototype remapping outperformed methods that applied post-hoc weight fusion, suggesting the gains come from the remapping structure itself rather than weight merging.

5. Significance and Claims

The paper claims that deterministic prototype reassignment offers a robust, train-free alternative to traditional pruning and merging. By treating compression as a consolidation of the expert pool into reusable prototypes, ConMoE achieves significant memory savings (logical reduction) while maintaining the original model's router interface and avoiding the instability often associated with weight fusion or fine-tuning.

The authors emphasize that their approach is modest regarding its scope:

• It is train-free but relies on calibration data for routing statistics.
• Cross-layer reuse is local; it does not assume global interchangeability of experts across the entire model depth.
• The reported savings are logical; realizing physical memory savings in deployment requires a shared-prototype runtime or checkpoint format, not just a materialized checkpoint.

The work distinguishes itself by showing that the "reuse structure" can be an explicit part of the compressed model rather than a by-product of deletion or fusion, providing a stable baseline for MoE compression.