Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling

Technical Summary: Dense2MoE

Problem Statement
The deployment of Large Language Models (LLMs) on resource-constrained edge devices (e.g., autonomous driving, edge AI) is severely hindered by the "inference memory wall." According to hardware Roofline theory, dense LLMs are memory-bound during autoregressive decoding, meaning that simply reducing theoretical FLOPs is insufficient to improve latency. Existing solutions face a fundamental trade-off:

1. Structured Pruning: Methods like UIDL reduce latency by discarding redundant layers but inevitably sacrifice model accuracy by removing Multi-Layer Perceptrons (MLPs), which serve as the primary knowledge repositories of LLMs.
2. MoE Upcycling: Converting dense models to Mixture-of-Experts (MoE) architectures increases capacity without linearly increasing dynamic compute. However, standard upcycling often suffers from "representation collapse" due to homogeneous expert initialization (duplicating native MLPs), requiring massive continual pre-training to induce specialization. Furthermore, recent sparse repurposing methods often focus on intra-layer redundancy, failing to address the macro-level inter-layer structural overlaps that cause redundant attention and KV-cache loading overheads.

Methodology: Dense2MoE
The authors propose Dense2MoE, a unified framework that integrates structured layer pruning with MoE upcycling via a novel technique called Layer-Fusion Upcycling (LF-UC). The method operates in three stages:

1. Inter-Layer Similarity Analysis: The framework identifies structural redundancy by analyzing adjacent layers for high cosine similarity in both decoder outputs and MLP inputs. Crucially, it imposes a strict feature norm matching constraint to ensure that reusing redundant MLPs does not introduce activation anomalies or scale shifts.
2. Layer-Fusion Upcycling (LF-UC): Instead of discarding redundant layers entirely, Dense2MoE physically prunes only the attention modules (MHA) of these layers to eliminate memory-bandwidth-heavy operations. The pre-trained MLPs from these pruned layers are preserved and repurposed as heterogeneous experts within the retained base layers.
   • This creates an expert pool where "base experts" are duplicated native MLPs and "supplementary experts" are the preserved MLPs from pruned layers.
   • A dynamic token-level router selectively dispatches tokens to these experts or bypasses them via residual paths.
   • This design leverages inter-layer structural heterogeneity to mitigate the "Winner-Takes-All" routing collapse common in homogeneous upcycling.
3. Continual Pre-Training: The modified architecture undergoes a lightweight continual pre-training phase (225B tokens) to optimize the routing mechanism and align the fused experts. This budget consumes less than 1% of the compute required to train a MoE model from scratch.

To determine the optimal pruning depth, the authors employ a hardware-aware reward metric inspired by MnasNet, mapping pruning constraints to the Pareto frontier of latency versus accuracy for specific target hardware (e.g., NVIDIA Jetson Thor-U).

Key Contributions

• Unified Framework: Dense2MoE seamlessly integrates structured layer pruning and MoE upcycling, resolving the trade-off between inference efficiency and model accuracy.
• Novel Architecture (LF-UC): By pruning attention bottlenecks while recycling MLPs as heterogeneous experts, the method reduces memory bandwidth consumption without destroying representational capacity.
• State-of-the-Art Pareto Optimality: The framework advances the Pareto frontier for on-device inference latency versus model accuracy, outperforming dense baselines, state-of-the-art compression methods, and standard upcycling approaches.
• Robust Scalability: The method is validated across varying model scales (0.5B to 7B) and distinct architectural families (Qwen2.5 and LLaMA2).

Experimental Results
Evaluated on 9 diverse benchmarks (including MMLU, ARC, GSM8K, and HumanEval) using Qwen2.5 and LLaMA2 models:

• Performance vs. Latency: The optimal Dense2MoE configuration (retaining 19 layers from a 24-layer Qwen2.5-0.5B seed) achieved a physical inference latency of 271.0 ms on the Thor-U SoC (down from 307.5 ms for the seed model) while achieving an average benchmark score of 47.31, significantly surpassing the dense baseline (37.53).
• Comparison with Baselines: Unlike traditional pruning methods (e.g., UIDL, LLM-Streamline) and sparse repurposing approaches (e.g., ToMoE, LLaMA-MoE), which suffered from severe capacity collapse in reasoning and coding tasks, Dense2MoE maintained robust representational retention. It outperformed the dense baseline by over 9.78 percentage points in average score while activating only 0.42B parameters.
• Scalability: When applied to larger models (Qwen2.5-1.5B and LLaMA2-7B), Dense2MoE consistently outperformed their dense counterparts by 3.04 and 3.54 percentage points, respectively, after fine-tuning on the same 225B-token dataset.

Significance
The paper claims that Dense2MoE establishes a practical, industrial-standard recipe for repurposing open-source dense LLMs into highly efficient on-device architectures. By systematically exploiting inter-layer structural redundancy, it enables the joint optimization of inference latency and model accuracy. The work demonstrates that it is possible to break the inference memory wall and preserve complex reasoning capabilities without the prohibitive costs of training MoE models from scratch or the accuracy degradation associated with aggressive pruning. This provides a flexible continuum of models optimal for latency-critical edge deployments.