Technical Summary: HodgeCover

Problem Statement

Sparse Mixture-of-Experts (MoE) layers route tokens through a small subset of experts, creating significant parameter redundancy that can be compressed without retraining. Existing learning-free compression methods (e.g., REAP, REAM, STUN) rely on pairwise signals to rank and merge experts. However, these methods suffer from a fundamental structural blind spot: three experts may be pairwise compatible (low merge barriers between A-B, B-C, and C-A) yet collectively form an irreducible cycle where merging all three incurs a high, unpredictable barrier.

Current approaches that attempt to use triplet information (e.g., hypergraph spectral cuts, triplet-loss penalties) typically aggregate these signals into a sum-of-pairs surrogate or a binary veto. They fail to isolate the specific topological residue that arises when pairwise compatibilities do not sum to a coherent global mergeability. This paper identifies this residue as the harmonic kernel of the simplicial Laplacian on a 2-complex constructed from the MoE layer.

Methodology: HodgeCover

The authors propose HodgeCover, a learning-free expert-selection objective that explicitly targets this higher-order obstruction.

1. The Mergeability Complex

For a layer with $n$ experts, the authors construct an abstract simplicial 2-complex $K = (V, E, T)$:

• Vertices ($V$): The $n$ experts.
• Edges ($E$): All pairwise combinations, carrying a signal $b_{ij}$ defined as the KL divergence (merge barrier) between the original layer and the layer where experts $i$ and $j$ are merged.
• Faces ($T$): A curated set of triangles (triplets), carrying a signal $b_{ijk}$ defined analogously for merging three experts.

2. Hodge Decomposition

The pairwise barrier signal $b \in C_1(K)$ is decomposed using the Discrete Hodge Decomposition via the 1-Hodge Laplacian $L_1 = \partial_1^\top \partial_1 + \partial_2 \partial_2^\top$:
$$b = b_{\text{grad}} + b_{\text{curl}} + b_{\text{harm}}$$

• Gradient ($b_{\text{grad}}$): Represents expert-level potential (is expert $i$ broadly mergeable?). This is the only component captured by existing pairwise methods.
• Curl ($b_{\text{curl}}$): Represents coherent cycles around triangles (triangle-boundary corrections).
• Harmonic ($b_{\text{harm}}$): The irreducible residue. It lies in the kernel of $L_1$ and represents merge barriers that cannot be explained by any combination of vertex potentials or triangle boundaries. This component captures the "irreducible cycle" obstruction.

3. Selection Objective

HodgeCover selects a survivor set $S$ of size $k$ by greedily maximizing a submodular objective $\Phi(S)$ that covers the most critical topological features:
$$\Phi(S) = \sum_{i \in S} \text{sal}(i) + \lambda_e \frac{|C_E(S)|}{|E^*|} + \lambda_t \frac{|C_T(S)|}{|T^*|}$$

• Saliency: A standard per-expert score (adapted from REAP).
• Harmonic-Edge Coverage: Covers the top-$p\%$ edges with the highest harmonic mass ($|b_{\text{harm}, e}|$).
• Triplet-Triangle Coverage: Covers the top-$q_T\%$ triangles with the highest joint merge barriers.
• Router Redirect: Non-survivors are redirected to the nearest survivor, with a penalty applied if the path involves an edge carrying significant harmonic mass, preventing the reintroduction of the obstruction.

4. Hybrid Variant (HodgeCover+Wanda)

To achieve aggressive compression, the authors combine HodgeCover (Stage 1: expert count reduction) with Wanda (Stage 2: unstructured weight pruning) on the surviving experts. This composes two orthogonal compression axes within a single learning-free pipeline.

Key Contributions

1. Identification of Higher-Order Obstruction: The paper formally defines the simplicial mergeability complex and proves that the harmonic component of the Hodge decomposition is the unique linear-algebraic object capturing irreducible mergeability cycles that pairwise methods miss.
2. Empirical Validation of Non-Triviality: Diagnostics across three production MoE families (OLMoE, Qwen 3.5-35B, Qwen 3.5-122B) show that the harmonic energy fraction is non-trivial (29–62% of per-layer barrier energy), confirming that this obstruction is a significant structural property of learned MoEs.
3. HodgeCover Algorithm: A practical, learning-free selection objective that greedily covers harmonic-critical edges and triplet-critical triangles, achieving a $(1 - 1/e)$ approximation guarantee via submodular optimization.
4. State-of-the-Art Performance: The hybrid HodgeCover+Wanda method achieves the best perplexity and downstream accuracy across aggressive compression regimes (66% expert reduction) compared to existing baselines.

Experimental Results

The method was evaluated on three open-weight sparse MoE backbones at 33% and 66% expert reduction rates:

• OLMoE-1B-7B (64 experts/layer)
• Qwen 3.5-35B-A3B (256 experts/layer)
• Qwen 3.5-122B-A10B (256 experts/layer)

Key Findings:

• Aggressive Compression (66%): HodgeCover+Wanda consistently outperforms the strongest baseline (STUN+Wanda). On Qwen 3.5-35B at 66% reduction, it improves downstream task average (DS-Avg) by +12.6 percentage points (74.6% vs. 62.0%), driven largely by gains in GSM8K (85.5% vs. 22.1%).
• Perplexity: It achieves the best C4 perplexity on every model/rate cell and the best WikiText perplexity at 66% reduction.
• Ablation Studies:
  • Removing the Hodge decomposition (replacing with a soft triplet penalty) costs -5.74 pp in DS-Avg on Qwen 3.5-35B at 66% compression.
  • Removing the triangle term entirely costs -11.5 pp.
  • Using a binary triplet veto (hard constraint) causes a collapse of -30.5 pp, highlighting the necessity of the soft, coverage-based formulation.
• Mechanism Analysis: Unlike other methods that trade off retaining harmonic mass for curl/triplet mass (or vice versa), HodgeCover maintains a balanced retention of all four Hodge components (harmonic, gradient, curl, triplet-barrier), closely matching the pre-compression decomposition of the original model.

Significance and Claims

The paper claims that exposing the harmonic kernel of a learned MoE structure fundamentally changes the landscape of compression. By moving beyond pairwise signals to address higher-order topological obstructions, HodgeCover enables a new frontier of aggressive, learning-free compression that preserves model capability where previous methods fail.

The authors emphasize that this approach:

• Does not require retraining: It operates entirely on a small calibration corpus without fine-tuning, knowledge distillation, or LoRA.
• Is structurally motivated: It treats the mergeability landscape as a topological object rather than a simple ranking problem.
• Balances retention: It uniquely balances the retention of mass across all Hodge components, avoiding the structural biases introduced by methods that only optimize for pairwise or triplet sums.

The work positions the harmonic kernel not just as a diagnostic tool, but as a practical engineering criterion for selecting parameters in large-scale models.