Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search

Technical Summary: Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search

Problem Statement
The paper addresses the challenge of exploring the vast combinatorial space of heterogeneous Mixture-of-Experts (MoE) architectures. Specifically, it targets the construction of 4-Expert MoE (MoE4) ensembles using the LEMUR neural network dataset, which contains 29 distinct base architecture families. The theoretical search space for selecting 4 unique families from 29 is $\binom{29}{4} = 23,751$ possible combinations. Prior work within the LEMUR ecosystem had manually designed only a small number of configurations, leaving the majority of this space unexplored. The authors aim to automate the discovery of high-performing heterogeneous ensembles without relying on Large Language Models (LLMs) for the generation loop, which had previously shown limited success in 2-expert scenarios.

Methodology
The authors developed a fully automated, 28-day search pipeline built on the NNGPT framework, consisting of four sequential stages:

1. Expert Pool Construction: The system queries the LEMUR database to retrieve 29 established base architecture families trained on CIFAR-10. It filters out LLM-generated mutations, existing MoE models, and lowercase-prefixed generated models to ensure a clean pool of distinct architectural families.
2. Deterministic Code Assembly: A generator script (AlterHeteroMoE4.py) programmatically assembles candidate MoE4 models. It retrieves source code for four selected families, renames classes to avoid collisions, merges imports, and instantiates a wrapper template. The architecture employs a HeterogeneousGate (a convolutional gating network with temperature scaling) to route inputs. The output is a weighted sum of expert logits: $y = \sum_{i=1}^{4} g_i(x) \cdot f_i(x)$. Training utilizes AdamW, mixup augmentation ($\alpha=0.2$), label smoothing, and cosine-annealed learning rates.
3. Multi-Stage Validation: Before GPU evaluation, candidates undergo three filters:
   • Syntax Check: Python's compile() verifies code correctness.
   • CPU Forward-Pass Probe: A dummy batch (2, 3, 256, 256) is passed through on the CPU to detect dimension mismatches that syntax checks miss.
   • Database Deduplication: An MD5 checksum ensures no duplicate models are evaluated.
4. Campaign Automation: A fault-tolerant orchestrator (CampaignMoE4.py) manages the generate-evaluate loop over 197 batches. It persists state to JSON files, allowing for SSH-safe resumption after interruptions, and targets 1,000 successful evaluations.

Key Contributions
The paper presents three primary engineering contributions:

• Deterministic Code-Assembly Generator: A tool that systematically combines base families into syntactically valid, forward-pass-verified MoE4 models without requiring LLM calls, reusing proven gating and training configurations.
• Multi-Stage Validation Pipeline: A pre-GPU filtering system that significantly reduces wasted computational resources by catching syntax errors, resolution-dependent dimension mismatches, and duplicates.
• Fault-Tolerant Campaign Automation: A robust orchestration system enabling unattended, long-running (28-day) execution with persistent state management.

Results
The campaign ran for 28 days on an NVIDIA RTX 4090, generating 4,463 candidate models and successfully evaluating 1,021 models.

• Search Space Coverage Bias: A critical post-campaign analysis revealed a severe coverage bias. Due to the use of itertools.combinations on a sorted list of model names, the generator enumerated combinations in lexicographic order. Since "AirNet" is alphabetically first, the generator exhausted all $\binom{28}{3} = 3,276$ combinations containing AirNet before moving to the next family. The campaign stopped at 1,146 combinations, meaning 100% of the explored space was anchored to AirNet, covering only 4.8% of the theoretical search space.
• Performance Metrics: Within the AirNet-anchored scope, the best model (AirNet + AlexNet + DPN68 + ResNet) achieved 68.0% Top-1 accuracy on CIFAR-10 in a single training epoch. The mean accuracy across successful models was 0.5221.
• Family-Level Analysis:
  • High-Yield Families: ShuffleNet (mean acc: 0.632) and MobileNetV3 (mean acc: 0.621) consistently co-produced the highest-accuracy ensembles with success rates >96%.
  • Low-Yield Families: FractalNet had a success rate of only 0.7% due to frequent Out-of-Memory (OOM) errors and shape mismatches. MNASNet consistently degraded ensemble quality (mean acc: 0.218).
• Failure Analysis: 99.7% of failures were due to CUDA OOM errors, primarily caused by the memory demands of combining four full networks at 256×256 resolution.

Significance and Claims
The paper claims that automated assembly of heterogeneous experts can produce competitive candidates without manual tuning, as evidenced by the 68.0% accuracy achieved in a single epoch. However, the authors are modest about the generalizability of the specific accuracy rankings, noting they are conditioned on the presence of AirNet due to the coverage bias.

The primary scientific contribution is the identification of a coverage blind spot inherent in large-scale combinatorial searches that use sorted deterministic enumeration with throughput-based stop conditions. The authors propose a simple fix—stratified random shuffling of the combination list—to eliminate this bias in future runs. They also identify specific architectural families (FractalNet, MNASNet) that should be excluded from the expert pool to improve efficiency and quality in future campaigns. The pipeline, analysis artifacts, and corrected generator are released as open-source.