Technical Summary: A Multi-Task Mixture of Experts Framework for Malware Classification, Packing Detection, and Family Attribution

Problem Statement

Malware analysis faces significant challenges due to the inherent heterogeneity of malware data, the prevalence of packed and obfuscated binaries, and the diverse distribution of malware families. Traditional single-model detection mechanisms often struggle to generalize across these varied data distributions, leading to degraded performance, particularly on rare or heavily obfuscated samples. Furthermore, existing frameworks typically rely on monolithic architectures that attempt to optimize multiple heterogeneous objectives (e.g., family classification, packing detection, and benign/malware discrimination) simultaneously. This approach often results in "negative task transfer," where the conflicting structural and semantic requirements of different tasks interfere with one another, preventing the model from learning effective representations for all objectives.

Methodology

The authors propose a unified multi-task learning framework based on Mixture of Experts (MoE) architectures to address these limitations. The core hypothesis is that decomposing the problem into specialized expert networks, coordinated by adaptive gating mechanisms, allows for effective task-specific learning while maintaining scalability.

1. Input Representations

The framework evaluates performance across two distinct input modalities:

• Structured EMBER Features: High-dimensional (2,381-dimensional) numerical descriptors extracted from Portable Executable (PE) files using the EMBER framework. These include parsed PE metadata, header information, section characteristics, import/export tables, and format-agnostic byte histograms and entropy measures.
• Raw 1D Image Representations: Direct conversion of raw executable bytes into one-dimensional grayscale intensity sequences (tested at lengths of 1,024, 4,096, and 16,384 bytes), allowing the model to learn from low-level binary data without explicit feature engineering.

2. Architectural Variants

The study systematically investigates three MoE paradigms:

• Homogeneous MoE (Homo-MoE): Utilizes three identical expert networks and a single shared gating network. All experts share the same architecture (Dense layers) but learn different feature patterns. It employs an Expected Squared Error loss and a reconstruction regularizer to ensure latent space meaningfulness.
• Heterogeneous MoE (Hetero-MoE): Combines structurally diverse experts (e.g., one with skip connections, one with shrinking dimensions, one with bottleneck layers) to capture complementary information. It utilizes Negative Log-Likelihood (NLL) loss to promote competitive learning among the diverse structures.
• Multi-Gate Mixture of Experts (MMoE): Designed specifically for multi-task scenarios, this architecture employs independent task-specific gating networks. Instead of a single gate routing inputs for all tasks, MMoE allows each task (Family Classification, Packing Detection, Benign/Malware Identification) to independently select and weight expert outputs. This decouples the routing process to mitigate negative transfer between tasks.

3. Training and Evaluation Strategy

• Multi-Task Objectives: The framework simultaneously performs three tasks:
  1. Malware family classification (multi-class).
  2. Packed vs. unpacked detection (binary).
  3. Malware vs. benign identification (binary).
• Adversarial Robustness: To evaluate resilience, the authors use PyMetaEngine to generate mutated malware samples, creating adversarial datasets that introduce distribution shifts while preserving program semantics.
• Metrics: Performance is measured using Accuracy, F1-score, and a Combined Detection Rate (CDR), which requires all three tasks to be correct for a sample to be counted as a success. The Attack Success Rate (ASR) is defined as $1 - CDR$.

Key Contributions

1. Unified Multi-Task Formulation: The paper formulates malware analysis as a single framework integrating family classification, packing detection, and malware/benign separation, moving away from isolated single-task models.
2. Architectural Investigation: It designs and evaluates Homogeneous, Heterogeneous, and Multi-Gate MoE architectures, highlighting the impact of expert diversity and gating strategies on malware analysis.
3. Feature Representation Comparison: A comprehensive comparative analysis is conducted between structured EMBER features and raw 1D image representations within the MoE context.
4. Robustness Evaluation: The models are tested under distribution shifts induced by mutation-based metamorphic adversarial attacks to assess real-world resilience.
5. Empirical Findings: The study demonstrates that Multi-Gate MoE (MMoE) achieves superior performance and stability, particularly when using structured EMBER features and mutation-augmented training data.

Experimental Results

The experiments were conducted on a dataset of 66,400 PE files (26,400 malware, 40,000 benign) across multiple random seeds.

• Standard Performance (EMBER Features): The MMoE model achieved the highest baseline Combined Detection Rate (CDR) of 0.9670, outperforming Homogeneous MoE (0.9466) and Heterogeneous MoE (0.9425). MMoE also showed the lowest Attack Success Rate (ASR) of 3.35%.
• Adversarial Robustness (Mutated Samples): When tested against mutated samples (Dataset D2), MMoE maintained a CDR of 0.9342, whereas Homogeneous and Heterogeneous MoE models suffered significant drops to 0.8635 and 0.8577, respectively. This indicates MMoE's superior ability to handle distribution shifts.
• Impact of Data Augmentation: Training MMoE on a dataset augmented with mutated samples (Dataset D3) further improved performance, reaching a CDR of 0.9744 with an ASR of only 2.56%.
• Raw 1D Image Results: When using raw 1D byte representations, the Heterogeneous MoE outperformed MMoE (CDR 0.9585 vs. 0.9478). This suggests that for raw data, architectural diversity in experts may be more beneficial than task-specific gating, whereas for structured features, task-specific gating is critical.
• Efficiency: The MMoE model demonstrated high computational efficiency, with a test time of approximately 2.05 µs per sample.

Significance and Claims

The paper claims that the proposed framework offers a promising direction for scalable and resilient malware detection systems. The primary significance lies in the demonstration that task-specific expert routing (via MMoE) significantly improves accuracy and generalization compared to unified gating strategies, particularly in complex, heterogeneous malware distributions.

The authors modestly note that the advantage of task-specific gating is not independent of the input representation; it is most effective when paired with the semantic richness of structured EMBER features. While raw 1D representations benefit more from heterogeneous expert structures, the MMoE framework with EMBER features represents the most robust configuration for handling the diverse challenges of modern malware, including packing and metamorphic mutations. The work highlights that expert specialization and conditional computation are effective mechanisms for reducing interference between concurrent security objectives.