Technical Summary: MAGMA – A Stochastic Consensus Multi-Agent RAG Framework for Robust Malware Detection

1. Problem Statement

Contemporary deep learning (DL) malware detectors, while dominant, suffer from a critical vulnerability: semantic blindness. By treating binary code as unstructured statistical data (e.g., raw byte sequences), these models act as stochastic pattern matchers lacking true semantic reasoning. This limitation manifests as "Confident Wrongness," where a model outputs a high-confidence verdict (e.g., >99% benign) on a malicious sample simply because an adversarial transformation (e.g., instruction substitution, dead code insertion) shifts the input into a sparse region of the embedding space, crossing the decision boundary without altering the underlying malicious functionality.

Current state-of-the-art systems lack a formal mechanism to quantify epistemic uncertainty (model ignorance). Consequently, they force binary decisions even when inputs lie outside the training distribution, failing to flag anomalous cases for human review and instead committing catastrophic automated errors.

2. Methodology: The MAGMA Framework

The authors propose MAGMA (Multi-Agent Generative Malware Analysis), a framework that reformulates detection from static pattern matching to a probabilistic evidence-gathering process. The architecture decouples structural representation from probabilistic reasoning through four core components:

A. Dual-Stream Lifting

To overcome semantic opacity, MAGMA lifts raw binaries into two parallel, human-readable intermediate representations before vectorization:

1. Syntactic Stream ($I_{ASM}$): A linear disassembly mapping producing normalized assembly instructions. This captures granular architectural dependencies, register manipulations, and cryptographic constants often lost in high-level decompilation.
2. Semantic Stream ($I_{Code}$): A decompilation into high-level pseudo-C code. This abstracts compiler-specific details to isolate the Control Flow Graph (CFG), data dependencies, and core API sequences.
   These dual embeddings are concatenated to form a composite query vector, ensuring that syntactic obfuscation (e.g., register renaming) does not evade detection if the semantic intent remains malicious, and vice versa.

B. Retrieval-Augmented Generation (RAG)

Instead of relying on the frozen parametric memory of a Large Language Model (LLM), MAGMA queries a Vector Knowledge Base (KB) populated with verified, analyst-curated Decision-Critical Functions (DCFs).

• The system retrieves the $k$-nearest neighbors (DCFs) based on cosine similarity in the latent space.
• This step grounds the generative process in known threat intelligence, preventing hallucinations and converting the classification task into an evidence-evaluation task.

C. Stochastic Consistency Ensemble

To probe the stability of the reasoning process, MAGMA employs a Homogeneous Agent Ensemble consisting of $N$ identical LLM instances.

• Stochastic Sampling: Unlike greedy decoding, agents operate with a temperature $T > 0$, forcing the model to explore alternative, lower-probability reasoning paths.
• Independent Reasoning: Each agent independently evaluates the retrieved DCFs and dual-stream context to produce a binary verdict (Malicious/Benign).
• Consensus: The system aggregates these independent outputs to measure the stability of the prediction.

D. Quantifiable Uncertainty Metrics

The framework derives two complementary metrics to quantify uncertainty and guide decision-making:

1. Function Evidence Strength (FES): A weighted aggregation of retrieval confidence. It calculates the mean probability mass assigned to the "Malicious" class across all agents, scaled by the similarity weight of the retrieved context. This prevents "hallucinated consensus" where agents agree on a verdict without strong evidentiary backing.
2. Evidence Conflict Score (ECS): Defined as the Shannon entropy of the ensemble's predictive distribution.
   • Low ECS ($\approx 0$) indicates high consensus (stable prediction).
   • High ECS ($\approx 1$) indicates significant disagreement among agents, signaling structural ambiguity, adversarial perturbation, or Out-of-Distribution (OOD) inputs.

E. Tri-State Decision Policy

MAGMA replaces binary classification with a tri-state policy:

• Malicious: High FES AND Low ECS.
• Benign: Low FES AND Low ECS.
• Uncertain (Reject): High ECS (exceeding threshold $\tau_{stable}$).
  Inputs marked "Uncertain" are automatically rejected and routed for human analyst review, preventing the system from rendering confident but erroneous verdicts on ambiguous samples.

3. Key Contributions

1. Dual-Stream RAG Architecture: A novel mechanism mapping syntactic assembly and semantic pseudo-code to a joint vector space, grounding inference in a curated KB of Decision-Critical Functions to mitigate neural network opacity.
2. Quantifiable Attribution (FES): Introduction of the Function Evidence Strength metric, which weights the influence of retrieved contexts, transforming black-box predictions into auditable evidence chains.
3. Stochastic Self-Consistency: Modeling malware detection as a probabilistic consensus game where a multi-agent ensemble (with $T>0$) evaluates evidence against adversarial criteria to produce a bounded predictive distribution.
4. Formalized Rejection via Entropy: Defining the Evidence Conflict Score (ECS) as a proxy for sensitivity to adversarial perturbations, enabling a principled "reject-option" policy that halts classification when epistemic uncertainty is too high.
5. Extensive Evaluation: Comprehensive testing against highly polymorphic variants and established baselines (MalConv, ResNeXt+), demonstrating the efficacy of the entropy-guided rejection policy.

4. Experimental Results

The framework was evaluated on a dataset of 40,814 binaries (Wild-Augmented Benchmark), split temporally to prevent look-ahead bias.

• Detection Performance: MAGMA achieved a 98.4% F1 score, surpassing the strongest external baseline (ResNeXt+) by 1.3 percentage points and MalConv by 4.2 percentage points.
• Error Reduction: Enforcing the entropy-guided rejection policy resulted in a 4.2% absolute improvement in detection accuracy against evasion attempts and a 4.0% reduction in the False Positive Rate (FPR) compared to non-rejecting baselines.
• Ablation Studies:
  • The dual-stream embedding alone achieved 95.8% F1, outperforming MalConv.
  • Adding the LLM ensemble ($N=5$) provided a further +2.6 pp increase, confirming that ensemble diversity resolves borderline cases that retrieval alone cannot.
  • Zero-shot LLM inference (without RAG) performed poorly (79.6% F1), highlighting the necessity of retrieval-augmented grounding.
• Uncertainty Calibration: Misclassified samples exhibited significantly higher ECS values ($\sim 0.84$) compared to correct predictions ($\sim 0.13$), validating ECS as an effective signal for human-in-the-loop rejection.
• Latency: The full pipeline averages ~42 seconds per binary (dominated by LLM inference), positioning MAGMA as a Tier-2 Cloud Verification Layer rather than a real-time endpoint scanner.

5. Significance and Claims

The paper claims that MAGMA addresses the fundamental asymmetry in malware defense, where defenders must bound all malicious variants while attackers need only one successful evasion path. By shifting from deterministic pattern matching to probabilistic verification, MAGMA explicitly quantifies the model's own blind spots.

The significance lies in the system's ability to:

• Identify Structural Ambiguity: Use entropy maximization to detect when an input is structurally perturbed near the decision boundary, even if the malicious functionality is preserved.
• Prevent Confident Wrongness: Automatically reject ambiguous inputs rather than forcing a binary decision, thereby reducing the risk of catastrophic automated errors.
• Bridge the Semantic Gap: Utilize RAG to connect the discriminative power of embeddings with the reasoning capabilities of LLMs, grounding the analysis in verified threat intelligence rather than hallucinated zero-shot inference.

The authors position MAGMA not as a replacement for high-throughput endpoint AV engines, but as a robust Tier-2 verification layer for "gray zone" samples that evade signature filters but lack definitive behavioral indicators for automated sandboxing.