Multimodal Multi-Agent Ransomware Analysis Using AutoGen

This paper proposes a multimodal multi-agent ransomware analysis framework leveraging AutoGen, which integrates static, dynamic, and network data through specialized agents and a transformer-based classifier with an iterative feedback mechanism to achieve superior family classification accuracy and reliable real-world deployment compared to traditional single-modality approaches.

The Gist

🛡️ The Problem: The Digital Burglar

Imagine your computer is a house. Ransomware is a sophisticated burglar who doesn't just steal your TV; they lock all your doors, hide your keys, and demand a ransom to let you back in.

For a long time, security guards (traditional antivirus) tried to catch these burglars by looking at their faces (static analysis) or watching them run around (dynamic analysis). But these burglars are tricky. They wear masks, change their clothes, and sometimes hide in plain sight. If you only look at their face, you might miss them. If you only watch them run, you might think they are just exercising.

🧠 The Solution: A Detective Team (The MMMA-RA Framework)

The authors of this paper propose a new way to catch these digital burglars. Instead of relying on one security guard, they built a team of specialized detectives who work together. They call this system MMMA-RA.

Think of it like a high-tech police precinct where three different types of experts investigate a crime scene simultaneously:

1. The Forensic Expert (Static Analysis): This agent looks at the "blueprints" of the file without opening it. They check the file's structure, its code, and its hidden metadata. Analogy: Checking the fingerprint on a door handle.
2. The Surveillance Officer (Dynamic Analysis): This agent watches the file while it runs in a safe, isolated room (a sandbox). They see what the file tries to do—does it try to encrypt files? Does it delete backups? Analogy: Watching a suspect try to pick a lock.
3. The Network Analyst (Network Traffic): This agent listens to the file's phone calls. Does it try to contact a strange server? Does it send a lot of data out? Analogy: Listening to the burglar's radio chatter to the getaway driver.

🤝 The Magic: The "AutoGen" Team Meeting

Here is where the paper gets really cool. Usually, these three experts would just write their own reports and hand them to a boss who makes a final decision.

But in this system, the three experts are AI Agents powered by a large language model (like a smart chatbot). They don't just report; they talk to each other in a loop.

• The Analyst: Gathers the evidence.
• The Critic: Acts as the "Devil's Advocate." They look at the Analyst's report and say, "Wait, this looks suspicious, but are you sure? Maybe we missed something. Let's look closer at the 'Dharma' family of ransomware."
• The Assistant: Helps organize the conversation and suggests the next steps.

The "Feedback Loop" Metaphor:
Imagine a group of chefs cooking a complex dish.

• The Analyst tastes the soup.
• The Critic says, "It's too salty, but the spice is perfect. Let's add more water and taste again."
• The Assistant adjusts the heat.
  They keep tasting and adjusting without changing the recipe itself. They just change how they cook it based on what they learn. This allows the system to get smarter over time without needing to be retrained from scratch.

🧩 Putting It All Together (Fusion)

Once the three agents have analyzed the file from their different angles, they combine their findings into one giant "super-report."

• If the Forensic Expert sees a weird code structure, but the Surveillance Officer sees no malicious behavior, the team might say, "Probably safe."
• But if the Forensic Expert sees a weird code structure AND the Surveillance Officer sees it trying to lock files AND the Network Analyst sees it calling a bad server, the team screams, "BUSTED!"

This combination is called Multimodal Fusion. It's like having a 3D view of the problem instead of a flat picture.

🎯 The Results: Smarter and Safer

The researchers tested this system on thousands of ransomware samples. Here is what happened:

1. Better Accuracy: The team of agents caught more criminals than any single detective could. They achieved a 94.6% success rate in identifying exactly which "family" of ransomware it was.
2. Knowing When to Say "I Don't Know": This is a huge deal. Sometimes, a file is so tricky that even the experts aren't 100% sure. Instead of guessing and making a mistake (a "false positive"), the system has the wisdom to abstain. It says, "I'm not confident enough to classify this. Let's flag it for a human to check." This prevents panic and false alarms.
3. Self-Improvement: Over 100 rounds of training (epochs), the agents got better at talking to each other. Their "quality score" went up steadily, proving that the conversation between the AI agents actually made the system smarter.

🚀 Why This Matters

In the real world, ransomware is evolving faster than humans can write new rules.

• Old way: Humans write a rule: "If it does X, it's bad." The bad guys change X, and the rule fails.
• New way (This Paper): The AI agents learn to recognize the pattern of bad behavior across different angles. Even if the bad guy changes their tactics, the team of agents can spot the inconsistency in their story.

🏁 The Bottom Line

This paper introduces a team of AI detectives that work together to catch ransomware. By looking at the file from three different angles (code, behavior, and network) and having the AI agents debate and refine their conclusions, the system becomes incredibly accurate.

It's like upgrading from a single security camera to a smart, talking security team that never sleeps, never gets tired, and knows exactly when to call for backup.

Technical Summary

1. Problem Statement

Ransomware has evolved into a sophisticated cybersecurity threat causing massive financial losses. Traditional detection methods (signature-based, static analysis, dynamic analysis, and network monitoring) often fail when used in isolation due to:

• Polymorphism and Obfuscation: Attackers use packers, code obfuscation, and delayed execution to evade static and dynamic detection.
• Modality Limitations: A ransomware family may appear benign in one modality (e.g., static features) while showing malicious intent in another (e.g., network traffic).
• Class Imbalance: Real-world datasets often suffer from severe imbalance between benign samples and specific ransomware families.
• Zero-Day Detection: Existing models struggle to generalize to unseen variants or families, particularly those with high behavioral variability (e.g., Dharma, WannaCry).
• Overconfidence: Standard classifiers often produce overconfident predictions, lacking reliable uncertainty estimation for real-world deployment.

2. Methodology: MMMA-RA Framework

The authors propose MMMA-RA (Multimodal Multi-Agent Ransomware Analysis), a framework that integrates static, dynamic, and network modalities using a multi-agent architecture powered by AutoGen and Large Language Models (LLMs).

A. Data Modalities

The system ingests three heterogeneous data types:

1. Static: PE headers, opcodes, entropy, imports/exports.
2. Dynamic: API call traces, registry/file system activity, process behavior (captured via CAPE v2 sandbox).
3. Network: Flow duration, packet statistics, protocol usage, and timing (captured via CICFlowMeter).

B. Core Architecture Components

1. Modality-Specific Deep Contrastive Autoencoders (DCAEs):

   • Each modality is processed by a dedicated DCAE (4-layer encoder/decoder).
   • Loss Function: Combines reconstruction loss ($L_{AE}$) with Supervised Contrastive Loss ($L_{sup}$). This forces embeddings of the same ransomware family to cluster together while pushing different families apart in the latent space.
   • Output: Low-dimensional, discriminative latent vectors ($z_{static}, z_{dynamic}, z_{network}$).

2. Gated Cross-Modality Fusion:

   • Latent vectors are aligned and concatenated to form a fused representation ($z_{fused}$).
   • A gated mechanism selectively integrates features, mitigating noise or redundant information if a specific modality is unavailable or unreliable (graceful degradation).

3. Family Classifier:

   • A Transformer-based Multi-Layer Perceptron (MLP) classifies the fused vector into specific ransomware families.
   • Class Imbalance Handling: Uses inverse frequency class weighting to prioritize minority families.
   • Calibration: Applies Post-hoc Probability Calibration (Vector Scaling) to ensure confidence scores reflect actual prediction reliability.

4. AutoGen Multi-Agent Feedback Loop:

   • The system employs three lightweight agents (using a local Phi-3.2B LLM) that interact without fine-tuning the LLM weights:
     • User Proxy: Aggregates model statistics and generates a structured triage message.
     • Critic Agent: Evaluates the Analyst's reasoning, identifies "weak families" (low F1 scores), and flags missing elements. It outputs structured control signals (e.g., target families for oversampling).
     • Assistant Agent: Generates forward-looking risk assessments and forecasts.
   • Mechanism: The agents operate in a feedback loop over 100 epochs. The Critic's analysis drives adaptive sampling (oversampling weak families) and threshold adjustments for the classifier, effectively steering the training process via text-to-data feedback.

3. Key Contributions

1. Unified Multi-Agent Framework: A novel architecture combining static, dynamic, and network modalities with an autonomous agent loop for iterative refinement.
2. Contrastive Representation Learning: Introduction of supervised contrastive learning within autoencoders to create modality-specific latent spaces that maximize inter-family separation.
3. Class Imbalance Awareness: Integration of class-weighted optimization and inversion frequency strategies to handle severe dataset skew.
4. Agentic Feedback without Fine-Tuning: Demonstrates that an LLM-based critic can improve model performance (sampling, calibration) by analyzing metrics and adjusting hyperparameters, without requiring LLM fine-tuning.
5. Confidence-Aware Abstention: The system prioritizes reliability over forced classification, using calibrated confidence to abstain from predicting on uncertain zero-day samples.

4. Experimental Results

The framework was evaluated on a custom dataset containing 3,000 samples (6 classes: Benign, Ryuk, LockBit, Dharma, Shade, WannaCry) across 100 epochs.

• Performance Metrics:

  • Macro-F1 Score: The Multimodal Multi-Agent system achieved 0.946, significantly outperforming Single Agent (0.919) and single-modality baselines (Static: 0.72, Dynamic: 0.45, Network: 0.13).
  • Calibration: Achieved the lowest Expected Calibration Error (ECE) of 0.017, indicating highly reliable confidence estimates.
  • Agent Convergence: Over 100 epochs, the "Composite Quality Score" of the agents improved monotonically by +0.75, reaching ~0.88 without LLM fine-tuning.

• Zero-Day Generalization:

  • LockBit: Achieved 0.99 Macro-F1 and >95% accuracy, demonstrating strong transferability for behaviorally similar families.
  • Ryuk: Moderate performance (0.66 F1) with high coverage, showing the system's ability to detect evolving variants.
  • Dharma & WannaCry: The system exhibited high abstention rates (>95%) and low F1 scores. This is a deliberate feature: the model recognized the high distribution shift/polymorphism and refused to make uncertain predictions, prioritizing safety over false positives.

• Statistical Significance:

  • Friedman tests confirmed significant differences in performance across modalities.
  • While pairwise Wilcoxon tests showed p-values slightly above 0.05 for some comparisons, the effect sizes (r ~ 0.905) were large, confirming the practical significance of the Multi-Agent approach.

5. Significance and Conclusion

The paper establishes that collaborative multi-agent systems can significantly enhance ransomware detection by:

• Synergizing Modalities: Combining static, dynamic, and network data overcomes the limitations of single-view analysis.
• Self-Improvement: The system demonstrates "learning" capabilities where agents iteratively refine the model's focus on weak classes through natural language feedback loops.
• Operational Reliability: By incorporating confidence-aware abstention, the framework is suitable for real-world deployment where false positives are costly.
• Scalability: The approach avoids the computational cost of fine-tuning large LLMs, instead using them as meta-controllers to optimize existing deep learning pipelines.

The findings suggest that Agentic AI offers a practical path toward robust, adaptive, and trustworthy cybersecurity defense systems capable of handling the evolving landscape of ransomware threats.