Technical Summary: MultiPhishGuard

Problem Statement

Phishing remains a primary vector for data breaches and financial loss, with attack strategies evolving from simple deceptive messages to sophisticated social engineering, spear-phishing, and AI-driven content generation. Traditional detection methods, including rule-based filters, denylists, and static machine learning models, struggle to keep pace with these dynamic tactics. While deep learning approaches (e.g., BERT) have improved accuracy by capturing contextual cues, they often suffer from "black-box" limitations, lack interpretability, and typically focus on single modalities (e.g., text only), overlooking critical signals in URLs and metadata. Furthermore, existing Large Language Model (LLM) approaches are predominantly single-agent architectures that lack adaptability to novel threats and fail to provide transparent, evidence-based reasoning for their decisions.

Methodology

The authors propose MultiPhishGuard, an LLM-based multi-agent framework designed to address these limitations through specialized analysis, adaptive coordination, and adversarial training. The system architecture comprises five cooperative agents:

1. Specialized Analysis Agents:

   • Text Agent: Analyzes the email body for suspicious patterns, keywords, and linguistic manipulation.
   • URL Agent: Scrutinizes embedded links for obfuscation, domain reputation, and redirections.
   • Metadata Agent: Examines headers, sender authentication records (SPF, DKIM, DMARC), and structural anomalies.
   • Mechanism: Each agent operates independently using specific prompts to generate a verdict, confidence score, and technical rationale.

2. Dynamic Weight Adjustment (Reinforcement Learning):

   • Instead of fixed ensemble weights, the system employs Proximal Policy Optimization (PPO) to dynamically weight the contributions of the three analysis agents based on the specific characteristics of each email.
   • The RL module receives email-specific features (e.g., number of URLs, authentication results) and learns a policy to maximize classification accuracy, allowing the system to prioritize the most informative modality (text, URL, or metadata) for a given context.

3. Explanation Simplifier Agent:

   • This agent aggregates the technical rationales from the specialized agents and synthesizes them into a single, coherent, plain-language explanation.
   • It is designed to reduce technical jargon, ensuring the output is accessible to non-expert users while remaining grounded in factual evidence.

4. Adversarial Training Module:

   • An Adversarial Agent (based on GPT-4o) generates subtle, context-aware variants of both phishing and legitimate emails.
   • These variants mimic evasion techniques (e.g., synonym substitution, sentence rewriting, homoglyph replacement) to expose model weaknesses.
   • The detection model acts as a discriminator, and misclassifications provide feedback to refine the adversarial agent's strategy, creating a loop that enhances robustness against ambiguous and evolving threats.

Key Contributions

The paper outlines four primary contributions:

• Novel Multi-Agent Architecture: MultiPhishGuard is presented as the first system to utilize multiple LLM agents to decompose phishing analysis across message bodies, URLs, and metadata, capturing heterogeneous cues that single-modality detectors miss.
• Adversarial Training Integration: The framework incorporates an adversarial agent to generate realistic email variants, specifically intended to improve robustness to subtle and ambiguous cases.
• Explainability via Simplification: The introduction of an explanation simplifier agent that consolidates multi-agent outputs into user-friendly rationales, addressing the transparency gap in current LLM-based detectors.
• Comprehensive Evaluation: The model was evaluated on six public datasets, demonstrating robust performance across diverse phishing scenarios.

Experimental Results

The system was evaluated on a pooled dataset comprising approximately 4,000 emails (979 phishing, 3,000 legitimate) from six sources: Nazario, Enron-Spam, TREC 2007, CEAS 2008, Nigerian Fraud, and SpamAssassin.

• Performance Metrics: MultiPhishGuard achieved an accuracy of 97.89%, an F1-score of 95.88%, a recall of 99.80%, and a precision of 92.26%. Crucially, it maintained a low False Positive Rate (FPR) of 2.73% and a False Negative Rate (FNR) of 0.20%.
• Comparative Analysis: The system outperformed established baselines, including Chain-of-Thought (CoT) prompting, a single-agent LLM model, and a RoBERTa-base classifier.
  • Compared to CoT, MultiPhishGuard significantly reduced the FPR (2.73% vs. 17.47%) while maintaining high recall.
  • Compared to RoBERTa-base, it achieved higher accuracy (97.89% vs. 95.73%) and lower FPR.
• Explanation Quality: Automated metrics (Perplexity: 25, Topic Coherence: 0.35, Flesch Reading Ease: 41) and human evaluation by a cybersecurity expert indicated that the generated explanations were more fluent, coherent, and aligned with expert reasoning than those produced by CoT or ablated variants.
• Ablation Studies: Removing individual components (URL agent, Metadata agent, PPO weighting, Adversarial agent, or Explanation agent) resulted in measurable performance declines, confirming the necessity of each module. Specifically, the absence of the URL agent led to a significant drop in precision, and the removal of adversarial training reduced robustness to evasive attacks.

Significance and Claims

The authors claim that MultiPhishGuard represents a promising direction for phishing detection by demonstrating that multi-agent LLM architectures with adaptive coordination and adversarial training can effectively handle the heterogeneity of phishing cues.

The paper emphasizes that the system's significance lies in its ability to:

1. Integrate Multi-Modal Analysis: By jointly modeling text, URLs, and metadata, the system captures malicious cues that single-modality detectors often overlook.
2. Adapt Dynamically: The PPO-based reinforcement learning allows the system to adjust its decision-making strategy in real-time based on the specific characteristics of an email, rather than relying on static heuristics.
3. Enhance Robustness: The adversarial training loop specifically targets the model's ability to generalize against subtle, evolving, and ambiguous attack patterns.
4. Provide Interpretability: The explanation simplifier bridges the gap between complex model outputs and human understanding, supporting trust and decision-making in security operations.

The authors conclude that while limitations exist regarding the evaluation of ground-truth explanations and generalization across different model families, the framework offers a scalable, interpretable, and adaptive foundation for future cybersecurity tools.