The Phish, The Spam, and The Valid: Generating Feature-Rich Emails for Benchmarking LLMs

1. Problem Statement

The paper addresses critical gaps in current email security research and dataset quality:

• Evolving Threats: Cybercriminals increasingly use Large Language Models (LLMs) to generate highly convincing, linguistically polished phishing and spam emails. Traditional rule-based detectors and static feature models struggle against these AI-rephrased attacks.
• Dataset Deficiencies: Existing public email datasets (e.g., Enron, SpamAssassin) are often outdated (pre-2010), lack structural granularity (missing full URLs, attachment names), and fail to distinguish clearly between three critical classes: Phishing, Spam, and Valid emails.
• Evaluation Limitations: There is a lack of rigorous benchmarks for state-of-the-art LLMs (like Qwen and Gemini) regarding their ability to detect LLM-generated threats, specifically when structural metadata is included or when emails are linguistically "fuzzed" (rephrased).

2. Methodology: The PhishFuzzer Framework

The authors propose PhishFuzzer, a metadata-enriched generation framework designed to create a rigorous, open-source dataset and benchmarking pipeline.

A. Dataset Creation Pipeline

The process involves four main steps to generate 23,100 emails (3,300 seeds + 19,800 synthetic variants):

1. Seed Dataset Creation:
   • Private Seeds (N=300): Manually curated, anonymized emails from personal/corporate inboxes, balanced across Phishing, Spam, and Valid.
   • Public Seeds (N=3,000): Aggregated from legacy public datasets (e.g., Nazario, SpamAssassin). These initially lacked granular metadata.
2. Intent Benchmarking & Label Augmentation:
   • The authors benchmarked 5 LLMs on a subset of 99 emails to identify attacker intent (e.g., "Follow link," "Open attachment"). Gemini-2.5-Flash was selected for its high accuracy (97.98%) and consistency.
   • This model was used to populate missing structural fields (URLs, attachment filenames) in the public seeds, ensuring logical consistency (e.g., "Follow link" emails must have a URL).
3. Label Enrichment: The validated model populated intent, URL, and attachment fields for all 3,000 public seeds, merging them with the private seeds to create a unified 3,300-email seed dataset.
4. Dataset Expansion (Seeding):
   • Each of the 3,300 seeds served as a template to generate 6 synthetic variants (Total K=7 per template).
   • Dimensions of Variation:
     • Entity Type: Globally recognized entities vs. fabricated but realistic ones.
     • Length: Short (4–8 sentences), Medium (10–16), Long (25–40).
   • Constraints: Phishing variants were forced to use deceptive domains; Spam/Valid variants used realistic domains. This process sanitized HTML artifacts and ensured clean plaintext.

B. Evaluation Metrics

The authors evaluated two state-of-the-art LLMs (Qwen-2.5-72B and Gemini-3.1-Pro) using:

• Prompt Configurations:
  • Basic: Subject + Body.
  • Full: Subject + Body + Sender + URLs + Attachments.
• Reliability Metrics:
  • Task Success Rate (TSR): Number of correctly classified variants per template.
  • Confidence Index (Conf@K): Percentage of templates where the model correctly classified all K variants.
  • Total Flip Score (TFS@K): Counts templates where the model failed all K variants, isolating systematic blind spots rather than stochastic errors.

3. Key Contributions

1. PhishFuzzer Framework: An open-source pipeline for generating structurally consistent, metadata-rich synthetic email variants from real-world seeds.
2. PhishFuzzer Dataset: The first open-source, three-class (Phishing, Spam, Valid) dataset with 23,100 emails, featuring:
   • Strict provenance tracking (Private vs. Public seeds).
   • Full structural metadata (URLs, attachments, sender domains).
   • Attacker intent annotations.
3. Rigorous LLM Benchmarking: A comprehensive evaluation demonstrating that while LLMs perform well on zero-shot phishing detection, they struggle with the subjective boundary between Spam and Valid emails.
4. Systematic Failure Analysis: Introduction of the TFS metric to identify "blind spots" where models consistently fail across linguistic variations, revealing inherent model weaknesses.

4. Key Results

The study evaluated Qwen-2.5-72B and Gemini-3.1-Pro across the 23,100-email dataset.

• Phishing Detection:
  • Both models achieved high F1 scores for phishing (>0.89).
  • Gemini-3.1-Pro was stricter, misclassifying only 13 phishing emails as Valid (Full prompt), achieving an F1 of 0.958.
  • Adding structural metadata significantly improved phishing detection for both models (reducing errors by ~20–30%).
• Spam Detection (The Critical Weakness):
  • Both models struggled significantly to distinguish Spam from Valid emails (F1 scores ≤ 0.43).
  • Counter-intuitive Finding: Adding metadata degraded spam detection. Models became less sensitive to spam, misclassifying more spam as "Valid."
  • Reasoning: The boundary between aggressive marketing (Spam) and legitimate promotions (Valid) is highly subjective; metadata did not help resolve this semantic ambiguity.
• Reliability & Robustness:
  • Gemini-3.1-Pro showed higher resilience to linguistic fuzzing (Confidence Index: 62.88% vs. Qwen's 55.33%), suggesting it relies on underlying logic rather than brittle lexical patterns.
  • Qwen-2.5-72B showed signs of data contamination, performing significantly worse on unseen private data compared to public datasets.
• Systematic Failures (TFS):
  • The TFS analysis revealed that while metadata fixed some "Valid" misclassifications, it created new blind spots where Spam was consistently misclassified as Valid.
  • One specific "blind spot" was identified as a legitimate email from an MIT researcher that was incorrectly labeled as phishing in legacy datasets, highlighting human labeling errors in public benchmarks.

5. Significance and Conclusion

• Security Implications: The study proves that while metadata (URLs, senders) is crucial for hardening defenses against phishing, it is insufficient for solving the spam vs. valid classification problem, which remains a semantic challenge for LLMs.
• Dataset Quality: The paper highlights that legacy datasets contain human labeling errors and lack the structural fidelity required to test modern AI-driven attacks. The PhishFuzzer dataset provides a necessary foundation for next-generation email security systems.
• Future Directions: The authors suggest that future security systems must move beyond pure LLM classification to hybrid architectures combining LLM semantic reasoning with heuristic filters. They also call for addressing human mislabeling in public datasets to improve training data quality.

In summary, PhishFuzzer provides a robust, open-source tool for evaluating email security, revealing that while LLMs are excellent at detecting obvious phishing, they remain vulnerable to the nuanced, subjective distinctions between spam and legitimate communication, a gap that metadata alone cannot bridge.