PhreshPhish: A Real-World, High-Quality, Large-Scale Phishing Website Dataset and Benchmark

1. Problem Statement

Phishing remains a pervasive cyber threat causing billions in economic damage. While machine learning (ML) has improved detection capabilities, progress is hindered by three critical issues in the current research landscape:

• Data Scarcity and Quality: Existing public datasets are often small, outdated, or contain significant noise (e.g., mislabeled data, takedown notices, or benign pages masquerading as phishing due to scraping failures).
• Data Leakage: Many datasets suffer from "leakage," where training and test sets share highly similar or identical data points (e.g., from the same phishing kit), leading to overly optimistic performance metrics.
• Unrealistic Evaluation: Most benchmarks use balanced datasets (50% phishing) or high base rates. In the real world, phishing attacks are rare events (base rates often <1%). Models trained on balanced data fail to maintain high precision (low false positives) when deployed in low-base-rate environments, which is critical for user experience.

2. Methodology

The authors developed PhreshPhish, a comprehensive pipeline for collecting, cleaning, and benchmarking phishing data.

A. Data Collection Pipeline

• Sources: Phishing URLs were aggregated from PhishTank, APWG eCrime eXchange, and NetCraft. Benign URLs were sourced from anonymized browsing telemetry (6M+ users) and Google search results for targeted brands.
• Scraping Strategy: To overcome adversarial tactics (dynamic content, cloaking, ephemerality), the authors used a distributed cluster of real browsers (Selenium/Chrome) rather than static crawlers.
  • Dynamic Content: Full browser rendering captures JavaScript-heavy pages.
  • Cloaking Mitigation: Nodes use varied User-Agents and IP addresses to bypass IP-based content filtering.
  • Ephemerality: URLs are scraped within minutes of reporting to capture pages before takedowns.

B. Cleaning and Quality Assurance

A two-stage cleaning process was implemented to ensure high fidelity:

1. Automated Filtering: Normalized URLs, removed duplicates, and filtered out pages with "bad titles" (e.g., "404 Not Found," "Access Denied," "Cloudflare Anti-Bot") or missing HTML.
2. Human-in-the-Loop Inspection:
   • Grouping: Pages were grouped using Locality Sensitive Hashing (LSH) on TF-IDF features and by page title.
   • Prototype Inspection: Annotators reviewed a single "prototype" page per group. Decisions to remove a group were applied to all nearest neighbors.
   • PII Removal: URLs containing personally identifiable information (PII) were stripped or removed, particularly from benign data.

C. Benchmark Construction

To create realistic evaluation standards, the authors applied three specific operations to the test set:

1. Leakage Minimization: The test set was temporally disjoint from the training set. Furthermore, any test point with a TF-IDF cosine similarity above a threshold to any training point was removed.
2. Diversity Enhancement: The benign set was augmented with legitimate URLs of top targeted brands to ensure coverage. Highly similar phishing/benign pairs were pruned to increase diversity.
3. Difficulty Filtering: "Easy" phishing samples (those classified with >85% confidence by baseline models) were randomly removed to force models to handle harder cases.
4. Base Rate Adjustment: The authors generated 975 benchmark datasets with base rates ranging from 0.05% to 5% (plus a 24% full set) to simulate real-world deployment scenarios.

3. Key Contributions

1. PhreshPhish Dataset: The largest publicly available phishing dataset to date, containing ~617k samples (250k phishing, 367k benign) collected over 17 months. It demonstrates a significantly lower rejection rate (<1%) during cleaning compared to existing datasets (which often had >40% rejection rates).
2. Realistic Benchmark Suite: A suite of 975 datasets designed to eliminate data leakage and simulate real-world low base rates (0.05%–5%), addressing the "unrealistic performance" gap in current literature.
3. Baseline Evaluation: Comprehensive evaluation of four distinct detection approaches on these benchmarks, establishing new baselines for the field.
4. Open Release: The dataset, benchmarks, and code are publicly available on Hugging Face and GitHub.

4. Results and Analysis

The authors evaluated four models:

• Linear Model: Trained on n-grams and hand-crafted features.
• Feedforward Neural Network (FFN): Shallow network with binary features.
• GTE (BERT-based): Fine-tuned embedding model on raw HTML/URL.
• LLM (Gemini-2.5-Pro): Zero-shot prediction using prompts.

Key Findings:

• Base Rate Sensitivity: Model performance (specifically Precision) degrades sharply as the base rate decreases. Models achieving >95% Average Precision (AP) on balanced datasets dropped significantly in low-base-rate scenarios.
• Model Performance:
  • The GTE (BERT-based) model outperformed all others, achieving an AP of 0.605 at a 0.05% base rate, significantly higher than the Linear (0.22) and LLM (0.10) models.
  • LLMs performed poorly in zero-shot settings compared to fine-tuned embedding models, likely due to the lack of specific training on phishing patterns and the difficulty of reasoning over raw HTML without fine-tuning.
• Precision vs. Recall Trade-off: In low base-rate environments, maintaining high recall requires accepting a massive drop in precision (many false positives), highlighting the difficulty of real-world deployment.

5. Significance

• Standardization: PhreshPhish provides the first standardized, high-quality benchmark that accounts for data leakage and realistic base rates, allowing for fair comparison of future detection methods.
• Realism: By demonstrating that performance metrics are heavily dependent on base rate, the paper challenges the validity of previous studies that used balanced datasets, urging the community to adopt more rigorous evaluation protocols.
• Research Catalyst: The release of a large-scale, clean dataset enables the training of more robust models and facilitates research into adversarial robustness, dynamic content analysis, and low-resource detection scenarios.
• Future Maintenance: The authors commit to periodic versioned updates to keep the dataset current against evolving phishing tactics.

In conclusion, PhreshPhish addresses the "garbage in, garbage out" problem in phishing detection research by providing a clean, large-scale dataset and a rigorous benchmarking framework that reflects the harsh realities of real-world deployment.