AI Powered Image Analysis for Phishing Detection

1. Problem Statement

Traditional phishing detection systems rely heavily on text-based (URL analysis, email content) and domain-based features. However, modern attackers increasingly employ visual deception, mimicking legitimate websites by copying logos, color schemes, and layout structures. These visual attacks often evade text and URL filters because the malicious content is embedded within images or obfuscated scripts.

The paper identifies three critical gaps in existing image-based detection research:

• Over-reliance on Accuracy: Most studies report high accuracy, which is misleading in imbalanced datasets (where legitimate sites vastly outnumber phishing sites).
• Lack of Threshold Analysis: Few studies analyze how models perform across different decision thresholds, making it difficult to determine the optimal operating point for real-world deployment (balancing false positives vs. false negatives).
• Dataset Limitations: Many models struggle to generalize to zero-day attacks or newly emerging brands due to a lack of diverse, up-to-date training data.

2. Methodology

The authors developed an end-to-end deep learning framework for image-based phishing detection using webpage screenshots. The methodology consists of five stages:

• Stage 1: Data Collection & Construction
  • Sources: A custom Python pipeline using Playwright captured 18,432 screenshots from live OpenPhish feeds. Additionally, 10,444 images were sourced from the PhishIRIS dataset.
  • Total Dataset: 28,876 labeled screenshots (balanced between phishing and legitimate).
• Stage 2: Preprocessing & Augmentation
  • Resizing: All images were resized to 224x224 pixels to match ImageNet input requirements.
  • Cleaning: Duplicate and corrupted images were removed using hash algorithms.
  • Augmentation: Techniques included random horizontal flips, brightness adjustments (±15%), Gaussian blur, and random cropping to improve model robustness.
  • Split: Data was divided into 80% training, 10% validation, and 10% testing.
• Stage 3: Model Selection
  Two state-of-the-art vision architectures were selected for comparative analysis:
  1. ConvNeXt-Tiny: A modern Convolutional Neural Network (CNN) optimized for local feature extraction (logos, layout details) and computational efficiency.
  2. Vision Transformer (ViT-Base): A transformer-based model utilizing self-attention mechanisms to capture global contextual dependencies and structural relationships.
  • Initialization: Both models utilized Transfer Learning with pre-trained ImageNet weights.
• Stage 4: Training
  • Framework: Supervised binary classification.
  • Loss Function: Binary Cross-Entropy.
  • Optimizer: Adam.
  • Process: Models were trained with hyperparameter tuning (learning rate, batch size, dropout) and monitored for overfitting.
• Stage 5: Evaluation & Threshold Optimization
  • Metrics: Instead of relying solely on accuracy, the study prioritized Precision, Recall, and F1-score.
  • Threshold Tuning: The models were evaluated across decision thresholds ranging from 0.1 to 0.9 to identify the optimal operating point that balances detection capability with false-alarm control.

3. Key Contributions

1. Threshold-Aware Evaluation Framework: The paper introduces a methodology that moves beyond static accuracy reporting. It systematically analyzes performance across varying decision thresholds to identify the most practical operating points for real-world deployment.
2. Architectural Comparison: A rigorous side-by-side comparison of ConvNeXt-Tiny (CNN) and ViT-Base (Transformer) under identical experimental conditions, highlighting the trade-offs between local feature extraction and global context modeling.
3. Curated Dataset Release: The authors committed to releasing their curated dataset of 28,876 screenshots to the research community to support reproducibility and future benchmarking.
4. Transfer Learning Implementation: Demonstrated the effectiveness of using ImageNet pre-trained weights to handle limited phishing-specific data, improving training stability and performance.

4. Key Results

The study evaluated both models at an optimized decision threshold of 0.8.

• ConvNeXt-Tiny Performance: Achieved the highest F1-score (0.992) and demonstrated superior stability across different thresholds. Its convolutional architecture proved highly effective at detecting local visual cues like logos and layout inconsistencies.
• ViT-Base Performance: While capable of capturing global structure, it showed lower recall (0.880) and was more sensitive to threshold changes. It required more computational resources and did not outperform the CNN in this specific visual detection task.
• Threshold Analysis: The study found that increasing the threshold improved precision but reduced recall. The optimal balance for ConvNeXt-Tiny was found at 0.8, whereas ViT-Base struggled to maintain a high F1-score at similar thresholds.

5. Significance and Conclusion

This paper addresses a critical gap in cybersecurity research by shifting the focus from theoretical accuracy to deployment-ready performance.

• Practical Applicability: By optimizing for F1-score and analyzing threshold sensitivity, the study provides a realistic assessment of how these models would behave in production environments where false positives (blocking legitimate sites) and false negatives (missing phishing sites) have distinct costs.
• Architectural Insight: The results suggest that for visual phishing detection—where local features like logos and specific layout elements are paramount—modern CNNs (ConvNeXt) currently offer a better balance of accuracy, efficiency, and robustness compared to Vision Transformers, which may require larger datasets to leverage their global context capabilities.
• Future Directions: The authors propose future work involving hybrid models (merging CNNs and Transformers) and the creation of dynamic phishing simulation datasets to address zero-day threats. The release of the dataset is a significant step toward standardizing evaluation in this domain.

In summary, the paper demonstrates that ConvNeXt-Tiny, when combined with rigorous threshold optimization, is a highly effective, efficient, and reliable solution for detecting visually deceptive phishing websites.