Technical Summary: Benchmarking Large Language Models for Zero-shot and Few-shot Phishing URL Detection

Problem Statement

Phishing attacks have evolved into a sophisticated cybersecurity threat, increasingly leveraging generative AI to create context-aware, deceptive URLs that are difficult for users and traditional detection tools to distinguish. While reactive protections like HTTPS exist, the URL structure itself lacks inherent mechanisms for security or resilience against fraud. Traditional detection systems, particularly those relying on blacklists or handcrafted lexical features, struggle with newly generated or obfuscated URLs due to their dependence on prior knowledge. Furthermore, the rapid escalation of phishing tactics—evidenced by a 4,000% volume increase since 2022—outpaces the production of labeled data required for supervised learning. This creates a critical need for detection methods capable of generalizing to unseen or adversarial examples with minimal supervision.

Methodology

The authors present a comprehensive benchmark evaluating three proprietary Large Language Models (LLMs)—GPT-4o (OpenAI), Claude-3.7-sonnet-20250219 (Anthropic), and Grok-3-Beta (xAI)—on the task of phishing URL classification. The study utilizes a unified prompting framework to assess performance under both zero-shot and few-shot settings.

Experimental Setup

• Dataset: The evaluation uses the PhiUSIIL Phishing URL dataset.
  • Balanced Evaluation: A corpus of 10,000 URLs (5,000 phishing, 5,000 legitimate) was constructed using random sampling.
  • Imbalanced Evaluation: Test sets of 1,000 URLs were created with phishing ratios of 1% and 10% to simulate real-world class distributions.
• Prompt Construction:
  • Zero-shot ($P_{ZS}$): Consists of a task instruction ($I$) and a query ($Q(u)$). The instruction defines the model as a cybersecurity expert and requests a binary response (0 for phishing, 1 for legitimate).
  • Few-shot ($P_{FS}$): Includes the instruction, the query, and a set of examples ($E$). In the balanced experiments, 6 examples (3 phishing, 3 legitimate) were used. In imbalanced experiments, the number of examples varied (1, 3, or 9).
• Inference: All models were accessed via their official APIs with a temperature of 0 and a maximum output of 10 tokens. Responses were parsed to extract binary predictions.
• Metrics: Performance was quantified using six metrics: Accuracy, Macro-averaged Precision, Recall, F1-score, AUROC, and AUPRC.

Key Contributions

The paper makes three primary contributions to the cybersecurity and AI research community:

1. Unified Benchmark: It establishes a standardized benchmark for instruction-tuned proprietary LLMs in phishing URL detection, comparing zero-shot and few-shot capabilities across multiple models.
2. Controlled Evaluation: The study evaluates models on a balanced dataset using consistent, model-agnostic prompt designs, addressing the lack of fair comparison in prior literature where datasets or metrics often varied.
3. Reproducibility: The authors release their codebase to facilitate further research and reproducibility.

Experimental Results

Balanced Dataset (10,000 URLs)

Few-shot prompting with six examples significantly improved performance across all models compared to zero-shot baselines.

• Grok-3-Beta emerged as the top performer in the few-shot setting, achieving the highest Accuracy (0.9405), Precision (0.9492), F1-score (0.9399), AUROC (0.9405), and AUPRC (0.9573). It demonstrated a notable shift in the precision-recall trade-off, increasing precision from 0.8361 to 0.9492 while slightly reducing recall.
• Claude-3.7 achieved the highest Recall (0.9526) in the few-shot setting but had lower precision compared to Grok-3-Beta.
• GPT-4o showed consistent gains across all metrics but trailed the other two models in overall performance.
• Confusion Matrices: Few-shot prompting reduced false negatives across all models. Grok-3-Beta reduced false negatives from 950 (zero-shot) to 248 (few-shot).

Imbalanced Dataset (1% and 10% Phishing Ratios)

• Zero-Shot Performance: Grok-3-Beta demonstrated strong zero-shot capabilities, achieving an accuracy of 0.976 and recall of 0.938 at a 10% phishing ratio.
• Few-Shot Sensitivity: The impact of the number of examples varied by model:
  • Grok-3-Beta: Peaked with only 1 example (F1: 0.906) and performance degraded slightly with 3 or 9 examples.
  • Claude-3.7: Showed a steady improvement as the number of examples increased, reaching its best F1-score (0.876) with 9 examples.
  • GPT-4o: Demonstrated consistent gains from 1 example (F1: 0.709) to 9 examples (F1: 0.861).
• Robustness: Models maintained robust performance across different random seeds (S123, S456), with F1-score differences remaining below 0.05 at 10% imbalance.

Significance and Claims

The paper argues that as phishing tactics evolve faster than labeled data can be produced, zero-shot and few-shot learning with LLMs offers a timely and adaptable solution. The study concludes that few-shot prompting effectively improves detection performance across multiple proprietary LLMs, enabling generalization with minimal supervision.

Specifically, the authors claim that:

• Prompt-based approaches can effectively detect phishing URLs with very few examples (e.g., Grok-3-Beta performing well with a single example).
• There are distinct operational trade-offs between models; some (like Grok-3-Beta) excel with minimal examples and high precision, while others (like Claude-3.7) may benefit from more examples to maximize recall.
• The current landscape lacks unified benchmarking for LLMs in this domain, and this work fills that gap by providing a consistent evaluation framework.

The authors maintain a modest scope, noting that future work is required to validate these findings across diverse datasets, explore prompt optimization strategies, and investigate the trade-offs between prompt-based inference and fine-tuned models in operational settings.