Technical Summary: Measuring and Evaluating the Performance of Generative AI Models for Scam Detection

Problem Statement

Online scams represent a growing threat, causing substantial financial and personal harm by exploiting human trust and emotions. While traditional machine learning techniques (e.g., Logistic Regression, SVM, Naive Bayes) have been widely applied to scam detection, they often generalize poorly to novel or unseen attacks. Consequently, there is a rapidly accelerating adoption of Large Language Models (LLMs) in security products, driven by a common belief that their strong capabilities in understanding intent, context, and reasoning make them broadly suitable for scam detection. However, this paper argues that despite the popularity of generative AI in domains like vulnerability and bug detection, a rigorous, comprehensive assessment of LLMs specifically for scam detection is currently missing.

Methodology

The authors conducted the first comprehensive empirical study evaluating the effectiveness and limitations of popular LLMs for scam detection. The methodology involved the following key components:

1. Dataset Curation (ScamBenchmark)

The researchers curated and released a unique benchmark dataset comprising 2,742 unique samples from three primary sources:

• Reddit: 1,270 posts from the "Is this a scam?" subreddit, processed via OCR.
• Google Images: 615 images gathered via specific queries (e.g., "hi mom scams"), processed via OCR.
• MTurk: 857 legitimate message screenshots submitted by participants.
• Proprietary Data: A validation set of 59,991 real-world samples provided by a security company.

Each sample was labeled as Scam, Clean, or Uncertain (for ambiguous cases requiring external metadata). The data was further categorized into 25 fine-grained scam types.

2. Model Selection

The study evaluated nine LLMs of varying sizes and architectures, alongside a fine-tuned BERT-based classifier:

• Open-Source: Mistral (7B, 8x7B), Llama 3 (8B), Llama 3.1 (8B, 70B), Llama 3.2 (11B).
• Proprietary: OpenAI ChatGPT (4 Turbo, 4o, 4o1).
• Baseline: Fine-tuned BERT.

3. Experimental Design

The evaluation focused on several variables:

• Prompting Strategies: Zero-shot, Few-shot (1, 5, 10 shots), Chain-of-Thought (CoT), prompts incorporating "Common Signs" of scams, and prompts augmented with "URL Intelligence" (using external safety assessments).
• Hyperparameter Tuning: Evaluation of temperature and top-p values (0.1, 0.5, 0.9) to assess the impact of output stochasticity.
• Fine-Tuning: Supervised fine-tuning of Mistral 7B, Llama 3.1 8B, and BERT using balanced training sets (400 samples per class).
• Ensemble Analysis: Assessment of model agreement using Cohen's Kappa and testing of ensemble combinations.

Key Results

Model Performance and Size

• Larger Models Outperform: Generally, larger models achieved better results. ChatGPT 4o1 and Llama 3.1 70B achieved the highest micro-F1 scores (64% and 65%, respectively) under zero-shot prompting.
• Prompt Sensitivity: Prompting strategies significantly impacted performance, but the effect was model-dependent.
  • Small Models: Effective prompting (e.g., "Common Signs") substantially boosted smaller models. For instance, Llama 3.1 8B's micro-F1 rose from 50% to 57% with tailored prompts.
  • Large Models: Some large models (e.g., ChatGPT 4o1) saw performance degradation or mixed results with complex prompting strategies like Chain-of-Thought.
• Few-Shot Learning: Increasing the number of shots generally improved performance, but not universally. For some models (e.g., Llama 3.2 11B), 1-shot or 5-shot settings outperformed 10-shot settings.

Generalization vs. Fine-Tuning

• Generalization: LLMs demonstrated superior generalization to unseen data compared to fine-tuned models. When tested on a large proprietary dataset, the fine-tuned BERT model's performance dropped significantly (micro-F1 0.393), whereas Llama 3.1 70B maintained a stable score (0.596).
• Fine-Tuning Limits: While fine-tuning improved the performance of smaller LLMs (e.g., Mistral 7B improved from 0.470 to 0.543 micro-F1), they still fell short of the best-performing zero-shot large models.
• BERT Competitiveness: In controlled settings, fine-tuned BERT performed competitively (0.59 micro-F1), surpassing smaller LLMs, but lacked the robustness of LLMs on distribution shifts.

Specific Categories

• Scam vs. Clean: Most models struggled to categorize "Clean" samples, with some achieving recall as low as 0.019. ChatGPT 4o1 was an outlier with 58% recall for clean samples.
• Uncertain Class: Many models struggled to classify samples as "Uncertain," though ChatGPT 4 Turbo performed best in this category.

Key Contributions

1. First Comprehensive Study: The paper presents the first evaluation of popular LLMs for scam detection using various prompting techniques and hyperparameters.
2. ScamBenchmark Dataset: The release of the first benchmark dataset specifically for evaluating LLMs in scam detection, including real-world samples across multiple formats and topics.
3. Empirical Limitations: The study empirically demonstrates that current LLMs are not a "silver bullet" for scam detection; their effectiveness is highly contingent on model-specific prompting and configuration.
4. Fine-Tuning Evaluation: The authors fine-tuned Mistral 7B, Llama 3.1 8B, and BERT, providing a comparative analysis of their performance against zero-shot large models.

Significance and Claims

The paper concludes that while larger LLMs (e.g., ChatGPT 4o1, Llama 3.1 70B) currently provide the best performance, they are not a universal solution. The authors claim that the path forward lies not in simply scaling model size, but in hybrid architectures that combine the semantic understanding of LLMs with the efficiency and precision of classical ML methods.

The study emphasizes that:

• LLMs generalize better to unseen scams than fine-tuned models, suggesting pre-trained knowledge is valuable.
• Prompt engineering is critical but must be tailored to specific models, as strategies that help small models may hinder large ones.
• No single model or approach solves scam detection universally; optimal deployment likely requires a pipeline where LLMs handle ambiguous inputs or feature extraction, while classical models handle deterministic classification.

The authors release their dataset and fine-tuned models to facilitate future research in robust scam detection, aiming to move beyond the "common belief" of broad suitability toward a more nuanced, evidence-based deployment strategy.