Technical Summary: Assessing AI-Generated vs. Human-Authored Spear Phishing SMS Attacks

Problem Statement

Personalized phishing (spear phishing) remains a critical cybersecurity threat because messages can be tailored to a target's specific work, interests, and social context, making them highly convincing. While Large Language Models (LLMs) promise to automate the creation of such targeted content at scale and low cost, there is a significant gap in empirical literature regarding whether AI-generated messages are actually more convincing than human-authored ones when both are tailored to the same individual. Furthermore, it is unclear if targets can reliably distinguish between AI and human sources, or if the text itself retains detectable signals of its origin. This study addresses these gaps by comparing GPT-4-generated spear phishing SMS (smishing) messages against those written by novice human authors for the same 25 targets.

Methodology

The study employed a pilot design involving 25 participants who acted as targets. The research utilized a proposed evaluation framework called TRAPD (Threshold Ranking Approach for Personalized Deception), designed to enable controlled, ethical comparisons of personalized deceptive content without deploying live attacks.

Experimental Design

1. Target Recruitment: 41 individuals completed a survey providing personal data (name, occupation, hobbies, recent social media posts, etc.). 25 of these participants returned for the evaluation phase.
2. Message Generation:
   • AI Condition: GPT-4 (via OpenAI API, temperature 0.7) generated messages based on a shared prompt structure using the target's data.
   • Human Condition: 99 novice student authors (undergraduate cybersecurity/honors students) wrote messages under time constraints (approx. 15 minutes) using the same prompt structure.
   • Topics: Messages were generated for three specific contexts per target: Job-related, Hobby-related, and Social Media-related.
   • Volume: Each target received 12 messages total (6 AI-generated, 6 human-authored; 2 of each per topic).
3. Evaluation Procedure (TRAPD):
   • Ranking: Participants physically arranged the 12 messages from most to least likely to induce a click.
   • Thresholding: Participants marked a threshold on the ranked list indicating the point above which they would intend to click.
   • Qualitative Explanation: Participants explained their reasoning for rankings and thresholds.
   • Source Attribution: Participants identified which messages they believed were AI-generated and explained their criteria.
4. Computational Analysis: A logistic regression classifier using semantic embeddings (text-embedding-3-small) was trained to distinguish between the two message sets. This analysis included sensitivity tests where obvious surface features (URLs, emojis, length, punctuation) were normalized to test if the classifier relied on deeper textual patterns.

Key Contributions

1. Controlled Comparison: Provides a direct, within-target comparison of GPT-4-generated and human-authored personalized smishing messages, a gap in existing literature which has largely focused on email or bulk phishing.
2. Source Attribution Assessment: Documents the accuracy of human targets in distinguishing AI from human sources and analyzes the specific cues (or lack thereof) they rely on.
3. Feature Analysis: Identifies message characteristics (relevance, sender identity, URL structure, style, urgency) that participants associate with convincingness or suspicion.
4. Computational Distinguishability: Demonstrates that while humans cannot distinguish the sources, the text of the study-specific message sets remains computationally distinguishable even after controlling for surface-level features.
5. TRAPD Framework: Introduces the TRAPD framework and documents its feasibility and limitations for controlled, within-target comparison of personalized deceptive content, trading some ecological realism for rigorous within-target control and rich qualitative data.

Results

Convincingness (RQ1 & RQ2)

• AI vs. Human: GPT-4-generated messages elicited an intention to click in 28.0% of cases, compared to 21.3% for human-authored messages. However, this difference was not statistically reliable (95% CI: -2.9 to 16.3 percentage points; p = 0.147). The study establishes neither that AI outperforms humans nor that the conditions are equivalent.
• Topic Impact: Job-related messages were significantly more convincing than hobby or social media messages. 38% of job-related messages exceeded the intended-click threshold, compared to 19% for hobbies and 17% for social media. Job messages had significantly higher intended-click odds than the other two categories (p < 0.05 after Holm adjustment).
• Qualitative Drivers: Participants cited personal relevance (76%), sender identity (68%), and URL characteristics (64%) as primary factors. Accurate personalization increased credibility, while mismatches (e.g., incorrect colleagues, irrelevant topics) raised suspicion.

Source Attribution (RQ3 & RQ4)

• Human Accuracy: Participants correctly identified the source (AI vs. Human) only 52.0% of the time, which is no better than chance (p = 0.597).
• Cues Used: Despite low accuracy, participants relied on inconsistent cues such as "style" (formality, generic tone), "word choice" (buzzwords vs. slang), "grammar" (perfection vs. errors), and "emojis." Many participants expressed uncertainty, relying on "gut feelings" rather than concrete criteria.

Computational Source Attribution (RQ5)

• Machine Distinguishability: In contrast to human performance, a computational classifier could distinguish the message sets with high accuracy even after strict normalization (removing URLs, emojis, standardizing length/case).
• Performance: In the strictest condition (length-equalized, normalized text), the classifier achieved a Balanced Accuracy of 88.7% and an ROC AUC of 0.954 on unseen targets. This suggests the study-specific message sets contained subtle, distributed textual patterns detectable by machines but not by humans.

Significance and Claims

The paper claims that accessible AI-assisted personalization may increase the practical scale of social engineering threats, even if individual AI messages are not currently superior to human ones. The key finding is that a simple prompt can produce personalized messages that are comparably convincing to those written by screened novice humans, within the uncertainty of this pilot.

The study highlights that workplace pretexts are particularly potent in smishing, warranting specific attention in training and defense. It also underscores a critical disconnect: while humans cannot reliably detect AI authorship, the text retains computable signals. However, the authors caution that these computational results are specific to the study's conditions (one model, one prompt, novice authors) and should not be treated as a general AI detector, especially given that attackers can revise text to evade detection.

Finally, the paper positions TRAPD as a valuable, ethically sound framework for future research, allowing for controlled comparisons and detailed reasoning analysis that live phishing experiments cannot provide, while acknowledging its limitations regarding ecological validity and generalizability.