A Synthetic Conversational Smishing Dataset for Social Engineering Detection

1. Problem Statement

Smishing (SMS phishing) is a rapidly growing cybersecurity threat, particularly targeting elderly and cyber-unaware individuals. While existing research focuses on classifying single messages as malicious or benign, real-world attackers often employ multi-stage social engineering strategies. These attacks involve extended conversational exchanges where attackers gradually build trust ("grooming") before extracting sensitive information.

Key Gaps Identified:

• Lack of Conversational Data: Existing datasets consist of isolated SMS messages, failing to capture the temporal dynamics, trust-building, and multi-turn interactions characteristic of modern attacks.
• Privacy Constraints: Collecting real-world conversational data involving vulnerable populations (e.g., the elderly) poses significant ethical and legal challenges.
• Detection Limitations: Single-message classifiers cannot detect malicious intent that unfolds over a dialogue, leading to high false negatives in conversational scenarios.

2. Methodology: The COVA Framework

The authors propose COVA (Cognitive Operations Virtual Assistant), a three-phase framework to generate synthetic data and establish detection baselines.

A. Synthetic Data Generation (Multi-Agent Simulation)

Instead of using monolithic prompts, the framework employs two independent Large Language Model (LLM) agents interacting iteratively:

• Attacker Agent: Configured with specific personas (name, role), scam types, and tactical parameters (urgency, payment method). It possesses varying levels of "knowledge" about the victim (cold call vs. partial vs. full knowledge).
• Victim Agent: Parameterized with demographics (age 65–85), personality traits (trust level, tech savviness), and resistance tendencies.
• Generation Process: The conversation proceeds turn-by-turn. The victim initiates, the attacker responds, and agents alternate turns conditioned on the full conversation history.
• Infrastructure: To bypass commercial API safety guardrails that prevent role-playing as scammers, the authors used Qwen 2.5 14B running locally via Ollama on an NVIDIA RTX 4080 Super GPU. This ensured data privacy and zero cost.

B. Dataset Composition

• Scale: 3,201 multi-turn conversations.
• Categories: 8 scam types disproportionately affecting the elderly (Grandparent, Virtual Kidnapping, Medicare Fraud, Romance, Government Impersonation, Investment, Lottery, Bank).
• Statistics: ~25,643 total turns (avg. 16.2 turns/conversation).
• Labeling & Quality Assurance:
  • Initial labels were assigned by a rule-based classifier, resulting in a 49.8% mismatch rate (e.g., misinterpreting hesitation as compliance).
  • A v2 Audit Pipeline was developed using negation-aware matching and heuristic analysis.
  • Outcome Classes: The original 5 fine-grained labels were collapsed into 3 robust classes for modeling:
    1. Complied: Victim fully complies.
    2. Partial: Victim engages but does not commit.
    3. Rejected: Victim rejects or terminates the call.

C. Experimental Setup

The study evaluated 8 models on the 3-class outcome prediction task:

• Traditional ML: Logistic Regression, Random Forest, Linear SVM, XGBoost.
• Transformers: DistilBERT (66M params), Longformer (148M params).
• Feature Sets:
  1. Engineered Features: 28 numeric features (conversation length, resistance counts, financial signals).
  2. TF-IDF: Textual representations (unigrams/bigrams) combined with engineered features.
  3. Contextual Embeddings: Used for transformer models.

3. Key Results

• Best Performing Model: XGBoost with TF-IDF + Engineered Features.
  • Accuracy: 72.5%
  • Macro F1 Score: 0.691
• Performance Comparison:
  • TF-IDF vs. Engineered Features: Models using TF-IDF significantly outperformed those using only engineered features (e.g., XGBoost improved from 56.5% to 72.5% accuracy). This indicates that lexical patterns are more discriminative than hand-crafted behavioral metrics.
  • Transformers vs. Traditional ML: Contrary to expectations, transformer models (DistilBERT and Longformer) underperformed compared to XGBoost + TF-IDF.
    • DistilBERT (Tail-only): 69.8% Accuracy.
    • Longformer (No truncation): 69.8% Accuracy.
• Error Analysis:
  • The model struggled most with distinguishing between Partial and Rejected classes, as both involve extended engagement before a final decision.
  • Five-Class Analysis: Performance dropped significantly when attempting to classify the original 5 classes due to severe class imbalance (e.g., only 2 samples for "Quick Rejection" in the test set).

4. Key Contributions

1. First Public Conversational Smishing Dataset: Introduction of a 3,201-conversation dataset specifically designed for elder-targeted, multi-turn social engineering attacks.
2. Synthetic Generation Framework: A reproducible, multi-agent LLM pipeline that generates realistic attacker-victim dynamics while avoiding privacy violations associated with real-world data collection.
3. Baseline Benchmarks: Comprehensive evaluation of 8 models, establishing that TF-IDF-based traditional ML currently outperforms fine-tuned transformers for this specific task and dataset size.
4. Quality Assurance Protocol: Demonstration of the necessity for rigorous human-in-the-loop auditing in synthetic data generation, revealing a nearly 50% initial labeling error rate.

5. Significance and Discussion

• Lexical Signals are Critical: The results suggest that specific keywords and phrases (e.g., "verify," "official," "card") carry more predictive power for smishing outcomes than abstract behavioral features in short-to-medium length conversations.
• Limitations of Transformers: The underperformance of transformers is attributed to two factors:
  1. Input Length Constraints: Even with Longformer's extended context, truncation or limited context windows may discard critical "outcome-critical" final turns.
  2. Data Scarcity: With only ~2,500 training samples, transformers overfit (validation loss climbed while training loss dropped), whereas XGBoost is more sample-efficient.
• Future Directions: The authors propose expanding the dataset to >10,000 conversations to better train transformers, developing incremental classification for real-time intervention, and adding non-English categories.

Conclusion

This paper bridges a critical gap in cybersecurity research by providing a dataset and baseline models for conversational smishing detection. It demonstrates that while deep learning is powerful, traditional machine learning with robust lexical features (TF-IDF) remains the most effective approach for detecting social engineering outcomes in current, smaller-scale conversational datasets. The work lays the foundation for developing real-time defensive systems against multi-stage attacks targeting vulnerable populations.