An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection

Technical Summary: An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection

Problem Statement
Conversational social engineering attacks (smishing) targeting elderly individuals represent a growing cybersecurity threat that unfolds across multiple dialogue turns. Unlike single-message phishing, these attacks rely on gradual trust-building, making them difficult to detect with traditional single-message classifiers. Real-world data for such interactions is scarce due to privacy and ethical constraints. The authors' prior work, COVA, introduced a synthetic dataset of 3,201 conversations but revealed a critical limitation: transformer models (DistilBERT and Longformer) underperformed compared to XGBoost with TF-IDF features. This gap was attributed to input truncation discarding outcome-critical final turns and insufficient training data for effective transformer fine-tuning.

Methodology: COVA-X Expansion and Quality Lifecycle
This paper presents COVA-X, an expanded synthetic dataset containing 10,985 multi-turn conversations (a 3.43× increase over the original), spanning eight elder-targeted scam categories (Romance, Government Impersonation, Medicare, Grandparent, Bank Impersonation, Virtual Kidnapping, Investment, and Lottery).

The generation pipeline utilized Qwen 2.5 14B via Ollama on local GPU hardware, employing two independent LLM agents (attacker and victim). Key methodological improvements over the first iteration include:

• Profile Expansion: Introduction of five victim profiles per scam type, including two "outcome-pinned" profiles (Carol for quick_rejection, George for scam_detected) to ensure behavioral diversity and signal strength.
• Prompt Engineering Fixes: Implementation of "Name Identity Lock" to prevent character mixing, "ABSOLUTE OUTPUT RULE" injections to suppress stage-direction bleed, and rewrites to eliminate "Grandparent-opener" contamination in non-grandparent scams.
• Architectural Intervention for Virtual Kidnapping: To address high artifact rates in the "Virtual Kidnapping" category (which involves three parties: attacker, victim, and hostage), the authors shifted from a two-role architecture (attacker generating hostage voice within its own turn) to a three-role architecture (Path B). In Path B, the hostage is a distinct agent with its own system prompt and turn, folded back into the attacker's turn at JSON-write time.
• Quality Lifecycle: A comprehensive pipeline including contamination scanning, stage-direction stripping, automated relabeling, and a post-cleanup audit.

Key Contributions

1. Dataset Expansion: The release of COVA-X (10,985 conversations) with five victim/attacker profiles per category, addressing the data scarcity that previously hindered transformer performance.
2. Quality Lifecycle Documentation: A quantified analysis of the dataset construction process, demonstrating a 12.7× improvement in label correction rates (reducing mismatches from 49.8% in the first iteration to 3.9%).
3. Architectural Validation: Evidence that the three-role intervention for Virtual Kidnapping reduced artifact flag rates from 67.1% to 46.5%, with the majority of the reduction (20.4 percentage points) attributed to the architectural change rather than post-processing.
4. Model Comparison: A retraining of all classifiers (XGBoost, DistilBERT, Longformer) on the expanded dataset to test the hypothesis that larger corpora enable transformers to leverage their contextual advantages.
5. Generation Model Characterization: Documentation of specific capability limits in Qwen 2.5 14B under sustained emotional pressure, including failure to adopt structural tags, salience bias in prompt engineering (where listing forbidden patterns increased their occurrence), and profile compliance breakdowns.

Results
Retraining classifiers on the cleaned COVA-X dataset yielded the following results on the 3-class outcome prediction task (complied, partial, rejected):

• Longformer (Full Context): Achieved 79.71% accuracy and 0.7786 macro F1.
• XGBoost + TF-IDF: Achieved 78.43% accuracy and 0.7563 macro F1.
• DistilBERT v2 (Tail Truncation): Achieved 77.53% accuracy and 0.7632 macro F1.

Key Findings:

• Hypothesis Confirmation: Longformer surpassed XGBoost on all metrics for the first time, confirming that transformer models require larger conversational corpora (>5,000 conversations) to realize their contextual advantages. Longformer showed an 11.16 percentage point improvement in macro F1 compared to the first paper, significantly outpacing XGBoost's 6.53 point improvement.
• Cross-Architecture Consistency: A pre/post-cleanup sensitivity analysis showed that all three architectures improved on the cleaned dataset. This suggests the cleanup pipeline recovered genuine label-relevant signal accessible across different representational substrates (bag-of-words vs. transformer embeddings).
• Per-Scam-Type Dynamics: Outcome distributions varied by scam mechanism. Virtual Kidnapping had the highest successful_scam rate (33%) due to emotional pressure, while Grandparent scams had the highest verification_attempt rate (63%). Medicare and Bank scams showed high quick_rejection rates, reflecting public awareness.

Significance and Claims
The paper claims that the central finding—Longformer's superiority over XGBoost on the expanded dataset—directly validates the data-size hypothesis from the authors' prior work. Furthermore, the authors argue that the transparent documentation of the iterative challenges in synthetic dataset construction (including the 12.7× label improvement and the specific failure modes of the generation model) constitutes a methodological contribution in its own right. This provides a practical roadmap for future synthetic dataset efforts, highlighting that pipeline refinements and architectural interventions (like the three-role generation for Virtual Kidnapping) are critical for dataset integrity.

The authors modestly note limitations, including the dataset being English-only and US-centric, the lack of real-time incremental classification capabilities, and the persistence of specific generation model limitations (e.g., Qwen 2.5 14B's inability to consistently follow structural tags under emotional pressure) which motivate future model-comparison studies.