Technical Summary: The Anatomy of Scam Scenarios: Large-Scale Characterization and Conversation-Aware Detection

1. Problem Statement

Online scams have evolved into a global epidemic, inflicting severe financial and psychological harm while imposing significant operational burdens on financial institutions. While prior research has extensively studied Psychological Techniques (PTs) (e.g., authority, urgency, fear) as isolated persuasive signals, it often overlooks the recurring operational schemes (scam scenarios) in which these techniques are systematically deployed. Scammers do not arbitrarily stack PTs; they embed them into reusable narratives (e.g., fake job offers, impersonation of utility services) to scale campaigns with minimal adaptation.

Furthermore, existing detection systems struggle in practical defense settings. Scam defense often occurs during partial and unfolding customer-service interactions, where defenders must infer the underlying scam scenario from incomplete, noisy, and evolving customer narratives. Current approaches fail to effectively transfer knowledge from large-scale retrospective incident reports to real-time, turn-by-turn conversation analysis, particularly when facing challenges such as incomplete context, uncertain evidence onset, and diverse narrative styles.

2. Methodology

The paper proposes a two-phase approach: a large-scale empirical study to characterize scam scenarios and a conversation-aware detection framework (EARS) to operationalize this knowledge.

2.1. Empirical Study: Scam Scenario Discovery

The authors conducted a data-driven analysis on 102,054 real-world scam incident reports from the BBB Scam Tracker (Feb 2024 – Oct 2025).

• Taxonomy Construction: Using BERTopic for initial clustering, the authors identified 99 preliminary topics. These were refined through an LLM-assisted summarization and human expert annotation pipeline (involving active learning) to consolidate into 18 distinct fine-grained scam scenarios.
• Hierarchical Grouping: The 18 scenarios were grouped into 6 high-level tactics based on their dominant PT profiles (e.g., "Authority & Compliance," "Relationship & Trust").
• Infrastructure Analysis: The study analyzed shared infrastructure (IPs, domains, emails) to identify coordinated campaigns, revealing that scammers reuse resources to launch large-scale, cross-tactic operations.

2.2. EARS: Conversation-Aware Scam Scenario Detection

To translate scenario-level knowledge into early detection, the authors developed EARS (Early Anti-scam Recognition System). The system treats detection as a sequential text classification task where the model updates predictions as conversation turns accumulate.

• Threat Model: The defender (e.g., a bank agent) observes a customer's narrative during a fraud investigation. The goal is to identify the scam scenario as early as possible, distinguishing between legitimate inquiries and evolving scams.
• Two-Stage Training Pipeline:
  1. Incident Report Pretraining: A scenario classifier is pre-trained on the large-scale corpus of incident reports. This allows the model to learn robust scenario-level semantics before adapting to the conversational format.
  2. Transfer Learning for Conversations: The pre-trained model is fine-tuned on customer-service conversations using three key innovations:
     • Phase-Aware Modeling: The system classifies each turn into one of three phases: Unrelated (greetings), Narrative (background), or Scam Evidence. It injects phase-state features and evidence onset offsets into the model to help it withhold judgment during benign phases and commit to a prediction once diagnostic evidence appears.
     • LLM-Based Data Augmentation: To address the scarcity of labeled conversation data and stylistic variability, an LLM rewrites conversations with varied personas and narrative structures while preserving the underlying scenario labels.
     • Phase-Aware Turn Loss (PATL): A custom loss function up-weights turns containing scam evidence and down-weights unrelated turns, encouraging the model to focus on diagnostic signals.

3. Key Contributions

1. Large-Scale Taxonomy: The first data-driven, hierarchical taxonomy of scam scenarios, comprising 18 fine-grained scenarios grouped into 6 tactics, derived from over 100k incident reports.
2. Empirical Insights:
   • PT Stacking: Scammers frequently stack PTs; each additional PT correlates with a 26.27% increase in expected monetary loss.
   • Strategic Adaptation: Scammers tailor scenarios to specific goals (e.g., broad exposure vs. high-value extraction) and adapt infrastructure (disposable vs. resilient domains) accordingly.
   • Campaign Coordination: Over 30% of incidents are linked to distinct campaigns, with the largest campaign spanning all 18 scenarios and 3,809 incidents.
3. EARS Framework: A novel detection system that successfully transfers scenario knowledge from static reports to dynamic conversations, handling incomplete context and uncertain evidence onset through phase-aware modeling and data augmentation.

4. Results

The system was evaluated on 1,115 customer-service conversations (synthesized from real fraud-investigation calls by industry partner Charm Security).

• Tactic-Level Detection: EARS achieved 84.41% accuracy and 84.14% F1 in classifying the high-level scam tactic.
• Scenario-Level Detection: EARS achieved 75.17% accuracy and 71.39% macro-F1 for fine-grained scenario classification.
• Early Detection Capability: The system ranks the correct scenario within its top-3 predictions in up to 91.04% of cases. Notably, accuracy improves significantly just one turn after the first scam evidence appears (jumping from ~56% to ~70%+).
• Comparative Performance:
  • EARS outperformed the best few-shot LLM baseline (GPT-5.4) by 25.62 points in scenario macro-F1.
  • It outperformed standard Transformer baselines (without phase modeling or augmentation) by 5.24 points in scenario macro-F1.
  • Ablation studies confirmed that both LLM-based data augmentation and phase-aware modeling are critical, contributing roughly 2–4 percentage points each to the final performance.

5. Significance and Claims

The paper claims that scenario-level knowledge distilled from large-scale incident reports is transferable and effective for early detection in customer-service conversations. By moving beyond isolated PT detection to a holistic scenario-aware approach, the authors demonstrate that:

• Scammers operate within structured, reusable operational schemes that can be systematically characterized.
• Automated systems can leverage these patterns to support timely intervention, reducing investigative workload and financial losses.
• The proposed methodology bridges the gap between retrospective forensic analysis and real-time defensive action, providing a practical foundation for both analyzing scam ecosystems and deploying early-warning systems in financial institutions.

The authors maintain a modest stance, acknowledging that the taxonomy is not exhaustive (including an "Other" category for unidentifiable reports) and that the system relies on synthetic data due to privacy constraints, though they argue the synthetic data faithfully reflects real-world interaction patterns.