Technical Summary: VEXA – Evidence-Grounded and Persona-Adaptive Explanations for Scam Risk Sensemaking

Problem Statement

Online scams across email, SMS, and social media have evolved into a pervasive security challenge, exacerbated by generative AI which enables fluent, context-aware deception. While transformer-based detectors achieve high predictive accuracy, their internal decision processes remain opaque to non-expert users. Existing Explainable AI (XAI) techniques (e.g., SHAP, LIME) typically provide token-level visualizations that are difficult for laypersons to interpret. Furthermore, Large Language Models (LLMs), while capable of generating natural language, often rely on parametric knowledge rather than the specific detector's decision process, leading to "unfaithful" explanations that may fabricate rationales or foster incorrect mental models of risk. Additionally, security psychology research indicates that users vary in their risk perception and information processing styles (e.g., based on personality traits), yet current security interventions often deliver monolithic, one-size-fits-all warnings that fail to provide adaptive scaffolding for diverse learners.

Methodology: The VEXA Framework

The authors propose VEXA (Vulnerability-aware and Evidence-grounded eXplanations with persona Adaptation), a system-level framework designed to generate learner-facing scam explanations that are both faithful to the detection model and stylistically adapted to user vulnerability profiles. The framework operates through four sequential stages:

1. Scam Detection: A frozen, BERT-based classifier (specifically DeBERTa-v3-base, selected for highest Macro F1) analyzes input messages from email, SMS, or SNS to produce a binary scam prediction.
2. XAI Evidence Extraction: The system applies GradientSHAP in the embedding space to identify salient tokens contributing to the prediction. These attributions are aggregated into word-level scores, filtered to remove stopwords while retaining risk-relevant tokens (URLs, currency mentions, emphatic punctuation), and form a compact set of evidence phrases.
3. Persona Selection: Instead of modeling individual users, VEXA utilizes theory-informed vulnerability personas derived from the Big Five personality traits. Two binary personas are defined:
   • High Vulnerability: Characterized by high neuroticism and low conscientiousness. Explanations for this persona are designed to be calmer, more contextual, and supportive.
   • Low Vulnerability: Characterized by high conscientiousness and low emotional reactivity. Explanations for this persona are concise, analytical, and foreground explicit evidence.
     These personas serve as design constructs to modulate tone and structure, not to predict individual behavior.
4. LLM Explanation Generation: A Large Language Model is prompted with the original message, the extracted XAI evidence, and persona-specific instructions. The LLM translates the detector-derived cues into accessible natural language, constrained to reference the evidence while adapting style according to the selected persona.

Key Contributions

The paper outlines four primary contributions:

1. Evidence-Grounded Framework: A system-level approach that ensures explanations remain faithful to detection models by grounding them in model-derived decision evidence across heterogeneous communication contexts (email, SMS, SNS).
2. Persona-Driven Adaptation: A demonstration that explanation style can be systematically modulated using theory-informed vulnerability personas to accommodate variability in user risk interpretation, without requiring individual-level user data or behavior prediction.
3. Multi-Channel Applicability: Validation of the framework across diverse scam modalities (SNS, SMS, email), showing consistent explanation design under different communication constraints.
4. Design-Focused Analysis: A framing of risk sensemaking as critical digital literacy, analyzing how evidence grounding and persona adaptation shape security explanations from a system design perspective.

Experimental Results

The authors evaluated VEXA using a multi-channel corpus of scam and legitimate messages, comparing four generation settings: Pure LLM, LLM + XAI (neutral), LLM + XAI + High Vulnerability, and LLM + XAI + Low Vulnerability.

• Faithfulness: Explanations grounded in XAI evidence (regardless of persona) achieved high and consistent faithfulness scores (~0.745–0.749), significantly outperforming the ungrounded baseline. Persona conditioning did not interfere with the incorporation of salient evidence tokens.
• Correctness (Semantic Reliability): Grounding explanations in evidence significantly improved semantic correctness (from 0.366 in the baseline to 0.592 in the XAI-only condition). This improvement occurred without increasing linguistic complexity.
• Readability (Stylistic Variation): Persona conditioning successfully modulated surface-level linguistic complexity. The High Vulnerability persona produced explanations with lower Flesch–Kincaid Grade Level (FKGL) scores (12.01), indicating simpler, more contextual phrasing. The Low Vulnerability persona yielded higher FKGL scores (14.39), reflecting concise, analytical language. Crucially, these stylistic shifts did not degrade semantic reliability.
• Expert Evaluation: Domain experts found the explanations plausible and aligned with real-world scam identification practices. They noted that persona-driven differences were reasonable (supportive vs. analytical) and that grounding in explicit cues improved clarity over generic warnings. Experts suggested further refinements in grouping related indicators and distinguishing promotional claims from action prompts.

Significance and Claims

The paper claims that VEXA demonstrates the feasibility of generating persona-adaptive, evidence-grounded explanations for non-formal security learning. The central design insight revealed is that evidential grounding governs semantic correctness, ensuring the explanation reflects the model's actual logic, while persona-based adaptation operates at the level of presentation (style, tone, structure) under constraints of faithfulness.

The authors argue that these two mechanisms are not competing objectives but operate on complementary dimensions. By decoupling semantic content (constrained by evidence) from stylistic delivery (modulated by persona), the framework offers a principled balance between interpretability, personalization, and reliability. The work provides design guidance for trustworthy security explanations that support risk sensemaking without requiring invasive user modeling.

The authors remain modest regarding their claims, acknowledging that the current evaluation relies on expert assessment rather than human-subject studies measuring actual learning or behavioral outcomes. They note that simplified persona definitions may not capture the full diversity of user experiences and that future work is needed to assess real-world effects on user decision-making.