Context-Aware Detection and Victim-Centered Response Generation for Online Harassment in Private Messaging

Technical Summary: Context-Aware Detection and Victim-Centered Response Generation for Online Harassment in Private Messaging

Problem Statement
Online harassment is a significant public health concern, particularly for adolescents, yet computational detection and intervention efforts have predominantly focused on publicly visible social media content. Private messaging environments (e.g., Instagram Direct Messages) present unique challenges: harmful interactions often unfold through context-dependent, multi-turn exchanges where meaning relies on conversational history, sarcasm, and relational dynamics rather than isolated text. Furthermore, existing interventions are largely reactive (e.g., reporting, blocking) or perpetrator-focused (e.g., counterspeech), often failing to provide timely, emotionally supportive assistance to victims during the harassment event. There is a critical lack of datasets and scalable methods for detecting harassment in private contexts, as well as a gap in AI-driven, victim-centered response generation.

Methodology
The study utilized a dataset of 80,053 Instagram direct messages donated by 26 adolescents (aged 12–18), including youth with suicide risk factors. The research addressed two primary research questions (RQs) using Large Language Models (LLMs).

1. Detection (RQ1): Context-Aware Cascading Classification

• Data Annotation: Human annotators labeled 7,525 messages (after filtering) for online harassment. To ensure context-awareness, annotators reviewed the target message alongside the preceding 10 messages. A two-annotator system with a third adjudicator resolved disagreements, resulting in a highly imbalanced dataset (41 harassment cases, 7,484 non-harassment).
• Pipeline Architecture: The authors developed a two-stage cascading LLM pipeline using the meta-llama/Llama-4-Scout-17B-16E-Instruct model.
  • Stage 1 (LLM 1): Received the target message and the preceding 50 messages as context. It was prompted to classify the message and provide a reasoning explanation.
  • Stage 2 (LLM 2): Received the output of LLM 1 (label + reasoning) along with the conversation context. This stage acted as a conservative filter to reduce false positives, specifically re-evaluating cases flagged as harassment by the first stage.
• Baselines: The pipeline was compared against a pretrained toxicity classifier (toxic-bert) and a supervised ensemble learning framework trained on public social media data.

2. Response Generation (RQ2): Victim-Centered Framework

• Pipeline Architecture: A two-stage generation pipeline using the same LLM model.
  • Stage 1 (Strategy Selection): Analyzed the conversation context to select evidence-informed response strategies (e.g., de-escalation, empathy, moral suasion, warning) and generated a rationale.
  • Stage 2 (Linguistic Realization): Generated the actual response messages conditioned on the selected strategies and the original context.
• Constraints: To ensure ecological validity, the model was constrained to generate responses resembling adolescent communication styles (e.g., textese, informal grammar) with specific length constraints (3–13 words per message, 1–2 consecutive messages). The system was explicitly instructed to avoid retaliation and prioritize de-escalation.
• Evaluation: A blinded human evaluation was conducted on 100 harassment cases. Three evaluators compared AI-generated responses against the original participant responses across six dimensions: effectiveness in stopping harassment, de-escalation, victim positioning, emotional support, naturalness, and the preference for ignoring vs. responding.

Key Results

• Detection Performance: The cascading LLM pipeline achieved an overall accuracy of 0.98 and a macro F1-score of 0.63. Crucially, for the harassment class, it achieved a recall of 0.60 and an F1-score of 0.28. This significantly outperformed both baselines: toxic-bert (Positive F1: 0.14) and the ensemble (Positive F1: 0.23). The results indicate that context-aware LLMs are better suited for private messaging harassment than models trained on isolated public posts.
• Response Evaluation: Human evaluators perceived AI-generated responses as significantly more helpful than original participant responses (95% CI: 0.767–0.815, p < .001). AI responses were preferred in 80.2% of cases for emotional support, de-escalation, and practical helpfulness. However, evaluators rated the original participant responses as more "natural" in terms of conversational style (95% CI: 0.035–0.089, p < .001).
• Error Analysis: The pipeline missed some indirect or implied harassment (false negatives) due to the conservative nature of the second-stage filtering, which prioritized reducing false positives.

Significance and Claims
The paper claims that context-aware and victim-centered AI systems hold significant potential for providing "just-in-time" support in private messaging environments.

• Detection: The study demonstrates that leveraging conversational context via cascading LLM pipelines is essential for detecting harassment in private spaces, outperforming traditional toxicity classifiers that fail to capture relational dynamics and escalation patterns.
• Intervention: The research introduces the concept of "synthetic buffering," where AI-generated responses absorb the emotional and cognitive burden of responding to harassment. By offering strategy-informed, de-escalatory drafts, these systems can empower victims without requiring them to formulate regulated responses while in distress.
• Limitations: The authors modestly note that while AI responses are perceived as more helpful, they currently lack the linguistic naturalness of human responses. They emphasize that these systems should not replace human support or professional intervention and require safeguards against misuse or unintended escalation. The findings are based on a specific, small-scale dataset of adolescents, necessitating future work on larger, more diverse populations and real-world outcome evaluation.