Technical Summary: SuiChat-CN

Problem Statement

Suicide remains a critical global public health challenge, necessitating timely and effective prevention strategies. While computational research has advanced suicide risk detection on post-based social media platforms (e.g., Twitter, Weibo), these approaches rely on isolated message analysis. This paradigm is insufficient for instant messaging (IM) environments like Telegram group chats, which exhibit distinct linguistic and interactional properties:

• Fragmentation and Brevity: Messages are short and often lack standalone context.
• Multi-party Dynamics: Conversations involve multiple participants, requiring the tracking of turn-taking and speaker roles.
• Implicit and Cultural Nuance: Risk signals often rely on metaphors, homophones, slang, or culturally specific euphemisms rather than explicit keywords.
• Context Dependence: As illustrated in the paper, a message appearing benign in isolation (e.g., "salt") may indicate high risk (drug use) when viewed within the conversational history.

Existing datasets lack the granularity to model these multi-turn, context-dependent dynamics, particularly for the Chinese language.

Methodology

The authors introduce SuiChat-CN, a benchmark dataset and construction pipeline designed specifically for contextual suicide risk assessment in Chinese group chats.

1. Dataset Construction Pipeline

The construction follows a four-stage pipeline (Figure 3):

• Data Collection: Raw logs were harvested from 20 public Chinese suicide-related Telegram groups (Jan 2020 – June 2025). Data was anonymized using SHA-256 hashing to remove user/group IDs.
• Signal Word Extraction: A hierarchical lexicon of 3,508 signal words was constructed, covering both explicit terms (e.g., direct suicide mentions) and implicit signals (e.g., internet slang, homophones, metaphors). An LLM (GLM4-32B) assisted in extracting implicit candidates, which were then validated by psychology experts.
• Context Expansion: A bidirectional context expansion algorithm was applied to suspicious messages. Centered on a seed message, the algorithm expands backward and forward based on temporal gaps (MTS), segment size (SS), and span (STS) constraints. It also retrieves referenced historical messages via reply links to preserve conversational coherence.
• Annotation: A rigorous two-stage process involving pilot annotation and main annotation was used.
  • Taxonomy: A six-class risk taxonomy was defined, adapted from the Columbia-Suicide Severity Rating Scale (C-SSRS):
    1. Irrelevant (No risk)
    2. Depressive Emotion
    3. Suicidal Ideation
    4. Suicide Method/Behavior
    5. Suicide Attempt
    6. Suicide Induction/Verbal Aggression
  • Quality Control: Human experts and a multi-model voting system (17 LLMs) were used to generate "silver" labels. Consensus was required ($\rho \ge 2/3$) for acceptance, with human adjudication for disagreements.

2. Evaluation Framework

The authors evaluated over 40 models, including proprietary API-based LLMs (e.g., GPT-5, Gemini, Kimi) and open-weight models (e.g., Llama, Qwen, DeepSeek), alongside PLMs.

• Scenarios: Zero-shot, Few-shot (with expert examples), Few-shot with masked keywords, and Few-shot with removed context.
• Partial Disclosure: Experiments were conducted where only the first $\tau \in \{30, 50, 70, 90, 100\}\%$ of a conversation was visible to simulate real-time streaming detection.
• Fine-tuning: Selected models were fine-tuned on the dataset to assess performance gains.

Key Contributions

1. SuiChat-CN Dataset: A benchmark containing 13,312 contextual segments derived from 258,228 raw messages across 1,406 users. It is the first dataset to focus specifically on the multi-party, context-dependent nature of suicide risk in Chinese group chats.
2. Construction Pipeline: A novel methodology combining signal-word discovery, expert validation, and bidirectional context expansion to transform fragmented chat logs into coherent risk assessment units.
3. Comprehensive Benchmarking: Extensive evaluation of PLMs and 40+ LLMs across diverse settings (prompting, ablation, fine-tuning, and partial context) to establish baselines for this specific domain.
4. Taxonomy Design: A six-class risk taxonomy that balances clinical interpretability with the practical needs of group-chat monitoring, covering the full spectrum from irrelevant content to active inducement.

Key Results

• Context is Critical: Removing conversational context or masking keywords universally degraded model performance. Models relying solely on isolated messages failed to capture implicit risk signals.
• Model Performance:
  • Proprietary models (e.g., Gemini-2.5-Pro, Seed-OSS-36B, Kimi-K2) generally outperformed open-weight models, achieving F1-scores above 70% in few-shot settings.
  • Among open-weight models, DeepSeek-R1 and Llama-3.3-70B approached the performance of mid-tier proprietary systems.
  • Fine-tuning significantly improved open-weight models (e.g., Llama-3-8B saw a +59.4 F1 point gain), narrowing the gap with proprietary systems, though it could not fully compensate for the lack of context.
• Partial Disclosure Findings:
  • High-risk cues (specifically Suicide Attempt, Class 4) were often delayed in the conversation. Models struggled to detect these risks when only the first 30% of the dialogue was visible.
  • Accuracy for high-risk classes improved significantly as more context was revealed (e.g., L4 accuracy for some models jumped from ~40% at 30% disclosure to >80% at 100%).
  • This challenges the assumption that urgent risk cues always appear at the beginning of a dialogue; in group chats, evidence is often distributed across turns.
• Implicit Signals: Analysis showed that implicit signals appear 2.3x more frequently than explicit signals in high-risk samples, highlighting the necessity of modeling culturally specific and evolving risk expressions.

Significance and Claims

The paper claims that SuiChat-CN addresses a critical gap in suicide risk assessment by shifting the focus from isolated posts to contextual, multi-party conversations.

• Realism: The dataset reflects the actual dynamics of instant messaging, where risk is often implicit and distributed.
• Necessity of Context: The results demonstrate that reliable early detection in group chats requires modeling conversational history, not just keyword matching.
• Ethical Stance: Due to the sensitivity of the data, the dataset is not publicly released. It is available only to accredited mental health and suicide-prevention research institutions under strict data-use agreements to ensure responsible use and protect user privacy.
• Future Direction: The work underscores the need for systems capable of continuous monitoring and long-context modeling to effectively identify delayed or distributed risk signals in real-world IM scenarios.

The authors emphasize that while current models show promise, the challenges of early detection in multi-party settings remain significant, particularly for the most severe risk categories which often require accumulated conversational evidence to be correctly identified.