CNSocialDepress: A Chinese Social Media Dataset for Depression Risk Detection and Structured Analysis

1. Problem Statement

Depression is a critical global public health issue, with approximately 95 million affected individuals in China. While machine learning (ML) and natural language processing (NLP) are increasingly used for automatic depression risk detection, existing solutions face significant limitations:

• Data Scarcity & Language Gap: Most high-quality datasets are English-centric (e.g., Twitter, Reddit). Chinese resources are scarce and often rely on binary classification (depressed vs. non-depressed) without nuanced insights.
• Lack of Structured Analysis: Existing datasets typically lack structured psychological attributes or professional validation, making them unsuitable for fine-grained clinical interpretation or explainable AI.
• Annotation Quality: Many existing datasets rely on automated metrics or non-expert annotations, raising concerns about domain-specific reliability.
• Generative Model Needs: As Large Language Models (LLMs) are adopted for mental health, there is a lack of datasets that combine risk labels with structured, interpretable psychological profiles to support reasoning and summarization tasks.

2. Methodology

The authors propose CNSocialDepress (CNSD), a benchmark dataset and an automated generation pipeline to address these gaps.

A. Dataset Construction (CNSD Gold)

• Source: Derived from the SWDD dataset (Sina Weibo Depression Dataset), selecting 233 users (116 positive/depressed, 117 negative/non-depressed).
• Annotation Protocol:
  • Expert-in-the-Loop: Annotations were drafted and validated by certified mental health professionals (psychologists).
  • Blinding: Annotators were blinded to the original binary labels to ensure independent judgment.
  • Six-Dimensional Framework: Based on DSM-5 criteria and PHQ-9, the annotation schema covers six specific dimensions:
    1. Depressive Psychological State (Primary): Self-worth, guilt, suicidal ideation.
    2. Medical Expressions (Primary): Medication, clinical diagnoses.
    3. Clinical Symptoms (Primary): Sleep, appetite, fatigue, pain.
    4. Negative Emotions (Secondary): Sadness, anxiety, loneliness.
    5. External Causes (Secondary): Relationship issues, family conflict, stressors.
    6. Language Use Patterns (Secondary): Negation, self-deprecation, rhetorical questions.
• Output: Each user entry includes a binary risk label and a structured summary with specific text spans (indices) and justifications for each dimension.
• Scale: 44,178 posts total; 10,306 expert-annotated depression-related segments.

B. Automated Dataset Generation Pipeline (CNSD Silver)

To scale annotation efforts, the authors developed a two-module pipeline to generate a "Silver" dataset:

1. Module I (Dimension-Wise Labeling): A mid-sized model (Qwen2.5-14B) is fine-tuned on the CNSD Gold data to automatically assign posts to the six dimensions. The training data is balanced via downsampling to the minimum dimension count (442 posts).
2. Module II (Verification & Summarization): The fine-tuned model labels all posts. A larger reasoning model (DeepSeek-R1-671B) verifies the labels and evidence case-by-case. Finally, the system generates structured summaries aligned with the Gold standard schema using few-shot prompting.

3. Key Contributions

1. CNSocialDepress (CNSD): The first publicly available Chinese dataset for depression risk detection that combines binary labels with structured, multidimensional psychological analyses annotated by professionals.
2. Expert-Validated Protocol: A rigorous annotation framework involving psychology experts, iterative guideline refinement, and cross-validation, ensuring high domain relevance.
3. Automated Generation Pipeline: A novel pipeline that leverages fine-tuning and LLM verification to generate high-quality "Silver" data, significantly reducing the manual effort required for large-scale structured annotation.
4. Comprehensive Benchmarking: Extensive evaluation of the dataset across multiple NLP tasks, including classification, structured analysis generation, summarization, and LLM fine-tuning.

4. Experimental Results

The authors conducted experiments to evaluate the generation pipeline and the utility of the dataset for LLMs.

• Data Generation Quality:

  • Models fine-tuned on CNSD Gold and Silver significantly outperformed baseline models (including GPT-4o and DeepSeek-R1-671B) and few-shot prompting strategies.
  • Metrics: The pipeline achieved the highest scores in BERTScore (0.791), ROUGE-1 (0.478), and BLEU (0.288) compared to few-shot baselines.
  • Human Evaluation: Fine-tuned models showed substantial improvements in accuracy (+42.9% for Gold), content coverage (+58.0%), and reduced hallucinations compared to the base model.

• Classification Performance:

  • Qwen2.5-14B (Fine-tuned on Silver) achieved the best classification performance with 94.4% accuracy and 94.1% F1-score, outperforming both the Gold-fine-tuned model and larger models like GPT-4o (91.7% accuracy) and DeepSeek-R1-671B (86.1% accuracy). This suggests the generated Silver data is of high quality and effective for training.

• Structured Summarization:

  • The dataset serves as a robust benchmark for user-level depression summarization. DeepSeek-R1-671B achieved the highest ROUGE-1 (0.454) in summarization tasks, demonstrating the dataset's utility for training generative models to produce clinically relevant insights.

5. Significance and Impact

• Clinical & Research Utility: CNSD bridges the gap between raw social media data and clinical needs by providing interpretable, structured psychological profiles. This supports downstream tasks like early risk identification and intervention planning.
• Scalability: The proposed automated pipeline offers a replicable method for creating structured mental health datasets in other languages or domains, addressing the bottleneck of expert annotation.
• Chinese Context: By focusing on Chinese social media (Weibo), the dataset captures cultural and linguistic nuances specific to East Asian expressions of depression, which are often missed in Western-centric datasets.
• Ethical Considerations: The authors emphasize that the dataset is for research and assistive tools only, not for clinical diagnosis. They implemented strict de-identification and privacy protection measures compliant with GDPR.

In conclusion, CNSocialDepress represents a significant advancement in computational mental health, providing a high-quality, structured resource that enables more interpretable and effective depression risk detection for Chinese-speaking populations.