A Benchmark Suite of Reddit-Derived Datasets for Mental Health Detection

This paper introduces a unified benchmark suite of four high-quality, human-verified Reddit datasets designed to facilitate reproducible and comparable research in mental health detection tasks, such as suicidal ideation and various mental disorder classifications.

The Gist

The "Mental Health Compass": Making Sense of the Digital SOS

Imagine you are a lifeguard at a massive, crowded beach. Thousands of people are splashing around, playing, and talking. Suddenly, you notice someone struggling under the water. You want to help, but how do you tell the difference between someone just diving deep for fun and someone actually drowning?

In the digital world, the "beach" is the internet (specifically sites like Reddit), and the "swimmers" are millions of people posting their thoughts. Some people are just sharing stories, but others are quietly sending out "digital SOS signals"—messages that indicate they are struggling with depression, bipolar disorder, or even thoughts of suicide.

The Problem: A Messy Toolbox
Right now, researchers trying to build "digital lifeguards" (AI programs that can detect mental health crises) are struggling. It’s like trying to build a high-tech rescue boat, but instead of having a standardized manual, every scientist is using a different, messy pile of random notes. One person uses a tiny bit of data; another uses data that is confusing or poorly labeled. Because everyone is using different "tools," it’s impossible to tell whose "rescue boat" actually works best.

The Solution: The Ultimate Training Manual
This paper introduces a Benchmark Suite. Think of this as a Gold-Standard Training Manual for AI.

Instead of scattered notes, the researchers have gathered four massive, highly organized "practice exams" for AI to study. These exams cover four different levels of difficulty:

1. The Emergency Alert: Detecting if someone is in immediate danger of suicide.
2. The General Check-up: Identifying if someone is generally struggling with mental health.
3. The Specialist Check-up: Specifically spotting signs of Bipolar Disorder.
4. The Deep Dive: A complex test where the AI has to distinguish between many different conditions (like ADHD, Anxiety, or PTSD).

How do we know the manual is accurate?
The researchers didn't just guess. They acted like "detectives" and "editors":

• Linguistic Fingerprints: They looked for "fingerprints" in the text. For example, they found that people in crisis often use more "inward-looking" words (like "I" and "me") and more intense emotional words, whereas people talking about general topics use more "outward-looking" words (like links to news or facts).
• The Double-Check: They had humans review the data to make sure the labels were correct. It’s like having two expert doctors look at the same X-ray to make sure they both see the same thing. They agreed almost perfectly, meaning the "manual" is incredibly reliable.

Why does this matter?
By providing this unified "Benchmark Suite," the researchers are giving the world a common playground.

Now, when a scientist in Japan builds a new AI, and a scientist in Brazil builds another, they can both test their models on the exact same exams. This allows us to finally see which AI is truly the best at spotting a cry for help.

The Big Picture
Ultimately, this isn't just about math or code; it’s about building better safety nets. By standardizing how AI learns to "read" emotional distress, we are moving closer to a world where technology can act as a silent, watchful guardian, helping to connect people in need with the support they deserve.

Technical Summary

Technical Summary: A Benchmark Suite of Reddit-Derived Datasets for Mental Health Detection

Problem Statement

The field of Natural Language Processing (NLP) for mental health is currently hindered by a lack of standardized, high-quality, and well-validated datasets. While online support groups (specifically on Reddit) offer a wealth of data, existing research tends to produce "task-specific" corpora that are not consolidated into widely available resources. This fragmentation leads to several critical issues:

• Poor Reproducibility: Researchers cannot easily replicate or validate findings across different studies.
• Lack of Comparability: Without a unified benchmark, it is difficult to perform fair cross-task comparisons or evaluate how models generalize across different mental health conditions.
• Methodological Bottlenecks: The absence of "gold-standard" datasets prevents the advancement of complex modeling techniques like multi-task learning.

Methodology

The authors propose a unified benchmark suite consisting of four distinct but complementary datasets derived from Reddit. The methodology for constructing these datasets emphasizes linguistic rigor and human validation:

1. Dataset Composition:

   • Suicidal Ideation Detection: Binary classification (suicidal vs. non-suicidal) using posts from r/SuicideWatch and various control subreddits.
   • Bipolar Disorder Detection: Binary classification (bipolar vs. non-bipolar) using r/bipolar and diverse mental health/control subreddits.
   • General Mental Disorder Detection: Binary classification (mental disorder vs. control) using a wide array of mental health subreddits and non-health control groups.
   • Multi-Class Mental Disorder Classification: Seven-class classification (ADHD, Anxiety, Bipolar, CPTSD, Depression, Schizophrenia, and Control).

2. Data Integrity & Filtering: To prevent data contamination, the authors applied strict filtering guidelines (following Cohan et al.) to eliminate duplicate users who might post in both mental health and control subreddits.

3. Validation Techniques:

   • Linguistic Analysis: The authors used the TextRank algorithm to identify key semantic markers and performed Part-of-Speech (POS) distribution analysis to characterize the stylistic differences between classes (e.g., higher pronoun and verb usage in mental health posts).
   • Statistical Validation: They utilized Mann–Whitney U tests to validate sentiment variance and Jensen–Shannon (JS) divergence to measure lexical separation between different disorder classes.
   • Human-in-the-loop Verification: Every dataset underwent independent human annotation by multiple coders. Reliability was quantified using Cohen’s $\kappa$, ensuring inter-annotator agreement consistently exceeded the 0.8 threshold (indicating "almost perfect" agreement).

Key Contributions

• Resource Consolidation: The integration of four previously isolated datasets into a single, cohesive benchmark suite available via Zenodo.
• Empirical Validation Framework: A multi-layered validation approach combining machine-led linguistic analysis (TextRank, JS divergence) with human judgmental verification.
• Standardization: Providing a foundation for the research community to conduct multi-task learning and standardized benchmarking in mental health NLP.

Results

The authors validated the "learnability" and quality of the datasets by reporting performance metrics from prior studies using state-of-the-art (SOTA) models:

• High Discriminative Power: Transformer-based models (e.g., RoBERTa, BERT, DistilBERT) and contextualized recurrent models (e.g., BERT+LSTM) achieved exceptionally high F1 scores.
• Performance Benchmarks:
  • Suicidal Ideation: F1 up to 93.14%.
  • Bipolar Disorder: F1 $\approx$ 98%.
  • General Mental Disorder: F1 up to 99.54% (hold-out) and 95.96% (external).
  • Multi-Class Classification: F1 ranging from 88.03% to 99.20%.
• Linguistic Distinction: The results confirmed that mental health posts are statistically distinct from control posts, characterized by longer text, higher emotional intensity, and more frequent use of first-person pronouns and verbs.

Significance

This work shifts the paradigm of mental health NLP from isolated, small-scale experiments to a structured, benchmark-driven discipline. By providing a heterogeneous and reliable resource, the authors enable:

1. Multi-Task Learning: Models can now be trained to recognize overlapping linguistic markers across different psychological conditions.
2. Improved Generalizability: The inclusion of "external sets" in the datasets allows researchers to test how well models perform on unseen disorders.
3. Scientific Rigor: The suite provides a "level playing field" for model comparison, facilitating more robust and reproducible computational mental health research.