💬 NLP

Depression Risk Assessment in Social Media via Large Language Models

This paper proposes a cost-effective, scalable system using Large Language Models to assess depression risk in Reddit posts via multi-label emotion classification and a weighted severity index, demonstrating competitive performance against fine-tuned models and revealing stable risk profiles across different online communities.

Original authors: Giorgia Gulino, Manuel Petrucci

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giorgia Gulino, Manuel Petrucci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, bustling town square where millions of people are constantly talking, sharing their thoughts, and venting their feelings. For years, mental health experts have known that if you listen closely to what people say in this square, you can often hear the early signs of a storm brewing in their minds—specifically, depression.

This paper is like a new, super-smart digital listening device designed to scan that town square, not to spy on people, but to help identify who might need a gentle nudge toward professional help.

Here is the breakdown of how this system works, using simple analogies:

1. The Problem: Finding the Needle in the Haystack

Depression is a silent killer; many people suffer without anyone knowing. Traditional doctors can't be everywhere at once to check on everyone. Meanwhile, people are posting thousands of messages every day on Reddit (a popular forum site) about their struggles.

The Old Way: Previously, researchers built "specialized robots" (AI models) that had to be taught specifically how to spot depression. It was like hiring a dog trainer to teach a dog to find only one specific type of weed in a garden. It worked well, but it was expensive, slow, and the dog couldn't handle new types of weeds if the garden changed.
The New Way: This paper uses Large Language Models (LLMs). Think of these as super-readers who have read almost every book, article, and conversation in human history. They didn't need to be "trained" specifically on depression; they already understand human emotion because they've seen it everywhere.

2. The Tool: The "Emotion Thermometer"

The researchers didn't just ask the AI, "Is this person depressed?" That's too vague. Instead, they gave the AI a specific checklist of 8 emotional "symptoms" (like sadness, hopelessness, feeling worthless, or thoughts of suicide).

They created a Weighted Severity Index, which is like a thermometer for emotional pain:

Low Heat (Score 0–1): The person is fine.
Warm (Score 2–4): Mild sadness or stress.
Hot (Score 5–6): Moderate depression.
On Fire (Score 7+): Severe depression or high risk.

The "Weighting" Trick:
Not all symptoms are treated equally. The AI knows that while "sadness" is a warning sign, "suicide intent" is a siren alarm. So, if the AI detects "sadness," it adds 1 point to the score. If it detects "suicide intent," it adds 3 points. This ensures that the most dangerous cases get the highest priority.

3. The Experiment: The "Zero-Shot" Magic

The researchers tested this system in two ways:

The Classroom Test: They gave the AI a set of 6,000 posts that humans had already labeled. The AI had to guess the labels without any prior study.
- Result: The AI (specifically a model called gemma3:27b) scored almost as high as the specialized "dog trainer" models, but it did it without any training. It just used its general knowledge of language.
The Real World Test: They let the AI loose on 470,000 real Reddit posts from 2024 to 2025.
- Result: The system worked beautifully. It could tell the difference between the "Anxiety" community (where people are worried but not necessarily hopeless) and the "Depression" community (where hopelessness was the dominant theme).

4. What Did They Find?

Consistency: The "temperature" of these communities stayed surprisingly stable over time. The AI didn't get confused by random noise; it saw the real patterns.
The "High-Risk" Signal: When the AI flagged a post as "High Risk" (Score > 7), it wasn't a fluke. Those posts almost always contained a dangerous mix of emotions: deep sadness, feeling worthless, and thoughts of suicide all at once.
Community Differences: The "Depression" subreddit was like a room where everyone was speaking a language of heavy despair. The "Anxiety" subreddit was more like a room where people were pacing nervously. The AI understood the difference perfectly.

5. Why This Matters (The Big Picture)

Think of this system as a triage nurse in a giant emergency room.

It doesn't replace the doctor (the clinical psychologist).
It doesn't diagnose patients.
It does the heavy lifting: It scans thousands of messages to find the few people who are in immediate danger, so human professionals can focus their time on those who need it most.

The Best Part?
Because these models can run on standard computers (locally) rather than requiring expensive, private cloud servers, this tool is cheap and scalable. It's like giving every community center a free, smart thermometer that can help spot a crisis before it becomes a tragedy.

Summary

This paper proves that we don't need to build a custom robot for every mental health problem. We can use a general-purpose "super-reader" AI, give it a simple checklist of emotions, and let it act as a digital safety net to catch people who are falling, all while respecting their privacy and keeping costs low.

1. Problem Statement

Depression is a leading cause of global disability, yet it remains frequently underdiagnosed and undertreated, particularly among young adults. While social media platforms like Reddit offer a rich source of naturalistic linguistic data for monitoring psychological well-being, traditional automated detection methods face significant challenges:

Data Dependency: Fine-tuned models require large, domain-specific labeled datasets, which are expensive to create and risk becoming obsolete as language evolves.
Scalability & Cost: Deploying large-scale monitoring systems often relies on proprietary APIs (e.g., GPT-4) which are costly, or smaller models that lack the nuance to detect complex emotional states.
Clinical Nuance: Existing methods often treat depression as a binary classification or fail to weight specific high-risk indicators (like suicide intent) appropriately against general distress signals.

The authors propose a solution using Large Language Models (LLMs) in a zero-shot setting to assess depression risk without domain-specific training, aiming for a cost-effective, scalable, and clinically grounded approach.

2. Methodology

The proposed system consists of three core components: a weighted severity index, a prompt engineering strategy, and a two-phase evaluation framework.

A. Weighted Depressive Severity Index ( $S$ )

Instead of a simple binary classification, the authors define a composite severity score based on eight clinically relevant emotion categories derived from the DepressionEmo dataset:

Anger
Cognitive Dysfunction
Emptiness
Hopelessness
Loneliness
Sadness
Suicide Intent
Worthlessness

The system assigns a weight ( $w_i$ ) to each emotion based on clinical relevance (inspired by PHQ-9 and BDI-II scales):

Weight 1 (Distress indicators): Anger, Cognitive Dysfunction, Emptiness, Loneliness, Sadness.
Weight 2 (Advanced suffering): Hopelessness, Worthlessness.
Weight 3 (Critical risk): Suicide Intent.

The total severity score $S$ is calculated as:
$S = \sum (w_i \times \text{presence}_i)$
Where $\text{presence}_i$ is 1 if the emotion is detected, 0 otherwise. The theoretical maximum is 13.

Severity Levels: $S=0\text{--}1$ (Minimal), $S=2\text{--}4$ (Mild), $S=5\text{--}6$ (Moderate), $S \ge 7$ (Severe/High Alert).

B. Prompt Engineering

The system utilizes zero-shot prompting to guide LLMs. The prompt instructs the model to:

Analyze the sentiment of a Reddit comment.
Classify the presence of the eight specific emotions (True/False).
Output the result as a structured JSON object.
Crucially, the prompt includes instructions to compute the weighted severity score ( $S$ ) directly within the model's output based on the assigned weights.

C. Experimental Setup

Controlled Benchmark: Evaluated on the DepressionEmo dataset (~6,000 annotated posts) using an 80/20 train/validation split. The LLMs were tested exclusively on the validation set without any fine-tuning.
In-the-Wild Analysis: Applied to 469,692 Reddit posts collected from four subreddits ($r/anxiety$, $r/depression$, $r/depression\_partners$ , $r/mentalhealth$) between January 2024 and May 2025.
Models Tested: Nine locally hosted LLMs (ranging from 0.6B to 27B parameters, including Gemma, Phi, Llama, Mistral, and Qwen families) were compared against fine-tuned baselines (SVM, LightGBM, XGBoost, GAN-BERT, BERT, BART).

3. Key Contributions

Clinical Severity Index: Introduction of a weighted scoring system that differentiates between general distress and high-criticality indicators (e.g., suicide intent), mapping directly to clinical severity levels.
Zero-Shot LLM Methodology: A prompt engineering framework that enables LLMs to perform multi-label emotion classification and score computation without any domain-specific fine-tuning.
Comprehensive Evaluation: A systematic comparison of nine local LLMs against state-of-the-art fine-tuned models, demonstrating that large local models can compete with specialized architectures.
Large-Scale Longitudinal Study: The first analysis of depression risk trends across major mental health communities over a 17-month period (2024–2025), revealing stable risk profiles and community-specific emotional signatures.

4. Results

Performance on Controlled Benchmark

Top Performer: Gemma3:27b achieved the best balance among LLMs with a Micro-F1 of 0.75 and Macro-F1 of 0.70.
Comparison: These results are competitive with purpose-built fine-tuned models. For instance, the fine-tuned BART model achieved Micro-F1 = 0.80 and Macro-F1 = 0.76. The gap between the best zero-shot LLM and the best fine-tuned model is only ~0.05 F1 points.
Model Behavior:
- Smaller models (e.g., Qwen3-0.6B) showed high precision but low recall (conservative).
- Specialized variants (e.g., Samantha-Mistral) showed high recall but low precision.
- Larger models (14B–27B) provided the best precision-recall balance.

In-the-Wild Analysis Findings

Emotional Co-occurrence: A correlation matrix revealed that "core" depressive emotions (Hopelessness, Worthlessness, Sadness, Emptiness) are highly correlated ( $\rho \approx 0.28\text{--}0.53$ ), supporting the hypothesis of a shared latent construct.
Community Differentiation:
- r/depression: Showed a structurally elevated risk profile with a mean score of 6.97 (median 7.00). Sadness and hopelessness were detected in >80% of posts.
- r/anxiety: Showed a low-risk profile with a mean score of 2.56 (median 2.00).
- High-Risk Posts: Posts with $S > 7$ were characterized by the systematic convergence of multiple critical indicators, particularly Suicide Intent and Worthlessness.
Temporal Stability: Risk scores remained stable throughout 2024–2025, with no erratic oscillations, validating the system's ability to detect structured linguistic variation rather than noise. A moderate upward trend in risk scores was observed in early 2025.

5. Significance and Conclusion

Feasibility of Local Deployment: The study proves that medium-to-large LLMs (up to 27B parameters) can run locally on affordable hardware to perform complex psychological monitoring tasks with performance comparable to fine-tuned models.
Cost-Effectiveness: By avoiding the need for domain-specific labeled data and proprietary APIs, this approach offers a scalable, low-cost solution for large-scale triage and monitoring.
Clinical Utility: The system is designed as a support tool, not a replacement for clinical judgment. It serves as a "first-pass filter" to identify high-risk posts (e.g., those with $S \ge 7$ ) for human review.
Future Directions: The authors suggest future work should focus on extending the system to non-English languages, integrating behavioral signals (post frequency/timing), and validating thresholds against certified clinical diagnoses.

Limitations: The study relies on public text data (self-selection bias), lacks individual clinical variables, and uses LLMs as a simplification of complex psychological processes. Ethical considerations regarding privacy and informed consent remain critical for real-world deployment.