Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challenges

Technical Summary: Building Multilingual Datasets for Predicting Mental Health Severity through LLMs

1. Problem Statement

While Large Language Models (LLMs) show promise in healthcare, most research regarding mental health detection is confined to the English language. This creates a significant gap in providing equitable AI-driven support for non-English speaking populations. Furthermore, mental health diagnosis is a high-stakes task where linguistic nuances, cultural contexts, and the scarcity of low-resource medical datasets in non-English languages pose significant challenges for automated systems.

2. Methodology

The authors propose a novel framework to evaluate LLM performance in detecting the severity of mental health conditions across multiple languages using a translation-based approach.

• Datasets: The study utilizes two benchmark English-language datasets:
  • DEP-SEVERITY: Reddit posts categorized into four levels of depression (Minimum, Mild, Moderate, Severe).
  • C-SSRS: Data based on the Columbia-Suicide Severity Rating Scale, categorized into five classes (Supportive, Indicator, Ideation, Behavior, Attempt).
• Multilingual Adaptation: Instead of manual labeling, the researchers used LLMs to automatically translate the English source posts into six target languages: Turkish, French, Portuguese, German, Greek, and Finnish.
• Experimental Setup:
  • Models: GPT-3.5-turbo, GPT-4o-mini, and Llama-3.1.
  • Prompting Strategies: Evaluated using both 0-shot (no examples) and 1-shot (one example per class) learning.
  • Task: A classification task where the LLM is prompted to categorize the translated text into the predefined severity levels.
  • Back-translation: A specific experiment was conducted where translated text was translated back to English to assess the impact of the translation step on prediction accuracy.

3. Key Contributions

• Novel Multilingual Dataset: Creation of a new dataset derived from social media content, translated into seven languages (including English) to facilitate multilingual mental health NLP research.
• Evaluation Framework: A systematic methodology for assessing how LLMs handle mental health severity prediction when moving from a high-resource language (English) to various other language families (Romance, Germanic, Uralic, Hellenic, etc.).
• Error & Nuance Analysis: In-depth qualitative analysis of how translation and linguistic nuances lead to misclassifications.

4. Key Results

• Performance Variability: LLM performance is inconsistent across languages. For example, in the DEP-SEVERITY task, GPT-3.5 saw performance increases in French and German but significant drops in Turkish.
• Model Comparison: GPT-4o-mini demonstrated significantly higher stability and better F1 scores compared to GPT-3.5 and Llama-3.1.
• The "Translation Boost": Interestingly, the study found that translating text into a target language before prediction can sometimes improve performance (as seen in the back-translation experiments), suggesting that LLMs may occasionally process certain linguistic nuances more effectively in a target language than in the original English.
• Language-Specific Challenges: Turkish showed the poorest performance, which the authors attribute to the scarcity of digital Turkish medical and mental health resources.
• Complexity of Severity: Detecting "Ideation" and "Behavior" remains significantly more difficult for LLMs than detecting "Supportive" or "Minimum" states, due to the subtle linguistic markers required to distinguish between them.

5. Significance and Implications

• Clinical Caution: The paper issues a strong warning: LLMs should not be used as standalone diagnostic tools. The observed variability and potential for misdiagnosis (e.g., misclassifying a "Supportive" comment as an "Indicator" due to translation nuances) pose high risks in medical settings.
• Cost-Effectiveness: The proposed methodology of using LLMs for dataset expansion is highly cost-effective (the GPT-3.5 experiments cost less than $30), offering a scalable way to build inclusive datasets for low-resource languages.
• Future Research: The work highlights the need for "culturally-aware" algorithms that go beyond literal translation to capture the socio-linguistic context of mental health distress.