Leveraging LLMs for Translating and Classifying Mental Health Data

Technical Summary: Leveraging LLMs for Translating and Classifying Mental Health Data

Problem Statement
While Large Language Models (LLMs) offer potential for the early identification of mental health conditions through linguistic markers in user-generated text, their application remains heavily skewed toward English-language datasets. There is a significant gap in research regarding the performance of these models in low-resource languages and non-English contexts. Furthermore, the efficacy of using LLMs to automatically translate mental health data from English to another language and subsequently classify it remains underexplored. This study addresses these gaps by investigating whether an LLM can accurately predict depression severity in English posts and whether this performance is maintained when the posts are automatically translated into Greek, a language lacking specific resources for this task.

Methodology
The authors propose a pipeline that utilizes a state-of-the-art multilingual LLM (GPT-3.5-turbo) for both translation and classification tasks.

• Dataset: The study utilizes the DEPSEVERITY dataset (Naseem et al., 2022), comprising 3,553 Reddit posts labeled with four levels of depression severity: Minimum, Mild, Moderate, and Severe. The dataset is noted to be highly imbalanced, with the majority of posts falling under the "Minimum" category.
• Procedure:
  1. Baseline (English): The LLM is prompted to classify the original English posts into the four severity levels using 0-shot learning.
  2. Translation: The same LLM translates the English posts into Greek.
  3. Target Classification (Greek): The translated Greek texts are fed back into the LLM with a prompt to predict the severity levels.
  4. Evaluation: Performance is measured using Precision, Recall, and F1-scores, comparing predicted classes against ground truth labels in both languages. The temperature parameter was set to 0 to ensure reproducibility.
• Prompting Strategy: Prior to the main experiments, the authors conducted preliminary prompting to understand the LLM's internal logic. The model identified patterns such as persistent negative emotions, self-criticism, and behavioral changes, adapting the severity levels to fit social media contexts (e.g., defining "Severe" as posts indicating self-harm risk or complete social withdrawal).

Key Results
The experimental results indicate that the LLM struggles to accurately distinguish between specific levels of depression severity in both languages, though performance varies by class and language.

• English Performance: The model achieved its highest F1-score for the "Minimum" class (0.25) and the "Severe" class (0.22). The overall macro F1-score was low at 0.17. The authors attribute this difficulty to the subtle distinctions between severity levels in user posts compared to other conditions.
• Greek Performance: After translation, performance generally declined across most classes.
  • The "Mild" class saw a slight drop (F1: 0.07 in English vs. 0.06 in Greek).
  • The "Minimum" class dropped significantly (F1: 0.25 in English vs. 0.14 in Greek).
  • The "Severe" class also dropped (F1: 0.22 in English vs. 0.20 in Greek).
  • Notably, the "Moderate" class showed an improvement in F1-score (0.17 in English vs. 0.23 in Greek).
• Error Analysis: The authors analyzed specific cases where the LLM misclassified anxiety or panic symptoms as depression. For instance, a post describing panic and anxiety was classified as "Severe" by the LLM, whereas the ground truth was "Minimum." The authors suggest the LLM may conflate general distress (panic/anxiety) with depressive severity, a nuance that ground truth labels distinguished.
• Cost: The experiments were conducted with a total API cost of less than $30, demonstrating that such evaluations can be performed without expensive GPU infrastructure or fine-tuning.

Significance and Claims
The paper claims that while LLMs show promise, they are not yet reliable enough to solely determine depression severity, particularly when relying on automatic translation for low-resource languages like Greek.

• Limitations of Automation: The study underscores that the performance of LLMs varies significantly across languages and severity classes. The authors emphasize that the current results are insufficient for direct patient diagnosis.
• Role of Human Supervision: Consistent with the findings of Stade et al. (2024), the authors assert that mental health diagnosis should never be left entirely to automatic systems. Human supervision remains crucial to avoid misdiagnosis.
• Potential Utility: Rather than replacing clinicians, the authors suggest this approach could be valuable for training mental health professionals, particularly in non-English speaking regions where resources are scarce.
• Future Needs: The work highlights the necessity for further research into culturally and linguistically sensitive approaches, better data collection for low-resource languages, and the development of more robust transfer learning techniques to handle the nuances of mental health terminology across different cultures.