Technical Summary: Fine-tuning LLMs for Passive Depression Severity Estimation from AI Mental Health Dialogue

Problem Statement

Depression is the leading cause of disability globally, yet early detection of symptom worsening remains challenging due to low completion rates of validated instruments like the Patient Health Questionnaire-9 (PHQ-9) in real-world settings. This introduces response bias and systematic missingness. While AI mental health platforms generate rich, transcribed dialogue capturing user experiences and emotional language, current architectures often prioritize counselor-focused interactions over user modeling. Existing research on predicting PHQ-9 scores from text is fragmented, relying on small samples (n=89–335), structured clinical interviews (e.g., DAIC-WOZ), or specific text types (diaries, clinical notes). Crucially, no prior work has attempted to predict full PHQ-9 severity scores (including suicidal ideation) from large-scale, naturalistic AI-therapy dialogue without structured clinical prompts.

Methodology

Data Collection and Preprocessing

The study utilized data from "Ash," an AI-powered mental health platform. The dataset comprised 3,953 users who completed a baseline PHQ-9 prior to their first conversation and exchanged at least 10 messages with the AI within the first seven days.

• Input: Conversation transcripts formatted as alternating user and assistant turns.
• Preprocessing: Messages were subsampled to a maximum of 300 tokens (evenly spaced) to preserve the temporal arc of the conversation.
• Distribution: The cohort was help-seeking, with 81.8% scoring above the clinical threshold (PHQ-9 ≥10), creating a significant class imbalance toward severe depression.

Model Architecture

The core model employs a Qwen3.5-27B backbone fine-tuned with a regression head.

• Input Processing: Transcripts are tokenized using the model's chat template with a maximum sequence length of 16,384 tokens.
• Training Mechanism: A LoRA adapter is trained for 5 epochs using Mean Squared Error (MSE) loss on the continuous PHQ-9 total score. The learning rate follows a cosine schedule with a 0.1 warm-up fraction and a batch size of 8 (via gradient accumulation).

Pseudo-Labeling Strategy (Semi-Supervised Learning)

To address data scarcity and class imbalance, the authors employed a four-stage iterative pseudo-labeling approach to expand the training set from 3,111 ground-truth users to 6,283:

1. Opus Pseudo-Labeling: Claude Opus 4.6 was prompted to estimate PHQ-9 scores for unlabeled conversations. To correct the dataset's skew toward severe cases, only users estimated to have scores below 10 were retained (605 users).
2. Intermediate Model Pseudo-Labeling (v1): A model trained on ground-truth plus Opus labels was used to generate labels for an additional 2,567 unlabeled users (retaining the full severity range).
3. Iterative Re-labeling (v2): The improved model from step 2 was used to re-label the same 2,567 users, yielding a refined dataset.
4. Ensembling: Four models trained on the final re-pseudo-labeled dataset were ensembled by averaging their predictions to mitigate noise and spurious signals.

Key Results

On a held-out test set of 842 users, the final ensemble model achieved the following performance metrics:

• Mean Absolute Error (MAE): 2.62
• Root Mean Squared Error (RMSE): 4.03
• Pearson Correlation (r): 0.803
• AUC at Clinical Threshold (PHQ-9 ≥10): 0.908

Cross-Severity Discrimination:
The model demonstrated robust discrimination across the entire clinical spectrum, achieving an AUC > 0.87 at every threshold from PHQ-9 ≥3 to PHQ-9 ≥24. This indicates the model can distinguish between mild, moderate, moderately severe, and severe depression, rather than merely detecting binary case status.

Error Analysis:

• Recall by Severity: The model correctly identified 70% of severe cases but only 22% of minimal cases. This reflects the training data's heavy skew toward high-severity users.
• Error Distribution: Most errors were adjacent-band misclassifications (within 1 severity band), with a within-band accuracy of 92%.
• Comparison: The model's Normalized MAE (NMAE) of 0.097 represents a 34–39% improvement over previous benchmarks on the DAIC-WOZ corpus (NMAE 0.148–0.160).

Key Contributions and Significance

1. Scale and Ecological Validity: This is the first large-scale study (n ≈ 4,000) to predict PHQ-9 scores from naturalistic, unguided AI-therapy dialogue, moving beyond the small, structured interview datasets (n < 300) that dominate prior literature.
2. Full-Spectrum Prediction: Unlike prior work that often omits Item 9 (suicidal ideation) or focuses on binary classification, this model predicts the full continuous PHQ-9 score (0–27), enabling graded clinical distinctions across all severity levels.
3. Passive Monitoring Capability: The approach demonstrates that depression severity can be inferred solely from conversation text without requiring users to complete self-report questionnaires. This enables passive, continuous symptom monitoring and early detection of deterioration in real-world AI mental health platforms.
4. Methodological Advancement: The study validates a semi-supervised pipeline using reasoning models (Claude Opus) and iterative self-labeling to effectively expand limited ground-truth datasets, achieving superior performance through ensemble averaging.

Limitations and Scope

The authors note that the dataset represents a self-selected, help-seeking population, which may limit generalizability to the broader population. The model's lower recall for minimal severity cases is a direct consequence of training on a cohort where 81.8% of users presented with clinical depression. The authors position this work as a foundational step toward scalable, passive mental health screening, acknowledging that further confirmation is needed for population-level generalization.