Technical Summary: The MADRS Pipeline

Problem Definition

Depression remains a major mental health disorder lacking definitive biological markers, necessitating reliance on structured clinical interviews for diagnosis and clinical trial assessment. The Montgomery–Åsberg Depression Rating Scale (MADRS) is the standard instrument for quantifying symptom severity in these trials, typically administered via the Structured Interview Guide for the MADRS (SIGMA). However, the current process is heavily dependent on clinician expertise and judgment, introducing variability that can compromise trial outcomes (e.g., distinguishing drug efficacy from placebo).

While automated depression detection has advanced using Natural Language Processing (NLP), existing solutions primarily focus on:

1. Non-clinical data: Analyzing online text or social media rather than standardized clinical interviews.
2. Binary classification: Determining the presence of depression rather than quantifying specific symptom severity.
3. Lack of interpretability: Offering limited insight into individual MADRS items.

There is a critical gap in automated support for the specific, structured environment of clinical trials where interviews follow SIGMA guidelines and require granular, item-level severity scoring (0–6) for ten distinct symptoms.

Methodology: The MADRS Pipeline

The authors propose an end-to-end Large Language Model (LLM) pipeline designed to support clinicians by processing audio interviews and generating interpretable assessments aligned with MADRS and SIGMA standards. The pipeline is implemented as a directed acyclic graph comprising four sequential components:

1. Interview Transcription

The system converts raw audio recordings of clinician–patient interviews into structured text.

• Technique: Utilizes pyannote.audio for voice activity detection and speaker diarization, followed by the Whisper Medium ASR model for transcription.
• Output: A chronological sequence of clinician and patient utterances.

2. Transcript Segmentation

This component aligns the continuous transcript with the specific structure of the MADRS scale.

• Technique: Employs a zero-shot prompting strategy using GPT-4.1. The model receives the full transcript and specific SIGMA administration guidelines for each of the ten MADRS items.
• Process: The model identifies and extracts the specific utterance ranges (start and end indices) relevant to each symptom item, decomposing the interview into ten symptom-specific segments ($S = s_1, \dots, s_{10}$).

3. MADRS Assessment (Severity Estimation)

The core component estimates the severity rating (0–6) for each of the ten MADRS items based on the segmented text.

• Architecture: A Transformer encoder (specifically fine-tuned RoBERTa) with a shared encoder and ten item-specific prediction heads.
• Training Objective: Multi-task ordinal regression. The model treats the 0–6 scale as a sequence of binary threshold decisions ($y > k$ for $k \in \{0, \dots, 5\}$), explicitly modeling the ordered nature of the ratings.
• Input: Each segment is processed independently to predict the ordinal rating.

4. Quality Assessment

This component supports clinical trial monitoring by identifying potentially non-compliant or unreliable clinical ratings.

• Mechanism: Compares the clinical rater's assigned scores ($\hat{r}$) against reference scores ($r$). Reference scores can be provided by a human reviewer team or generated by the pipeline itself (using the fine-tuned encoder).
• Algorithm: A Random Forest binary classifier predicts a quality label ($q \in \{\text{Compliant, Non-compliant}\}$).
• Features: Input features include the absolute differences ($|r_i - \hat{r}_i|$) for each item, along with total and mean absolute differences.
• Noise Handling: When using pipeline-generated scores as references, the system applies a noise quantile threshold (80th percentile of training errors) to filter out expected model inaccuracies, ensuring only significant discrepancies are flagged as rater non-compliance.

Key Contributions

1. Specialized Pipeline for Clinical Trials: Unlike prior work focusing on general text or binary detection, this is the first computational solution specifically designed for depression clinical trials adhering to SIGMA guidelines.
2. Granular Symptom Analysis: The pipeline moves beyond binary classification to estimate severity for all ten individual MADRS items, providing interpretable support for specific symptoms.
3. End-to-End Audio Processing: The system processes raw audio directly, handling speaker diarization and segmentation without requiring manual transcript preparation.
4. Quality Monitoring: It introduces a mechanism to detect rating inconsistencies, supporting the validation of clinical rater performance in trials.
5. Empirical Validation on Real Data: The system is validated on a dataset of approximately 1,602 real-world clinical interviews (approx. 16,000 expert-rated instances), a scale and domain rarely seen in public NLP depression research due to data confidentiality.

Experimental Results

The pipeline was evaluated on a dataset split into Train, Dev, and Test sets, with ground truth provided by a reviewer team.

• Transcription & Segmentation:
  • Transcription achieved high semantic similarity (average cosine similarity of 0.814) with manual references.
  • Segmentation using GPT-4.1 achieved a weighted F1 score of 0.983, demonstrating robust alignment with SIGMA sections.
• MADRS Assessment:
  • The fine-tuned RoBERTa model achieved a Spearman correlation of 0.867 with expert ratings on total MADRS scores.
  • It achieved a Mean Absolute Error (MAE) of 2.956 and an Accuracy@1 (predictions within ±1 point of expert rating) of 0.895.
  • Performance varied by item; notably, RoBERTa outperformed decoder LLMs (like GPT-4.1) on "Suicidal Thoughts" (correlation 0.812 vs. 0.588), potentially due to safety guardrails in LLMs affecting conservative predictions on sensitive topics.
• Quality Assessment:
  • When using human reviewer scores as the reference, the Random Forest classifier achieved a Matthews Correlation Coefficient (MCC) of 0.704.
  • When using pipeline-generated scores as the reference, performance dropped (MCC 0.123 for RoBERTa), indicating that while the pipeline can flag discrepancies, human reviewer scores remain the gold standard for quality control.

Significance and Claims

The authors position the MADRS Pipeline not as a replacement for clinical judgment, but as a supportive tool to enhance consistency, scalability, and interpretability in psychiatric assessment workflows.

• Clinical Utility: The system provides interpretable, item-level severity estimates that can help clinicians verify their ratings or identify potential inconsistencies in multi-site trials.
• Scalability: By automating the transcription and initial scoring process, the pipeline offers a method to scale expert-level assessment support across large clinical trials.
• Limitations: The authors explicitly acknowledge limitations, including:
  • Modality Mismatch: The system relies solely on text transcripts, missing paralinguistic cues (prosody, facial expressions) used by human clinicians.
  • Transcription Dependency: Performance is inherently tied to the quality of the ASR output; errors in transcription can propagate to downstream assessments.
  • Data Imbalance: The quality assessment dataset is highly imbalanced (few non-compliant cases), reflecting real-world conditions but posing training challenges.
  • Confidentiality: Due to the sensitive nature of clinical trial data, the underlying datasets and trained models cannot be publicly released, though configurations and synthetic examples are provided.

In conclusion, the paper demonstrates that a structured, multi-component LLM pipeline can effectively support the assessment of depression in clinical trials, achieving strong agreement with human experts while offering a novel approach to quality monitoring in standardized psychiatric interviews.