💬 NLP

Explainable Multimodal Depression Recognition in Clinical Interviews via PHQ-Aligned Symptom Summarization

This paper introduces Explain-MDRC, an explainable multimodal depression recognition framework that enhances clinical interpretability and performance by generating PHQ-8-aligned symptom summaries and integrating them with nonverbal cues through a novel dataset (Explain-DAIC) and a contrastive learning model (PhqCML).

Original authors: Wenjie Zheng, Qiming Xie, Jianfei Yu, Yang Wang, Lei Cao, Fei Wang, Shijin Wang, Rui Xia, Chengqing Zong

Published 2026-08-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenjie Zheng, Qiming Xie, Jianfei Yu, Yang Wang, Lei Cao, Fei Wang, Shijin Wang, Rui Xia, Chengqing Zong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Depression is a condition that affects hundreds of millions of people worldwide, yet a vast number of cases go unnoticed or undiagnosed. In a clinical setting, doctors do not rely on a single test to identify it; instead, they engage in a conversation, listening to what a patient says while also observing how they speak, their facial expressions, and their body language. They piece together these verbal and non-verbal clues to form a structured picture of the patient's mental state, often checking specific symptoms like sleep disturbances, loss of interest, or feelings of hopelessness against a standard checklist. For years, researchers have tried to teach computers to do the same thing by feeding them recordings of these interviews. However, most computer systems that attempt this task operate like a black box: they take in the audio and video and spit out a label, such as "depressed" or "not depressed," without showing their work. This lack of transparency makes it difficult for doctors to trust the technology or understand how the machine reached its conclusion, limiting its usefulness in real-world healthcare.

A team of researchers has now developed a new approach that changes how these systems think. Instead of asking a computer to jump straight to a diagnosis, they designed a framework that forces the machine to first act like a clinician by writing a structured summary of the patient's symptoms before making a final judgment. This system, which they call Explain-MDRC, starts by analyzing the text of a clinical interview, the tone of the voice, and the movements of the face. It then generates a written report that mirrors the way a human doctor thinks, breaking down the conversation into four specific parts: checking for a history of depression, assessing the specific symptoms mentioned, identifying possible causes for those symptoms, and suggesting a plan of action. Crucially, this summary is not just a vague description; it is tightly aligned with a standard medical questionnaire known as the PHQ-8, ensuring that the machine focuses on the exact same eight symptoms that doctors look for, such as trouble sleeping, low energy, or difficulty concentrating.

To train this system, the researchers created a new dataset based on existing recordings of clinical interviews. They had licensed mental health professionals read through the transcripts and write out these structured symptom summaries by hand, linking specific parts of the conversation to the medical criteria. This created a bridge between the raw data and the clinical reasoning process. They then taught a computer model to mimic this process. The model first learns to generate these summaries, but with a special twist: it is trained to ensure that the internal "understanding" of the summary matches the patient's actual symptom profile. If two patients have similar patterns of symptoms, the computer is taught to view their summaries as similar; if their symptoms are different, the computer learns to keep their summaries distinct. This step ensures that the machine's intermediate reasoning is grounded in clinical reality, not just in the words it happens to generate.

Once the machine has produced this structured summary, it combines the text with the acoustic and visual cues from the interview to make its final prediction about whether the patient is depressed. The results of this approach were striking. When tested on the new dataset, the system significantly outperformed previous methods that tried to guess the diagnosis directly without writing a summary first. It achieved a level of accuracy that was nearly seven percentage points higher than the best existing systems. More importantly, the summaries it generated were not just accurate; they were readable and useful. When three licensed psychiatrists reviewed the computer's summaries and compared them to those generated by a powerful general-purpose artificial intelligence, they consistently preferred the new system. The experts found that the summaries were more complete, more factually correct, and more concise, often capturing the subtle evidence in the conversation that other systems missed.

The study suggests that by forcing an artificial intelligence to explain its reasoning in a structured, clinically relevant way, the system not only becomes more trustworthy but also becomes better at its primary job. The researchers found that the act of generating the symptom summary actually helped the computer recognize depression more accurately, likely because it forced the model to focus on the most important details of the interview. While the system is currently a research prototype and not yet a tool for real-world diagnosis, it demonstrates a clear path forward. It shows that the future of AI in mental health may not lie in systems that simply guess an answer, but in tools that can walk a doctor through the evidence, step by step, just as a human colleague would.

Technical Summary: Explainable Multimodal Depression Recognition in Clinical Interviews via PHQ-Aligned Symptom Summarization

Problem Statement
Recent advances in Multimodal Depression Recognition for Clinical Interviews (MDRC) have demonstrated the potential of AI systems to integrate textual, acoustic, and facial cues for assessing depression severity. However, existing methods predominantly treat the task as a black-box classification problem, focusing on predictive accuracy while neglecting interpretability. This lack of transparency constrains reproducibility and hinders clinician review, as current systems fail to generate human-interpretable evidence that aligns with clinical reasoning processes. In standard clinical workflows, clinicians synthesize structured symptom summaries based on standardized scales (e.g., PHQ-8) and nonverbal cues to form assessment impressions. Current MDRC systems cannot replicate this intermediate reasoning step, creating a gap between recognition capability and clinician-reviewable evidence.

Methodology: Explain-MDRC and PhqCML
To address this, the authors introduce Explain-MDRC, a framework that explicitly simulates core aspects of clinical workflows by generating structured symptom summaries as an intermediate output before performing depression recognition.

Dataset Construction (Explain-DAIC):
The authors constructed Explain-DAIC, an extension of the widely used DAIC-WOZ dataset. Unlike conventional datasets that provide only session-level labels, Explain-DAIC includes manually annotated, PHQ-8–aligned structured symptom summaries. These summaries are generated by clinically trained professionals and include four components:
- Screening for a history of depression.
- Assessment of scale-listed symptoms (anchored to dialogue evidence).
- Evaluation of possible underlying causes.
- Formulation of action plans.
  The dataset retains the original DAIC-WOZ binary labels for recognition tasks while adding these structured, dialogue-grounded explanations.
Model Architecture (PhqCML):
The authors propose PhqCML (PHQ-aware Contrastive Multimodal Learning), a two-stage trainable method:
- Stage 1: PHQ-aware Symptom Summarization. An LLM is instruction-tuned to generate the structured symptom summaries. Crucially, the authors introduce PHQ-aware Contrastive Learning (PhqCL). Instead of relying solely on token-level generation loss, this objective aligns the hidden representation of the generated summary with the participant's eight-dimensional PHQ-8 score vector. This ensures that summaries from participants with similar clinical profiles are pulled closer in the embedding space, while dissimilar profiles are pushed apart.
- Stage 2: Multimodal Depression Recognition. The PHQ-aware summary representation is fused with acoustic (via Wav2Vec2) and visual (via PointMLP) features. These modalities are compressed using Perceiver Resamplers and concatenated with the textual summary embeddings. A unified LLM architecture performs cross-modal fusion to predict the final depression label.

Key Contributions

Explain-MDRC Framework: A novel formulation that couples PHQ-8-aligned symptom summarization with multimodal depression recognition, making clinician-readable symptom evidence an explicit intermediate output rather than a post-hoc explanation.
Explain-DAIC Dataset: A new dataset derived from DAIC-WOZ, enriched with manually annotated, structured symptom summaries that link participant utterances to clinically meaningful symptom dimensions.
PhqCML Model: A method combining PHQ-aware contrastive learning and summary-informed multimodal fusion. It treats the symptom summary as an explicit reasoning layer, improving both recognition performance and interpretability.

Experimental Results
Experiments conducted on the Explain-DAIC dataset demonstrate the effectiveness of the proposed approach:

Recognition Performance: PhqCML outperforms the previous state-of-the-art (MTRB) by 7.10 absolute percentage points in Macro-F1 (reaching 95.10% with the Qwen3 backbone). It also significantly outperforms general-purpose LLM APIs (e.g., GPT-5.5) and single-stage LLM baselines that map raw dialogue directly to labels.
Summarization Quality: The inclusion of PHQ-aware contrastive learning improves the quality of generated summaries, evidenced by higher ROUGE-2 scores and better alignment with clinical symptom phrases.
Representation Analysis: Probing experiments show that PhqCL makes summary embeddings more linearly predictive of individual PHQ-8 item scores. Attention shift analysis reveals that the model focuses more on PHQ-8 symptom keywords and clinical context compared to vanilla models.
Expert Evaluation: In a blinded evaluation by three licensed psychiatrists, PhqCML significantly outperformed GPT-5.5 in completeness, correctness, and conciseness ( $p < 0.001$ ). Experts noted that PhqCML's summaries were more faithfully grounded in interview evidence and avoided unsupported clinical inferences.
Cross-Dataset Transfer: When applied to the CMDC dataset (a Mandarin PHQ-9 corpus) without fine-tuning, the symptom summarization component successfully inferred depression labels with competitive performance. Specifically, the variant adapted with a PHQ-9 instruction (PhqCML-SymSum-Qwen3*) reached 95.79% Macro-F1, suggesting the model learns transferable clinical reasoning rather than dataset-specific cues.

Significance and Claims
The paper claims that this work moves MDRC research from black-box classification toward clinician-aligned symptom reasoning. By generating structured, evidence-backed symptom summaries, the framework provides transparent intermediate evidence that mirrors clinical workflows. The authors posit that enhancing interpretability not only aids clinician review but also strengthens recognition performance by encouraging models to ground predictions in clinically meaningful cues.

The authors maintain a modest stance regarding deployment, explicitly stating that the current framework is a screening-oriented research prototype rather than a deployable diagnostic system. They acknowledge limitations, including the lack of external clinical knowledge graphs, the static nature of the learning paradigm, and the need for broader safety evaluations and prospective clinical validation before real-world use. The work is presented as a promising direction for transparent, AI-assisted depression recognition research.

More like this