💬 NLP

A Multi-Probe Audit of Clinical-Interview Depression Detection Benchmarks

This paper audits clinical-interview depression detection benchmarks through four probes, revealing that strict subject-disjoint evaluation yields a new performance baseline, official test splits poorly correlate with cross-validation rankings, zero-shot transfer to external corpora is weak, and text-based models significantly outperform audio models on symptom-dense interview segments.

Original authors: Takehiro Ishikawa, Jon Duke

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Takehiro Ishikawa, Jon Duke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the field of AI depression detection is like a high-stakes cooking competition. Researchers are trying to build the perfect "recipe" (an algorithm) that can taste a person's voice or read their words and say, "This person is depressed." They use a specific set of ingredients (datasets) called E-DAIC, CMDC, and ANDROIDS to test their recipes.

For years, the judges have been using a very small, specific tasting menu (the "official test split") to decide who wins. This paper is a food safety audit. The authors, Takehiro Ishikawa and Jon Duke, decided to check if the competition rules are actually fair, if the winners are truly the best chefs, and if the recipes work outside the specific kitchen they were tested in.

Here is what they found, broken down into four simple "probes" or tests:

1. The "Strict Judge" Test (Probe A)

The Problem: In the past, researchers tested their recipes on a tiny, pre-selected group of people (the "official test"). It's like a chef practicing on the exact same 50 customers every time. They might memorize those customers' preferences rather than learning how to cook for anyone. This leads to scores that look amazing but might be fake.

The Audit: The authors re-ran the tests using a much stricter rule: Leave-One-Subject-Out. Imagine a chef cooking for 100 people, but for every single person, they cook without having ever seen that specific person before. They have to guess based on general skills, not memory.

The Result: When they did this, the scores dropped. The best recipe they found scored a 72.3% (Macro-F1). This is lower than the "90%+" scores often reported in the news, but it's a honest, realistic score. It proves that the previous high scores were likely inflated by the "memorization" of the small test group.

2. The "Leaderboard Lottery" Test (Probe B)

The Problem: The competition uses a tiny group of people to rank the top 100 recipes. The authors asked: "If we shuffle the deck slightly, does the winner stay the winner?"

The Audit: They ran 96 different variations of recipes (changing the ingredients, the cooking method, and the mixers) and saw how they ranked.

The Result: The leaderboard is chaotic.

The recipe that won the official test was actually ranked 41st when judged by the stricter rules.
The recipe that was ranked 1st by the strict rules was only 20th on the official test.
There was zero overlap between the top 3 winners of the two methods.
Even the "winner" of the official test only won about 32% of the time if you shuffled the test group slightly.

The Takeaway: Winning first place on this specific leaderboard doesn't mean you have the best recipe. It might just mean you got lucky with the specific group of people you were tested on. The "gap" between first and second place is likely just noise, not a real difference in quality.

3. The "New Kitchen" Test (Probe C)

The Problem: Some researchers claimed to have "solved" depression detection on two other datasets (CMDC and ANDROIDS), with scores near 95%. It sounded like the problem was finished.

The Audit: The authors took these "solved" recipes and tried to use them in completely different kitchens (other datasets like MODMA and PDCH) without changing anything (Zero-Shot Transfer).

The Result: The recipes failed miserably.

A recipe that scored 95% in its home kitchen dropped to 26% or even 12% in the new kitchens.
It turns out these recipes weren't learning how to detect depression; they were learning the specific quirks of the original kitchen (like the accent of the interviewer or the specific questions asked). They couldn't generalize to new people or new languages.

The Takeaway: Just because a model scores perfectly on one dataset doesn't mean it works in the real world. High scores on one dataset do not mean the problem is "solved."

4. The "Topic Sensitivity" Test (Probe D)

The Problem: Text-based models (reading what people say) usually get the highest scores, beating audio (voice) and video (facial expressions). Everyone assumes this means text is the "superpower" of depression detection.

The Audit: The authors sliced the interviews into two types of moments:

Heavy Topics: When the person is talking directly about sadness, sleep, or suicide (the "symptom-dense" parts).
Neutral Topics: When the person is talking about the weather or their commute (the "symptom-light" parts).

They tested if the models got better at detecting depression only during the heavy topics.

The Result:

Text Models: Their scores skyrocketed when the person talked about symptoms. They basically just recognized the keywords.
Audio Models: Their scores stayed flat. They didn't get better or worse based on the topic.

The Takeaway: The text models aren't necessarily "smarter." They are just keyword detectors. They perform well because the interview questions force people to talk about their symptoms. If you take away those specific words, the text model loses its advantage. It's not detecting the feeling of depression as much as it is detecting the words about depression.

Summary

This paper is a reality check for the AI depression detection field.

Don't trust the official leaderboard: It's unstable and easy to game.
Don't trust "solved" claims: High scores on one dataset don't mean the model works everywhere.
Text isn't magic: Text models are good because they spot specific words, not because they understand the human condition better than voice or video.

The authors are calling for the field to stop chasing tiny score improvements on small, biased test groups and start building models that are robust, honest, and actually work on new people.

Technical Summary: A Multi-Probe Audit of Clinical-Interview Depression Detection Benchmarks

Problem Statement

The field of clinical-interview depression detection relies heavily on a few benchmark datasets, particularly the DAIC family (specifically E-DAIC). Current evaluation practices face three critical challenges:

Over-reliance on Privileged Splits: Most studies report performance on a single, small official holdout set (56 participants). This creates a risk of overfitting to split-specific noise or idiosyncrasies, leading to inflated scores that do not reflect generalizable performance.
Evaluation Instability: The small size of the official test set makes fine-grained leaderboard rankings unstable. Small differences in macro-F1 may result from finite-sample noise rather than genuine model superiority.
Lack of External Validation: High in-domain performance on datasets like CMDC and ANDROIDS (often near-ceiling) is frequently interpreted as a "solved" problem. However, there is limited evidence that these pipelines generalize to external corpora or different clinical contexts.
Modality Interpretability: While text modality consistently yields the highest performance, it is unclear whether this reflects the discovery of subtle, context-independent linguistic markers of depression or merely a sensitivity to explicit, symptom-dense content within the interview protocol.

Methodology

The authors conduct a four-probe audit across five datasets (DAIC/E-DAIC, CMDC, ANDROIDS, MODMA, PDCH) using strict subject-disjoint protocols to prevent data leakage.

Probe A: Re-establishing a Conservative Anchor for E-DAIC

Protocol: Leave-One-Subject-Out (LOSO) cross-validation, strictly excluding the privileged official test set.
Model: A lightweight hybrid model (T+L) combining:
- Text (T): Turn-level embeddings from sentence-transformers/all-mpnet-base-v2 and intfloat/e5-large-v2, augmented with turn-structure features (duration, gaps, length).
- LLM Score (L): A participant-level probability (depression_prob) generated by openai/gpt-oss-120b acting as a fixed, zero-shot DSM-5/PHQ-8 assessor on full-dialogue transcripts. The LLM is not fine-tuned on E-DAIC labels.
Goal: To provide a reproducible, out-of-fold reference point that does not depend on the official split.

Probe B: Official-Split Instability Analysis

Protocol: A sweep of 96 model configurations (8 modality bundles × 2 pooling strategies × 6 learners) evaluated on the official E-DAIC split.
Analysis: Compared development-side 5-fold cross-validation rankings against official test rankings. Performed 4,000 subject bootstraps to assess the stability of the "winner."
Goal: To determine if the official split supports fine-grained model ranking.

Probe C: Zero-Shot External Validation

Protocol: Reproduced two strong, public baselines (CMDC text pipeline and ANDROIDS multimodal pipeline) and applied them to external corpora (MODMA, PDCH, E-DAIC, ANDROIDS) without any target-label tuning or adaptation.
Goal: To test whether near-ceiling in-domain performance transfers to external data.

Probe D: Topic Sensitivity Stress Test

Protocol: Constructed paired "symptom-dense" (heavy-topic) and "symptom-light" (neutral-topic) slices from E-DAIC interviews using a pre-specified automated SRDS-based annotator.
Analysis: Compared the performance shift of text vs. audio models when moving from neutral to symptom-dense slices within the same participants.
Goal: To isolate whether the dominant text signal is driven by explicit symptom content.

Key Results

1. E-DAIC Performance under Strict LOSO (Probe A)

The hybrid T+L model achieved a macro-F1 of 0.723 under strict LOSO.
This is the highest reported performance under this protocol but is notably lower than many optimistic scores reported on the official split.
The L-only component (LLM score) achieved a macro-F1 of 0.686, while T-only achieved 0.621.

2. Instability of Official Rankings (Probe B)

Low Alignment: Cross-validation and official test rankings showed only moderate alignment (Pearson $r=0.637$ , Spearman $\rho=0.704$ ).
Ranking Discrepancy: The best configuration by cross-validation ranked 20th on the official test. The official test winner ranked 41st by cross-validation.
Zero Overlap: There was zero overlap in the top-3 configurations between cross-validation and the official test.
Bootstrap Uncertainty: Even the apparent official-test winner was ranked #1 in only 32.3% of 4,000 bootstrap resamples, with a 95% rank range spanning positions 1 to 19.

3. Failure of Zero-Shot Transfer (Probe C)

CMDC Baseline: While achieving ~0.95 macro-F1 in-domain, zero-shot transfer to MODMA dropped to 0.265 macro-F1, and to PDCH ranged from 0.127 to 0.446.
ANDROIDS Baseline: Reconstructed the source-side fusion macro-F1 of 0.955. However, on external targets, the audio branch was the most portable (e.g., 0.647 on CMDC), while the text branch (using an Italian sentiment encoder) and fusion branches performed significantly worse (e.g., 0.288 AUROC on CMDC text).
Conclusion: High in-domain scores do not guarantee generalizability; pipelines may exploit corpus-specific regularities.

4. Text Sensitivity to Symptom Content (Probe D)

Text Shift: Text model scores rose sharply on symptom-dense slices (mean shift +0.422).
Audio Shift: Audio model scores remained nearly flat (mean shift -0.004).
Gap: The text-minus-audio gap was positive and significant across all five seeds ( $p=0.0002$ ).
Conclusion: The superior performance of text models is strongly tied to the presence of explicit, symptom-dense content in the interview, rather than subtle, context-independent markers.

Significance and Claims

The paper claims to provide a "conservative anchor" and a "reusable reference point" for the field, rather than proposing new state-of-the-art models. Its primary contributions are:

Redefining Benchmarks: It argues that the single official split of E-DAIC is insufficient for claiming model superiority. The authors provide a new, reproducible LOSO baseline (macro-F1 0.723) that avoids the risks of split-specific optimization.
Warning Against Leaderboard Optimization: The study demonstrates that fine-grained rankings on small official splits are unstable and should not be interpreted as definitive evidence of superiority. Claims of "state-of-the-art" should be accompanied by uncertainty estimates or external validation.
Auditing "Solved" Baselines: By systematically applying strong public baselines to external data, the paper challenges the assumption that high in-domain scores on CMDC and ANDROIDS indicate a solved problem. It highlights a lack of generalizability in current pipelines.
Interpretability of Text Modality: The paper clarifies that the dominance of text models in depression detection is likely driven by their sensitivity to explicit symptom discussions (e.g., sleep, mood, anhedonia) elicited by the interview protocol, rather than the discovery of novel, subtle linguistic biomarkers.

The authors maintain a modest tone, acknowledging limitations such as potential LLM pretraining exposure (though deemed unlikely for controlled-access data) and the reliance on automated topic slicing rather than manual expert annotation. The core message is a call for more rigorous, subject-disjoint, and externally validated evaluation practices in clinical depression detection research.