When Consistency Becomes Bias: Interviewer Effects in Semi-Structured Clinical Interviews

This paper reveals that automatic depression detection models trained on semi-structured clinical interviews often achieve inflated performance by exploiting systematic biases in interviewer prompts rather than genuine linguistic cues from participants, highlighting the critical need to restrict analysis to patient utterances for valid interpretability.

The Gist

Imagine you are trying to teach a computer to recognize when someone is feeling depressed. To do this, you give the computer thousands of recordings of therapy sessions. In these sessions, a therapist (or a robot acting like one) asks a series of questions, and the patient answers.

The researchers in this paper discovered a sneaky trick that these computers were pulling. Instead of actually listening to the patient's sad or happy words, the computers were learning to "cheat" by looking at the therapist's script.

Here is the story of their discovery, broken down with some everyday analogies.

1. The Setup: The "Scripted" Interview

Think of a clinical interview like a standardized driving test.

• The Examiner (Interviewer): Has a strict script. "Turn left," "Stop at the red light," "Check your mirrors." They ask these questions in the exact same order for every single driver.
• The Driver (Patient): Answers freely. Some drivers are nervous and stutter; others are confident and smooth.

The goal is to see if the computer can tell which drivers are "bad" (depressed) just by listening to the conversation.

2. The Problem: The Computer is Cheating

The researchers trained two types of AI models:

• The "Patient-Only" Model: This AI was only allowed to listen to the driver's answers.
• The "Interviewer-Only" Model: This AI was only allowed to listen to the examiner's questions.

The Shocking Result: The "Interviewer-Only" model often did better than the "Patient-Only" model.

The Analogy: Imagine a teacher grading a test.

• The Honest Way: The teacher reads the student's essay to see if they understand the material.
• The Cheating Way: The teacher looks at the order of the questions. "Ah, the student is answering Question 4, which is always about 'family trauma.' If they are answering Question 4, they must be in the 'sad' group."

The AI wasn't learning about depression; it was learning the pattern of the script. It realized that certain questions (like "How is your family?") always appeared at specific times, and the type of person answering them (depressed vs. not) was predictable based on the question itself, not the answer.

3. The "Shortcut" (Bias)

The paper calls this "Prompt-Induced Bias."

Think of the interview script like a recipe.

• If you are baking a cake (diagnosing depression), you want to taste the batter (the patient's words) to see if it's sweet enough.
• But the AI found a shortcut: It realized that if the recipe says "Add eggs now," the cake is always going to be a "depressed cake" in this specific dataset. So, the AI stopped tasting the batter and just looked at the recipe steps.

Because the script is so consistent (the "examiner" asks the same things in the same order), the AI could guess the diagnosis just by knowing which question was being asked, completely ignoring what the patient actually said.

4. The Evidence: Heatmaps

The researchers used "heatmaps" (like a weather map showing hot and cold spots) to see where the AI was looking.

• The Cheating AI (Interviewer Model): The heatmap showed bright, narrow lines. It was laser-focused on specific moments where the therapist asked a specific question. It ignored 90% of the conversation.
• The Honest AI (Patient Model): The heatmap was spread out. It was looking at the patient's words throughout the whole conversation, which is how it should work.

5. Why This Matters

This is a big deal because many researchers thought that including the therapist's questions helped the AI understand the context better. This paper says: "No, it's actually making the AI lazy."

If we build a real-world AI doctor that relies on these shortcuts, it might fail miserably in a real situation where the therapist asks questions in a different order or uses different words. The AI would be confused because it learned the script, not the human.

The Takeaway

The authors are saying: "Don't let the AI read the script; make it listen to the person."

To build truly helpful AI for mental health, we need to strip away the interviewer's questions and force the computer to learn from the patient's actual words, emotions, and stories. Otherwise, we aren't detecting depression; we're just detecting how well the AI memorized the therapist's checklist.

Technical Summary

1. Problem Statement

Automatic depression detection using clinical interviews has advanced significantly due to public corpora and language modeling. However, a critical interpretability gap exists: high-performing models often fail to reveal what drives their predictions.

• The Core Issue: Recent studies train models on both participant responses and interviewer prompts. It is unclear whether these models learn genuine linguistic markers of depression from the patient or if they exploit systematic biases inherent in the interviewer's script.
• The Hypothesis: In semi-structured interviews, the interviewer follows a fixed protocol (standardized prompts, specific question ordering). Models may learn to classify depression based on these fixed "script artifacts" (e.g., specific question positions or recurring prompts) rather than the participant's actual language, leading to inflated performance metrics that do not generalize to real-world clinical reasoning.

2. Methodology

Datasets

The study analyzes three distinct clinical interview corpora to ensure cross-dataset validity:

1. DAIC-WOZ: A North American English dataset featuring a virtual interviewer ("Ellie") controlled by a human (Wizard-of-Oz). It includes full transcripts of both speakers and PHQ-8 depression ratings.
2. E-DAIC: An extension of DAIC-WOZ with a fully automatic virtual interviewer. Originally, only participant transcripts were released; the authors generated interviewer transcripts using WhisperX (ASR) to ensure fair comparison.
3. ANDROIDS: An Italian speech corpus collected "in the wild" with a human interviewer instructed to use minimal prompts. It includes only audio; the authors generated both participant and interviewer transcripts using WhisperX.

Experimental Setup

To test if the bias is model-agnostic, the authors evaluated two distinct architectures:

• Longformer: A transformer model with sparse attention, suitable for long documents (interviews).
• GCN (Graph Convolutional Network): A graph-based model representing the corpus as nodes (words and documents), allowing for interpretability regarding which specific words drive classification.

Training Variants:
For each dataset and architecture, two model variants were trained and evaluated:

• Participant-only (P): Trained exclusively on the patient's utterances.
• Interviewer-only (I): Trained exclusively on the interviewer's prompts/questions.

Data Processing:
To ensure a fair comparison, the authors avoided using "gold" transcripts for participants in E-DAIC and ANDROIDS. Instead, they used ASR-generated transcripts for both speakers in these datasets to simulate realistic conditions where interviewer transcripts might be noisy or automatically generated.

3. Key Contributions

1. Quantification of Prompt-Induced Bias: The study demonstrates that models trained only on interviewer prompts (I-models) often match or significantly outperform models trained only on participant responses (P-models).
2. Architecture Agnosticism: The bias is shown to be independent of the model architecture, persisting across both semantic transformers (Longformer) and keyword-focused graph models (GCN).
3. Qualitative Evidence: Through heatmaps and keyword analysis, the authors prove that I-models rely on narrow, fixed segments of the interview (specific prompts), whereas P-models distribute decision evidence broadly across the conversation.

4. Key Results

Performance Metrics (Development Sets)

• DAIC-WOZ: I-models consistently outperformed P-models.
  • Longformer: I (0.73) > P (0.71).
  • GCN: I (0.88) > P (0.85).
• ANDROIDS: The bias was most pronounced here.
  • Longformer: I-model achieved 0.98 macro-F1, a 19% advantage over the P-model (0.79).
  • GCN: I-model (0.97) outperformed P-model (0.93).
• E-DAIC: I-models generally outperformed or matched P-models, particularly with GCN.

Test Set Performance

• On E-DAIC, the bias persisted on held-out test data for both architectures, confirming the models learned the script structure rather than overfitting to the dev set.
• On DAIC-WOZ, the effect was architecture-dependent. GCN models still benefited from interviewer turns, while Longformer (semantic modeling) showed a reversal where P-models outperformed I-models on the test set, suggesting semantic context can partially mitigate shortcut learning.

Qualitative Analysis (Heatmaps & Keywords)

• I-Models (Interviewer-only): Decision evidence was concentrated in narrow, high-contrast bands corresponding to specific prompts (e.g., "How do you cope with that?", "Do you still go to therapy?"). The model learned that the presence or position of these specific questions in the script correlated with depression labels.
• P-Models (Participant-only): Decision evidence was distributed low-contrast activity across the entire timeline, indicating the model relied on the content of the patient's speech rather than fixed structural cues.

5. Significance and Implications

• Methodological Warning: The study reveals that including interviewer turns in training data can artificially inflate performance metrics. High scores may reflect the model's ability to "read the script" rather than its ability to detect depression.
• Generalizability: This is a cross-dataset phenomenon affecting different languages (English, Italian) and collection settings (lab vs. wild, human vs. virtual interviewers).
• Recommendations:
  • Future research should prioritize participant-only evaluations to ensure models learn genuine linguistic markers.
  • Evaluation protocols must localize decision evidence by time and speaker to detect "shortcut learning."
  • Clinical AI systems should be designed to be bias-aware, isolating the patient's contribution to avoid relying on interviewer artifacts.

Conclusion

The paper concludes that consistency in semi-structured interviews, while methodologically sound for human clinicians, creates a "bias trap" for AI models. Models exploit the fixed nature of interviewer prompts as shortcuts. To build truly interpretable and robust depression detection systems, the field must shift focus toward analyzing the participant's language in isolation from the interviewer's script.