Technical Summary: Evidence-Weighting Patterns in Large Language Model Psychiatric Screening

Problem Statement

The global demand for mental health care significantly outpaces the availability of clinician-delivered assessments, creating a critical need for scalable screening tools. While Large Language Models (LLMs) offer potential for extracting psychiatric risk signals from patient narratives, their reliability remains uncertain. Existing literature often focuses on final classification accuracy or single-disorder tasks using self-disclosed labels, leaving three critical gaps for clinical translation:

1. Lack of SCID-Anchored Benchmarks: Few studies evaluate LLMs using semi-structured experiential interviews paired with gold-standard Structured Clinical Interview for DSM (SCID) diagnostic reference labels.
2. Insufficient Subgroup Characterization: Performance variations across demographic strata (sex, age, race) are not well understood.
3. Opaque Error Mechanisms: It is unclear whether false-negative errors stem from a failure to detect symptoms or from how models differentially weight symptom evidence against functional impairment and protective-context cues (e.g., coping ability, social support).

Methodology

The authors introduced a benchmark comprising 555 semi-structured experiential interviews collected from participants (Mage = 39.4; 50.1% female; 77.8% White). Participants underwent structured clinical interviews (SCID) to establish reference labels for four diagnostic categories: Anxiety Disorder, Major Depressive Disorder (MDD), Post-Traumatic Stress Disorder (PTSD), and Any Current Mental Health Disorder (MHD).

Experimental Design:

• Models Evaluated: Five state-of-the-art LLMs (LLaMA 3, DeepSeek, GPT-4o Mini, GPT-4.1 Mini, GPT-5 Mini).
• Prompting Strategy: Zero-shot, task-specific prompting. Models were instructed to evaluate one condition at a time, generate a clinical-style inference rationale, and provide a binary screening prediction ("Yes"/"No").
• Performance Metrics: Accuracy, Precision, Recall, Specificity, F1 Score, and Matthews Correlation Coefficient (MCC). Statistical significance was assessed using Cochran's Q tests and McNemar tests with Bonferroni correction.
• Evidence-Integration Analysis: A post-hoc analysis of GPT-4.1 Mini's inference rationales (selected for consistent structure and performance) categorized text into three evidence domains using rule-based regular expressions:
  1. Symptom Evidence: Task-specific psychiatric symptoms (derived from PHQ-9, GAD-7, PCL-5).
  2. Functional Impairment: Disruption in occupational, social, or daily functioning.
  3. Protective Context: Preserved functioning, active coping, and social support.
• Statistical Modeling: Mann–Whitney U tests compared evidence counts between True Positives (TP) and False Negatives (FN). Standardized logistic regression models determined how these evidence features predicted model outputs versus SCID-derived labels.

Key Results

1. Screening Performance

Model performance varied substantially across tasks and families (Accuracy: 0.49–0.86; MCC: 0.16–0.38).

• Top Performers: GPT-4.1 Mini and GPT-5 Mini demonstrated the most consistent disorder-specific accuracy. GPT-4.1 Mini achieved the highest accuracy for Anxiety (0.72), Depression (0.86), and PTSD (0.81).
• Variability: GPT-4o Mini showed the lowest accuracy for anxiety screening (0.49).

2. Demographic Disparities

• Sex: A consistent pattern emerged where depression classification accuracy was higher for male participants than female participants across all models (e.g., GPT-4.1 Mini: Male 0.91 vs. Female 0.82).
• Age & Race: No consistent age-related pattern or uniform degradation across race strata was observed, though model-dependent variations existed.

3. Evidence-Weighting Patterns (The Core Finding)

Analysis of false-negative errors revealed that models did not simply miss symptoms; they actively down-weighted them in the presence of protective cues.

• Symptom Presence in Errors: False-negative cases for Anxiety and PTSD often contained higher counts of symptom evidence than true-positive cases.
• The Role of Context:
  • Protective Context: The presence of preserved functioning, coping ability, or social support cues was negatively associated with positive model outputs. In Anxiety and PTSD, false negatives frequently included these protective cues alongside symptoms.
  • Functional Impairment: Evidence of functional impairment was positively associated with positive classifications.
• Model vs. Ground Truth: Logistic regression showed that evidence features (especially functional impairment and protective context) were stronger predictors of the LLM's decisions than of the SCID-derived diagnostic labels. This suggests the models apply a stricter or different weighting heuristic than the clinical reference standard, particularly requiring a convergence of symptoms and impairment before assigning a positive label.

Significance and Claims

The paper claims that LLMs can support scalable psychiatric screening but highlights a specific mechanism of failure that requires careful validation before clinical deployment.

• Beyond Accuracy: The study argues that aggregate accuracy is insufficient for clinical evaluation. The primary finding is that false negatives often arise not from a lack of symptom detection, but from the model's tendency to discount symptom evidence when the narrative includes preserved functioning or protective contexts.
• Clinical Implication: This "evidence-weighting" behavior suggests that LLMs may under-diagnose conditions where patients report symptoms but also describe effective coping or social support, potentially missing individuals who still require clinical evaluation.
• Future Direction: The authors conclude that future systems must move beyond single-pass inference to calibrated, multi-turn, or multi-agent frameworks capable of explicitly resolving uncertainty and integrating competing clinical evidence (symptoms vs. functioning) to reduce false-negative errors.

The study emphasizes that while GPT-4.1 Mini and GPT-5 Mini show promise, their decision-making logic regarding the balance between symptomatology and functional context differs from standard clinical diagnostic criteria, necessitating further investigation into fairness and error mechanisms.