Technical Summary: MENTALBENCH

Problem Statement

Large Language Models (LLMs) have shown promise in mental health analysis, yet existing benchmarks for evaluating their psychiatric diagnostic capabilities suffer from significant limitations. Most current datasets rely on social media data or supportive dialogue settings, which lack the rigorous structure required to assess whether models can apply formal diagnostic criteria (such as those in the DSM-5) and navigate complex differential diagnostic rules. Consequently, it remains unclear if LLMs can reliably resolve clinical boundaries between competing diagnoses under inherent clinical ambiguity, a critical requirement for high-stakes clinical decision support. The construction of high-quality, expert-validated diagnostic data is further hindered by strict privacy regulations, the absence of objective biomarkers, and the complexity of formal diagnostic logic.

Methodology

To address these gaps, the authors introduce MENTALBENCH, a benchmark grounded in a psychiatrist-built knowledge graph called MENTALKG.

1. MENTALKG: A DSM-Grounded Knowledge Graph

MENTALKG serves as the logical backbone for the benchmark. It is a structured knowledge graph encoding DSM-5 diagnostic criteria and differential diagnostic rules for 23 psychiatric disorders.

• Structure: The graph comprises 23 Disorder nodes, 23 Symptom Group nodes, 84 Symptom nodes, and 87 Differential Diagnosis nodes.
• Formalization: It transforms unstructured DSM-5 text into structured entities, explicitly encoding mandatory symptoms, duration constraints, minimum symptom thresholds, and exclusionary logic.
• Differential Logic: Differential diagnosis nodes capture the directional logic required to distinguish between overlapping disorders (e.g., Major Depressive Disorder vs. Bipolar I), defining triggering conditions, key differences, and specific discriminating rules.
• Diversity: To avoid rigid, textbook-style phrasing, the schema integrates symptom subtypes, allowing for varied linguistic and symptomatic manifestations of core symptoms.
• Validation: The graph construction involved two experts (a board-certified psychiatrist and a licensed clinical psychologist) who validated symptom subtypes, criterion alignment with DSM-5, and the clinical validity of differential diagnosis logic.

2. MENTALBENCH: Benchmark Construction

Leveraging MENTALKG, the authors generated 24,750 synthetic clinical cases organized into a multi-stage pipeline. The benchmark evaluates models on two primary challenges: Information Incompleteness and Diagnostic Complexity.

The cases are categorized into four types:

• Type 1 (Professional Summary): Structured, third-person medical chart summaries where all diagnostic evidence is explicit and complete.
• Type 2 (Patient Self-Report): Fragmented, colloquial, first-person narratives reflecting selective disclosure and expression diversity, where models must recover diagnostic cues from incomplete information.
• Type 3 (Controlled Ambiguity): Differential diagnosis scenarios where shared symptoms satisfy multiple disorders, but the discriminating evidence required to rule out candidates is intentionally omitted. The correct answer is a set of plausible diagnoses.
• Type 4 (Definitive Diagnosis): Differential diagnosis scenarios where the discriminating evidence is included, requiring the model to narrow the candidate set to a single definitive diagnosis based on fine-grained criteria.

The generation process involves seed generation (sampling from MENTALKG), profile augmentation (assigning demographics and symptom subtypes), and data synthesis using an ensemble of three distinct LLMs to mitigate generator bias. All outputs underwent rigorous expert validation for linguistic naturalness, diagnostic validity, and clinical realism.

Key Contributions

1. MENTALKG: An expert-built knowledge graph that formalizes DSM-5 diagnostic criteria and differential diagnosis rules for 23 disorders, providing a structured logical backbone for clinical assessment.
2. MENTALBENCH: A large-scale benchmark comprising 24,750 synthetic cases that systematically vary information completeness and diagnostic complexity to evaluate LLMs under clinical ambiguity.
3. Comprehensive Evaluation: A systematic assessment framework that moves beyond surface-level pattern recognition to test the rigorous application of diagnostic logic, including the ability to handle fragmented narratives and resolve overlapping symptom profiles.

Experimental Results

The authors evaluated a diverse range of open-source (Qwen, Gemma, Llama, MentaLLaMA) and closed-source (GPT, Gemini, Claude) models on MENTALBENCH.

• Overall Performance: Even state-of-the-art models achieved a maximum overall accuracy of only 62.69%, highlighting the benchmark's difficulty.
• Information Incompleteness: Models performed well on Type 1 (structured summaries) but showed substantial performance degradation on Type 2 (narrative self-reports). Smaller models (<30B parameters) suffered a significant average drop (10.58 points), whereas larger models were more resilient, suggesting that scale aids in semantic inference from fragmented data.
• Diagnostic Complexity: Performance dropped precipitously in differential diagnosis tasks (Type 3 and Type 4), with many models scoring below 20%.
  • Open-source models tended to exhibit over-diagnosis in unambiguous cases (Type 4), failing to apply exclusion rules and selecting multiple plausible distractors.
  • Closed-source models tended to exhibit under-diagnosis in ambiguous cases (Type 3), forcing single-label predictions even when clinical evidence supported multiple conditions.
• Disease-Specific Findings: Models struggled significantly with disorders underrepresented in prior research (e.g., Schizoaffective Disorder) and those requiring fine-grained temporal distinctions (e.g., Persistent Depressive Disorder vs. Major Depressive Disorder).
• Calibration Failure: Analysis revealed that models lack intrinsic constraint awareness. They struggle to autonomously calibrate their diagnostic scope (deciding between single vs. multiple diagnoses) without explicit prompting instructions.

Significance and Claims

The paper claims that MENTALBENCH exposes critical limitations in the current capabilities of LLMs for psychiatric decision support. While models possess latent knowledge of DSM-5 criteria, they fail to rigorously apply this knowledge in ambiguous, real-world-like scenarios. Specifically:

• Models struggle to map real-world, fragmented patient expressions onto formal diagnostic criteria.
• Models fail to calibrate diagnostic commitment, oscillating between overly expansive and excessively restrictive tendencies depending on the model architecture and scenario ambiguity.
• Current models lack the intrinsic ability to define decision boundaries without external guidance, raising concerns about their reliability in real-world settings where the number of plausible diagnoses is inherently uncertain.

The authors conclude that MENTALBENCH advocates for future research to prioritize adaptive, robust decision-making that can handle clinical ambiguity and rigorous logical constraints, rather than relying on surface-level associations. The resources (code and data) are publicly released to facilitate further evaluation and development in this domain.