Technical Summary: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry

Problem Statement

Large Language Models (LLMs) are increasingly deployed for clinical decision support, yet existing medical benchmarks (e.g., MedMCQA, PubMedQA) typically assume that complete information is available upfront. In real-world clinical practice, particularly in psychiatry, information is often incomplete, ambiguous, or evolves over time. When evidence is insufficient, the clinically safe action is not to guess, but to request clarification or abstain from diagnosis. Current benchmarks fail to evaluate how models handle this "evolving diagnostic uncertainty," often overlooking the risks of premature commitment (diagnosing with insufficient data) or excessive abstention (refusing to diagnose when data is sufficient). Furthermore, existing abstention-focused evaluations often treat uncertainty as a binary choice, neglecting the critical intermediate step of seeking clarification.

Methodology

The Safe-Psych Benchmark

The authors introduce Safe-Psych, a sequential evaluation benchmark comprising 1,048 real-world psychiatric clinical notes from a hospital in Romania. The dataset was anonymized, translated into English, and segmented into five ordered sections to simulate the incremental disclosure of clinical evidence:

1. Presenting symptoms
2. Psychiatric history
3. Psychiatric examination
4. Psychological examination
5. Other diagnoses (medical factors)

Annotation Protocol: Two psychiatrists independently annotated each case to determine:

• Information Sufficiency: Whether the available text supports a diagnosis.
• Earliest Sufficient Section: The specific point in the sequence where enough evidence exists to diagnose.
• Ground Truth Diagnosis: An ICD-10 code (if sufficient).
  Disagreements were resolved by a third senior psychiatrist. These judgments were mapped to three sequential action labels for each step: DIAGNOSE, CLARIFY (request more info), or ABSTAIN (decline to diagnose).

Evaluation Protocol

The study employs a sequential evaluation protocol where models receive information incrementally (Section 1, then 1+2, etc.). At each step, the model produces a free-form response, which is classified by an independent LLM judge (GPT-5.4) into one of the three action labels.

• DIAGNOSE: The trajectory terminates; the model commits to a diagnosis.
• CLARIFY: The trajectory continues to the next section.
• ABSTAIN: The model declines to diagnose. If this occurs before the final section, it is recorded as "premature abstention."

Models and Strategies

Nine state-of-the-art LLMs were evaluated, including closed-source models (GPT-5.4, Claude-Opus-4.6, Gemini-2.5-Flash), open-source models (Gemma-3, Mistral-3.1-24B, Qwen-3-32B), and medical-specialized models (MedGemma-27B, Med42-v2-8B).
Four inference strategies were tested:

1. Static (Full Information): Models see the complete note.
   • No Abstention: Forced to diagnose.
   • Abstention Aware: Instructed to abstain if evidence is insufficient.
2. Sequential: Models see notes incrementally.
   • No Abstention: Neutral prompting.
   • Clarification & Abstention Aware: Instructed to clarify if unsure and abstain if evidence remains insufficient at the end.

Key Results

1. Capability Does Not Ensure Calibration

Even the most capable models demonstrated poor calibration under incomplete information.

• Under-abstention: Most models diagnosed cases that experts deemed insufficient over 60% of the time, even when explicitly allowed to abstain. Gemma-3-27B diagnosed over 90% of insufficient cases.
• Over-abstention: Models that reduced under-abstention often compensated by abstaining on sufficient cases (e.g., Mistral-3.1-24B abstained on nearly 60% of sufficient cases).
• Conclusion: Safety-aware prompting shifts errors from over-diagnosis to over-abstention rather than solving the calibration problem.

2. Limited Clarification-Seeking Behavior

In sequential settings with neutral prompting, models rarely sought clarification unless explicitly instructed.

• Clarification rates at the first stage (highest uncertainty) were low for most models (e.g., <10% for Gemini-Flash-2.5 and Med42-v2-8B).
• Models tended to default to early diagnosis rather than requesting additional evidence, indicating that clarification behavior does not emerge naturally from neutral prompting.

3. Premature Diagnosis and Accuracy Trade-offs

• Timing: Under neutral prompting, the vast majority of diagnoses were "premature" (made before expert sufficiency was reached).
• Accuracy: "On-time" diagnoses (made when experts deemed sufficient) were consistently more accurate than premature ones across all label granularities (Full ICD, 3-character, high-level class).
• Sequential vs. Static: Diagnostic accuracy dropped significantly when moving from full-information to sequential settings. While some models (e.g., Claude-Opus-4.6) retained relative robustness, no model fully compensated for the loss of global context.
• Prompting Effects: Abstention-aware prompting reduced premature diagnoses but increased "missed diagnoses" (failing to diagnose sufficient cases), highlighting a trade-off between safety and timeliness.

4. Impact of Specialization and Reasoning

• Medical Specialization: Models fine-tuned for medicine (MedGemma, Med42) did not outperform generalist models; in fact, they exhibited high under-abstention and lower diagnostic accuracy.
• Reasoning: Increasing reasoning budgets (e.g., in Gemini-2.5-Flash "thinking" mode) improved accuracy slightly but failed to resolve the abstention trade-off, with under-abstention remaining unstable.

Significance and Claims

The paper claims that Safe-Psych reveals a fundamental limitation in current LLMs: the inability to reliably recognize when clinical evidence is incomplete and additional information is needed.

• Safety Implications: The authors argue that a model which commits to a psychiatric diagnosis based solely on presenting symptoms, without waiting for history or examination findings, is unsafe for clinical deployment. The benchmark measures the specific skill of recognizing information gaps, which distinguishes a "clinically useful assistant" from a "confident-sounding one."
• Benchmark Contribution: Safe-Psych is presented as the first benchmark to evaluate psychiatric diagnosis through a sequential, evolving evidence lens, moving beyond static question-answering to assess dynamic decision-making (clarify, diagnose, abstain).
• Modest Scope: The authors acknowledge that the dataset is limited to one Romanian hospital and that translation may introduce noise. They state that while the sequential structure reflects psychiatric workflows, the core question of "recognizing insufficient evidence" may apply to other domains (e.g., emergency triage), though this requires further direct evaluation. The paper concludes that while prompting can shift error types, it does not currently solve the calibration problem, leaving the question of whether training or fine-tuning can improve uncertainty recognition as an open area for research.