Moving Beyond Medical Exams: A Clinician-Annotated Fairness Dataset of Real-World Tasks and Ambiguity in Mental Healthcare

1. Problem Statement

Current benchmarks for evaluating medical Large Language Models (LMs) suffer from three critical limitations, particularly in psychiatry:

• Over-simplification: Most benchmarks rely on multiple-choice board exam questions (e.g., USMLE, MedQA) that test factual recall rather than pragmatic clinical decision-making.
• Lack of Ambiguity: Real-world mental healthcare involves subjective judgments, interpersonal nuances, and scenarios with no single "correct" answer (e.g., triage, documentation, billing). Standard benchmarks often enforce a single ground truth, failing to capture this complexity.
• Fairness and Bias: Existing datasets rarely systematically evaluate how patient demographics (age, gender, ethnicity) influence model decisions. In psychiatry, where diagnosis and treatment rely heavily on subjective interpretation, models may exhibit significant bias against specific demographic groups, a risk that is often overlooked in fact-based evaluations.

2. Methodology: The MENTAT Dataset

The authors introduce MENTAT (MENtal health Tasks AssessmenT), a dataset and evaluation framework designed and annotated entirely by human clinicians without LM assistance.

A. Dataset Design and Scope

• Domains: The dataset covers five critical domains of psychiatric practice:
  1. Diagnosis: Assigning DSM-5-TR diagnoses.
  2. Treatment: Developing treatment plans, including medication dosing.
  3. Monitoring: Assessing treatment efficacy and side effects.
  4. Triage: Determining acuity and escalation levels (e.g., inpatient vs. outpatient).
  5. Documentation: Summarizing clinical encounters and selecting billing codes (CPT).
• Structure: The base dataset contains 203 expert-verified questions (50 diagnosis, 47 treatment, 28 triage, 49 monitoring, 29 documentation).
• Demographic Variable Injection: To study bias, all task-irrelevant demographic information (age, ethnicity, gender) is replaced with variables. The dataset is scaled into four evaluation sets ($D_0, D_G, D_A, D_N$) by varying patient gender, age, and ethnicity across the base questions.
• Handling Ambiguity:
  • Diagnosis, Treatment, Monitoring: Designed with a single correct answer.
  • Triage, Documentation: Designed to be ambiguous with multiple plausible answers. These categories do not enforce a single "correct" fact but rather a "best" answer defined by the creator, acknowledging that reasonable clinicians may disagree.

B. Annotation Pipeline

• Expert Team: A diverse group of nine U.S.-based practicing psychiatrists (MDs) with varied specializations (pediatric, forensic, etc.) created and verified the questions.
• Preference Scoring: For ambiguous categories (Triage/Documentation), eight experts rated each of the five answer options on a 0–100 scale of validity.
• Statistical Modeling:
  • The authors used a Hierarchical Bradley-Terry (HBT) model to convert raw annotation scores into preference probabilities.
  • The HBT model accounts for annotator-specific biases (offsets and slopes) to smooth data while preserving legitimate differences in expert strictness.
  • Krippendorff's $\alpha$ was used to measure inter-annotator disagreement. The authors intentionally accepted low $\alpha$ values (indicating disagreement) as a feature of the dataset, reflecting the inherent ambiguity of the tasks rather than a failure of annotation.
• Post-Processing: Objectively incorrect answers (e.g., factually wrong billing codes) were manually identified and their probabilities set to 0, with remaining probabilities renormalized.

3. Key Contributions

1. First Expert-Curated Ambiguity Dataset: MENTAT is the first dataset to move beyond fact recall to capture the inherent ambiguities of real-world psychiatric decision-making, specifically in triage and documentation.
2. Fairness Evaluation Framework: It provides a systematic method to evaluate how demographic variables (gender, race, age) impact model accuracy across different clinical tasks, revealing biases that are invisible in standard benchmarks.
3. Hierarchical Preference Annotation: The introduction of a preference dataset with uncertainty estimates (via HBT) allows for "soft" labeling, enabling the evaluation of model confidence and alignment with expert uncertainty rather than just binary correctness.
4. Open Source & Reproducibility: The dataset, annotation code, and evaluation pipelines are publicly available under an MIT license.

4. Experimental Results

The authors evaluated 16 off-the-shelf and 6 fine-tuned LMs (including Llama, Gemma, Qwen, Claude, GPT, and medical-specific models like PMC-LLaMA and Meditron).

A. Task-Specific Accuracy

• Performance Gap: Models performed well on Diagnosis (77% avg accuracy) and Treatment (74%), but struggled significantly with Triage (51%) and Documentation (44%).
• Model Comparison: Larger, closed-source models (e.g., GPT-4o, Claude 3.5) generally outperformed smaller open-source models. However, newer open-source models (Qwen3, Gemma3) began to close the gap in ambiguous tasks.
• Fine-Tuning Limitations: Models fine-tuned on synthetic or web-scraped clinical data (e.g., MMedS-Llama-3-8B) did not outperform their base models on MENTAT, suggesting that synthetic data fails to capture the nuance of real-world expert decision-making.

B. Demographic Bias Analysis

The study found statistically significant biases across all tested models and demographic variables:

• Gender: Male-coded patients received higher accuracy than female-coded patients in Monitoring (+10%), Triage (+8%), and Documentation (+10%). Non-binary patients saw the lowest accuracy in Triage (-25% compared to males).
• Ethnicity: "African American" patients received higher accuracy in Diagnosis than "White" patients, while "Native American" patients received higher accuracy in Treatment compared to other groups.
• Age: Younger patients (18–33) received higher accuracy in Diagnosis and Triage, while middle-aged patients (33–49) performed better in Documentation.
• Conclusion: These biases are inconsistent and unpredictable at the individual case level, surfacing only through statistical aggregation, highlighting the danger of deploying models without rigorous fairness testing.

C. Free-Form Consistency

• When models were prompted to generate free-form responses (without multiple-choice options), their answers often deviated significantly from the expert-annotated "correct" options, even when they achieved high multiple-choice accuracy.
• Metrics like 1-BERTScore, 1-ROUGE-L, and 1-BLEU revealed high inconsistency, particularly in Treatment and Triage tasks, indicating that high exam-style scores do not guarantee reliable free-form clinical reasoning.

5. Significance and Impact

• Paradigm Shift: MENTAT challenges the reliance on medical board exams as the gold standard for AI evaluation in psychiatry, arguing that clinical competence requires handling ambiguity and subjective judgment, not just fact retrieval.
• Safety and Ethics: By exposing significant demographic biases and the inability of current models to handle real-world ambiguity, the paper warns against the premature deployment of AI in mental healthcare. It emphasizes the need for "human-in-the-loop" validation.
• Future Research: The dataset provides a foundation for developing fairness-aware models, calibrating model confidence against expert uncertainty, and creating alignment strategies that account for the "gray areas" of psychiatric care.

In summary, MENTAT represents a critical step toward realistic, safe, and fair AI evaluation in mental healthcare, moving the field from "can the model pass a test?" to "can the model make a safe, unbiased, and nuanced clinical decision?"