🤖 AI

Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing

This paper demonstrates that in mental health AI safety testing, expert human feedback exhibits systematic and principled disagreement rather than random error, challenging the assumption of a valid ground truth and suggesting that safety-critical AI evaluation must shift from consensus-based aggregation to methods that preserve diverse professional perspectives.

Original authors: Kiana Jafari, Paul Ulrich Nikolaus Rust, Duncan Eddy, Robbie Fraser, Nina Vasan, Darja Djordjevic, Akanksha Dadlani, Max Lamparth, Eugenia Kim, Mykel Kochenderfer

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kiana Jafari, Paul Ulrich Nikolaus Rust, Duncan Eddy, Robbie Fraser, Nina Vasan, Darja Djordjevic, Akanksha Dadlani, Max Lamparth, Eugenia Kim, Mykel Kochenderfer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Expert Consensus" Myth

Imagine you are trying to teach a robot how to be a helpful therapist. The standard way to do this is to ask human experts (psychiatrists) to grade the robot's answers. If the experts agree that an answer is "good," the robot learns to do more of that. If they say it's "bad," the robot learns to avoid it.

This paper asks a simple but scary question: What if the experts can't agree?

The researchers tested this in the high-stakes world of mental health. They hired three board-certified psychiatrists to grade 360 different AI-generated responses to mental health questions. They expected the experts to mostly agree because they all went to the same schools and follow the same medical rules.

The Result: The experts barely agreed at all. In fact, their disagreement was so high that it would be considered "unreliable" in almost any other scientific field.

The Analogy: The Three Judges

To understand what happened, imagine a cooking competition with three judges:

Judge Safety: Only cares if the food is poisonous. If it's not toxic, they give it a 5-star rating, even if it tastes like cardboard.
Judge Engagement: Cares if the food is delicious and makes you want to eat more. If it's safe but boring, they give it a 2-star rating.
Judge Culture: Cares if the food respects the diner's background and dietary restrictions. If the food ignores cultural context, they give it a 1-star rating, even if it's safe and tasty.

In this study, the three psychiatrists acted like these three judges. They weren't making mistakes or misreading the instructions. They were looking at the same AI answer and seeing three completely different things because they had different "philosophies" about what a good response should be.

The Key Findings

1. The "Safety" Paradox
You might think experts would agree the most on dangerous topics like suicide or self-harm. You'd be wrong.

The Paper's Claim: The experts disagreed the most on the most dangerous topics (suicide and self-harm).
The Analogy: Imagine a fire drill. One expert thinks, "Don't panic, just walk out slowly." Another thinks, "Run immediately!" A third thinks, "Call 911 first." They all want to save lives, but they have different ideas on how to do it. Because the stakes are so high, their differences became huge.

2. The "Noise" is Actually "Signal"
When AI systems are trained, they usually take all the expert scores and average them out. If Judge A gives a 5 and Judge C gives a 1, the AI learns a "3."

The Paper's Claim: This averaging process is broken. It creates a "compromise" answer that doesn't actually reflect any real expert's opinion. It's like mixing red paint and blue paint to get purple, then telling the artist, "This is the perfect color for your sky," when neither artist wanted purple.
The Result: The AI learns a "middle ground" that might be therapeutically useless because it ignores the specific, valid reasons why each expert gave their score.

3. The "Boundaries" Problem
The experts disagreed the most on whether the AI should clearly state, "I am a robot, not a doctor."

The Paper's Claim: One expert (Expert C) was extremely strict, giving almost every answer a failing grade because the AI didn't explicitly say "I am not a human." Another expert (Expert A) thought it was fine if the AI just suggested seeing a real doctor later.
The Analogy: It's like a teacher grading a student's essay. One teacher fails the essay because the student didn't write their name at the top. The other teacher gives an A because the essay is brilliant. If you average those grades, you get a B, which doesn't tell the student anything useful.

Why This Matters for AI

The paper argues that we cannot simply "average out" human disagreement to find the "truth."

The Problem: In mental health, there is no single "correct" answer that can be measured like a blood test. What is helpful for one person might be harmful to another.
The Consequence: If we train AI by averaging these conflicting expert opinions, we aren't teaching the AI to be safe or helpful; we are teaching it to be a confused middle-ground that might not fit anyone's needs.

The Paper's Recommendations

The authors don't say we should stop using experts. Instead, they suggest we change how we use them:

Stop pretending everyone agrees: We need to admit that experts have different, valid philosophies.
Don't just average: Instead of averaging scores, we should keep the scores separate and tell the AI, "Expert A thought this was safe, but Expert B thought it was risky."
Use disagreement as a warning sign: If experts are fighting over an answer, that's a signal that the AI needs a human to step in and check it before showing it to a real person.

Summary

This paper is a reality check. It shows that even highly trained experts can't agree on what a "good" mental health response looks like. Because they disagree so much, the current method of training AI by averaging their opinions is flawed. The "noise" we thought we could ignore is actually a deep, principled disagreement about how to care for people, and we need new ways to handle that complexity.

Technical Summary: Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing

Problem Statement

The prevailing paradigm for developing and evaluating AI systems, particularly in high-stakes domains like mental health, relies on Learning from Human Feedback (LHF). This approach assumes that aggregating expert judgments yields a valid "ground truth" signal for training reward models (e.g., RLHF, DPO) and safety classifiers. However, this assumption is rarely tested, and when reliability is measured, it is often unreported.

In mental health, the stakes are exceptionally high: inappropriate AI responses can lead to fatality, reinforcement of delusions, or perpetuation of stigma. Evaluation typically depends on a small number of specialists rating AI outputs. The core problem investigated is whether these experts can actually reach a consistent consensus. If experts hold fundamentally different clinical philosophies, aggregating their labels may not reduce noise but rather erase grounded professional frameworks, creating "arithmetic compromises" that do not reflect any coherent clinical reality.

Methodology

The authors employed a sequential mixed-methods design combining quantitative reliability analysis with qualitative investigation of clinical frameworks.

Dataset Construction

Content: A dataset of 360 AI-generated responses to synthetic mental health prompts.
Models: Responses were generated by four large language models: GPT-5, Claude 4 Sonnet, Grok-4, and Llama 3.2.
Design: Prompts were systematically varied using a 3×3 matrix crossing clinical severity (low, moderate, high) with communication directness.
Risk Categories: Ten high-risk categories were defined (e.g., supporting suicidal ideation, reinforcing hallucinations, punishing ADHD behaviors), derived from clinical guidelines and prior work on harmful AI responses.

Evaluation Framework

Rubric: A calibrated rubric covering two dimensions (Safety and Quality) across eight factors:
- Safety: Severity of potential harm, Likelihood of potential harm.
- Quality: Correctness, Relevance, Active listening, Empathy & validation, Boundaries & ethical disclaimers, Actionability & encouragement.
Scale: 5-point Likert scales (1 = worst, 5 = best).

Participants and Procedure

Raters: Three board-certified psychiatrists with diverse subspecialties (adult, child/adolescent), theoretical orientations (cultural, biopsychosocial), and patient populations.
Process:
1. Calibration: A 90-minute session reviewing scales and anchor points.
2. Annotation: Independent rating of all 360 responses across all 8 factors (totaling 8,640 ratings).
3. Interviews: Semi-structured interviews (45–60 mins) to explore evaluation processes and underlying frameworks.
Analysis:
- Quantitative: Intraclass Correlation Coefficients (ICC) for reliability and Krippendorff's $\alpha$ for ordinal data. Mean Absolute Deviation (MAD) to measure disagreement magnitude.
- Qualitative: Reflexive thematic analysis of interview transcripts to explain quantitative patterns.

Key Results

1. Poor Inter-Rater Reliability

Despite similar training and shared instructions, inter-rater reliability was consistently poor, falling below the 0.40 threshold typically considered minimally acceptable for consequential assessment.

Single-Rater Reliability ( $ICC_{(2,1)}$ ): Ranged from 0.087 (Boundaries) to 0.295 (Active listening).
Averaged Reliability ( $ICC_{(2,k)}$ ): Even when averaging all three raters, reliability only reached "fair" levels (max 0.556), with "Boundaries" remaining in the "poor" range (0.223).
Krippendorff's $\alpha$ : Four factors yielded negative values (e.g., Boundaries $\alpha = -0.203$ ), indicating systematic disagreement worse than chance. This suggests that when one rater scored high, others systematically scored low.

2. The High-Stakes Paradox

Disagreement was highest on the most safety-critical items, contradicting the assumption that experts align better on critical issues.

Suicide and Self-Harm: Produced the greatest disagreement (MAD 0.598 and 0.566, respectively).
ADHD: Showed the lowest disagreement (MAD 0.461), likely because guidelines are less prescriptive.
Severity: High-severity cases generated more disagreement than low or moderate cases.

3. Systematic, Not Random, Disagreement

Disagreement was directional and reflected coherent but incompatible clinical frameworks rather than measurement error.

Rater Asymmetry: A 3.5:1 asymmetry was observed between raters. Rater A (mean 4.186) was consistently more lenient than Rater C (mean 3.639).
Framework-Driven Divergence:
- Expert A (Safety-First): Prioritized harm avoidance; rated responses higher if they avoided active harm, even if therapeutically limited.
- Expert B (Engagement-Centered): Focused on therapeutic connection and help-seeking; penalized "safe" but cold responses.
- Expert C (Culturally-Informed): Applied the strictest standards, emphasizing cultural context and explicit AI limitations; rated 92% of responses as inadequate on "Boundaries" because they lacked explicit disclaimers.
Boundaries Factor: The lowest reliability factor ( $\alpha = -0.203$ ) exemplified this. Expert C required explicit "I am AI" disclaimers, while Expert A accepted implicit boundaries through referrals. Aggregating these views produced a mean that reflected neither expert's actual standard.

4. Model-Specific Patterns

Disagreement varied by the source model. GPT-5 showed the highest disagreement on "Boundaries" (MAD 0.921), while Grok-4 showed the highest overall disagreement, particularly on interpersonal dimensions. This indicates that reliability estimates from one model do not necessarily generalize to others.

Key Contributions

Existence Proof of Irreducible Disagreement: The study demonstrates that even among board-certified psychiatrists, expert disagreement on AI safety is substantial, systematic, and persists despite calibration. This challenges the assumption that human feedback provides a stable ground truth for RLHF and safety classification.
Identification of Systematic Bias: The paper shows that disagreement is not random noise but reflects competing clinical philosophies (safety-first vs. engagement vs. cultural context). These differences are most pronounced in high-stakes scenarios (suicide, self-harm).
Qualitative Mechanism: Through interviews, the authors identify that experts rely on holistic risk heuristics rather than granular factor discrimination. The "noise" current alignment methods seek to average away is actually a contested signal representing valid, divergent professional frameworks.

Significance and Implications

The paper argues that in safety-critical domains with longitudinal consequences, human feedback may not provide the stable ground truth assumed by current AI alignment methods.

For Reward Modeling: Training reward models on aggregated labels may teach models to mimic "arithmetic compromises" rather than genuine clinical quality, potentially leading to therapeutically incoherent behaviors.
For Safety Classification: Classifiers trained on expert labels in high-stakes domains (e.g., suicide risk) may inherit contested judgments as ground truth, creating false confidence in safety.
For Evaluation Benchmarks: Benchmarks treating human ratings as objective truth are fundamentally flawed if experts cannot agree on what constitutes a "good" response.

Recommendations

The authors propose shifting from consensus-based aggregation to methods that preserve and learn from expert disagreement:

Report Reliability: Inter-rater reliability must be reported in all evaluation studies.
Framework-Declared Annotation: Experts should declare their clinical framework prior to annotation, creating labeled distributions indexed by framework rather than collapsed scores.
Disagreement as Escalation Signal: High disagreement should trigger human-in-the-loop review or conservative system behavior, rather than being absorbed into an aggregate.
Context-Aware Weighting: For specific use cases, weighting annotators whose frameworks match the deployment context may be necessary, though dynamic framework selection remains an open challenge.

The study concludes that expert disagreement in safety-critical AI is a sociotechnical phenomenon where professional experience introduces sophisticated layers of principled divergence that cannot be resolved through better measurement or larger annotator pools.