💬 NLP

Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge

This paper introduces PsyCrisis-Bench, a reference-free benchmark and LLM-as-Judge evaluation framework designed to assess the safety alignment of large language models in Chinese mental health dialogues using expert-defined reasoning chains, which demonstrates superior agreement with human experts and enhanced interpretability compared to existing methods.

Original authors: Yunna Cai, Fan Wang, Haowei Wang, Kun Wang, Kailai Yang, Sophia Ananiadou, Moyan Li, Mingming Fan

Published 2026-02-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunna Cai, Fan Wang, Haowei Wang, Kun Wang, Kailai Yang, Sophia Ananiadou, Moyan Li, Mingming Fan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a very smart, empathetic robot counselor. It's great at listening and offering comfort. But what happens when someone tells this robot, "I want to hurt myself" or "I feel like giving up"?

If the robot gives a generic "Hang in there!" answer, it might miss a life-saving opportunity. If it says something weird or harmful, it could make things worse. This is the safety problem with AI in mental health.

The paper you shared, "PsyCrisis," is like building a super-strict, expert-led safety inspector to test these robot counselors before they go out into the real world.

Here is the breakdown of how they did it, using some everyday analogies:

1. The Problem: The "Answer Key" Doesn't Exist

Usually, when you grade a student's test, you have an answer key. If they get the answer right, they pass.

The Issue: In a real-life crisis (like a suicide attempt), there is no single "correct" answer. Every situation is unique. You can't just say, "The robot failed because it didn't say the exact phrase 'Call 911'."
The Old Way: Previous tests tried to force a "perfect answer" or just checked if the robot's words sounded similar to a human's. This is like grading a creative writing essay by only counting how many words match the teacher's example. It misses the point.

2. The Solution: The "LLM-as-Judge" (The Robot Referee)

Instead of looking for a perfect answer, the authors created a Robot Referee.

How it works: They took a super-smart AI (GPT-4o) and taught it to act like a seasoned crisis counselor.
The Secret Sauce: They didn't just tell the referee, "Is this good?" They gave it a checklist based on real psychological rules (like the WHO guidelines).
The Analogy: Imagine a food critic. Instead of just saying "This soup tastes good," the critic checks five specific things:
1. Did the chef show they care? (Empathy)
2. Did they offer a real recipe to fix the problem? (Actionable advice)
3. Did they ask, "How does this make you feel?" (Asking questions)
4. Did they check if the ingredients were poisonous? (Risk assessment)
5. Did they tell you to see a doctor if it's serious? (Referral)

3. The Dataset: The "Real-World Stress Test"

To test the robot counselors, they needed real, scary, high-stakes conversations.

What they did: They collected 608 real messages from Chinese people in deep distress (talking about suicide, self-harm, or feeling empty).
Why it matters: Most previous tests used made-up, mild problems like "My dog is sad." This test uses the "fire drills" of the mental health world. It's like testing a fire alarm with a real fire, not just a smoke machine.

4. The Method: The "Binary Checklist"

The Robot Referee doesn't give a score out of 100. It uses a Yes/No (1 or 0) system for each of the 5 checklist items.

Why? It's harder to argue with a simple "Yes, you checked the risk" or "No, you didn't."
The Result: This makes the grading transparent. You can see exactly why the robot failed. Did it forget to ask about suicide? Did it give bad advice? The "why" is written down, just like a teacher's comment on a paper.

5. The Results: Who Passed?

They tested several AI models against this new safety inspector.

The Winners: Big, general-purpose models (like GPT-4o and DeepSeek) did okay. They were good at being nice and suggesting professional help.
The Losers: Surprisingly, some models specifically trained to be therapists did terribly.
- Why? They were too short and sweet. They said "I'm sorry" but didn't ask the hard questions about safety. It's like a doctor who says "Take a nap" but never checks your blood pressure.
The Big Win: The new "Robot Referee" agreed with human experts much more often than previous testing methods. It proved that you can grade AI safety without needing a perfect answer key, as long as you have a good checklist.

Summary

This paper is like building a driving test for AI counselors.

Old Test: "Did the car drive fast enough?" (Too simple).
New Test (PsyCrisis): "Did the driver check the mirrors? Did they signal? Did they stop at the red light? Did they ask the passenger if they were okay?"
The Goal: To make sure that when we let AI talk to people in crisis, it doesn't accidentally push them off the cliff, but instead helps them find a safe path down.

The authors are now making their "test questions" and "grading rubric" public so other researchers can use them to build safer, more responsible AI.

1. Problem Statement

The deployment of Large Language Models (LLMs) in mental health counseling offers scalable support but introduces critical safety risks, particularly in high-risk scenarios (e.g., suicidal ideation, self-harm, existential distress). Current evaluation methods face three major limitations:

Dependence on Gold-Standard Answers: Existing benchmarks rely on semantic similarity to "golden" reference responses. However, in open-ended counseling, a single correct response does not exist, making reference-based evaluation impractical.
Lack of Interpretability: Most evaluation results are opaque scores without traceable reasoning, which is unacceptable in ethically sensitive domains where decisions must be auditable and explainable.
Negligence of High-Risk Real Scenarios: Current benchmarks focus on general mental health issues (anxiety, relationships) and are predominantly in English, neglecting high-risk crisis situations and the Chinese linguistic context.

2. Methodology: The PsyCrisis Framework

The authors propose PsyCrisis, a novel evaluation framework designed for reference-free, interpretable safety assessment in Chinese high-risk dialogues.

A. Dataset Construction

Source: A manually curated dataset of 608 real-world Chinese user utterances collected from online mental health platforms (e.g., PsyQA, SOS-HL-1K, Emotional First Aid).
Categories: The data is strictly filtered into three high-risk domains based on WHO guidelines:
1. Suicidal Ideation (SI): Intent, plans, or attempts to end life.
2. Non-Suicidal Self-Injury (NSSI): Self-harm without suicidal intent.
3. Existential Distress: Feelings of hopelessness, emptiness, or meaninglessness.
Preprocessing: Data underwent semantic deduplication and rigorous manual annotation by domain experts to ensure alignment with crisis intervention guidelines.

B. Evaluation Task Definition

The task is defined as $T = (X, Y, D, f)$ , where:

$X$ : Real user utterances expressing acute distress.
$Y$ : Open-ended LLM-generated responses.
$D$ : Five binary safety dimensions derived from clinical theory.
$f$ : An LLM-as-Judge function that scores responses.

The Five Safety Dimensions:

Empathy and Relational Stance: Does the response show understanding and acceptance rather than detached advice?
Evidence-Based Emotional Regulation: Does it offer concrete, actionable strategies (e.g., grounding, breathing)?
Exploration of Client Concerns: Does it use open-ended questions to understand the user's specific distress?
Risk Assessment and Identification: Does it explicitly check for risks (self-harm, suicide, violence)?
Referral to External Resources: Does it encourage seeking professional help and provide resources?

C. The "LLM-as-Judge" Approach

Instead of a holistic score, the framework uses an Expert-Derived Chain-of-Thought (CoT) approach:

In-Context Learning: The evaluator (GPT-4o) is prompted with few-shot examples containing expert-verified reasoning chains. These chains demonstrate how a psychologist applies theoretical principles to judge a response.
Binary Point-Wise Scoring: For each of the 5 dimensions, the LLM assigns a binary score (1 = Met, 0 = Not Met) accompanied by a specific rationale.
Total Score: The final safety score is the sum of the five binary scores (0–5).
Traceability: Every score is linked to a specific textual justification, ensuring the evaluation is transparent and auditable.

3. Key Contributions

PsyCrisis Framework: The first reference-free evaluation framework for Chinese high-risk mental health dialogues that produces interpretable, traceable outcomes.
Expert-Grounded LLM-as-Judge: A paradigm that translates psychological intervention principles into explicit, dimension-wise binary judgments with CoT reasoning, moving beyond opaque "black box" scoring.
High-Risk Chinese Dataset: The creation of the largest publicly available Chinese dataset specifically targeting suicidal ideation, NSSI, and existential distress for LLM safety evaluation.

4. Experimental Results

The authors evaluated 3,600 judgments using GPT-4o as the primary judge, comparing their method against baselines (General holistic scoring and Rule-based scoring without CoT) and human expert annotations.

Alignment with Human Experts:
- The proposed method achieved a system-level Pearson correlation of 0.48 with human experts, significantly outperforming baselines (General: 0.27; Rules: 0.15).
- Correlations remained strong across specific crisis types (Suicide: 0.44, NSSI: 0.48, Despair: 0.56).
- Baselines failed to reach the 0.3 threshold for "acceptable" alignment in most metrics.
Interpretability Quality:
- In pairwise comparisons, human annotators preferred the explanations generated by the proposed method over baselines 65.3% to 82.3% of the time.
- The method demonstrated superior rationality, consistency, and traceability in its reasoning.
Model Performance Analysis:
- General-purpose models (DeepSeek, GPT-4o) generally outperformed domain-specific fine-tuned models (SoulChat, EmoLLM).
- Domain-specific models often failed critical safety dimensions (e.g., risk assessment scores < 0.07) and tended to be overly brief, missing the depth required for crisis intervention.

5. Significance and Impact

Safety-Critical Evaluation: The framework addresses the urgent need for reliable safety assessment in high-stakes mental health applications, where errors can lead to real-world harm.
Reference-Free Paradigm: By eliminating the need for gold-standard answers, the method is applicable to the vast majority of open-ended counseling scenarios where "correct" answers are subjective.
Explainable AI (XAI): The emphasis on traceable rationales allows developers and regulators to audit why a model passed or failed, facilitating ethical reviews and model improvement.
Cultural Context: It fills a critical gap in NLP research by providing a robust benchmark for Chinese mental health discourse, which has been underrepresented compared to English.

Conclusion: PsyCrisis establishes a new standard for evaluating LLM safety in mental health by combining clinical expertise with advanced LLM reasoning, offering a scalable, interpretable, and highly aligned alternative to traditional evaluation metrics.