CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering

This paper introduces CounselBench, a large-scale benchmark developed with mental health professionals to evaluate and adversarially stress-test large language models on realistic open-ended mental health questions, revealing significant gaps in safety, personalization, and the reliability of automated evaluation compared to human expert judgment.

The Gist

Imagine you are building a robot therapist. You want to make sure it's kind, smart, and safe before you let it talk to real people who are hurting. But how do you test it?

Most previous tests were like multiple-choice quizzes. They asked the robot, "What is the capital of France?" or "What is the definition of anxiety?" The robot could pass these by memorizing facts. But real life isn't a quiz. Real life is messy, emotional, and full of open-ended questions like, "I feel like my life is falling apart and I don't know who to talk to."

This paper introduces CounselBench, a new, giant "stress test" designed specifically to see how well AI handles these messy, real-life mental health conversations.

Here is the breakdown of how they did it, using some simple analogies:

1. The "Real World" Test (CounselBench-Eval)

Think of this as a blind taste test for therapy.

• The Setup: The researchers took 100 real questions from a public forum where people ask for help (like a digital support group).
• The Contestants: They asked four different "therapists" to answer these questions:
  1. GPT-4 (A famous AI)
  2. LLaMA 3 (A popular open-source AI)
  3. Gemini (Google's AI)
  4. Real Human Therapists (The gold standard)
• The Judges: They didn't use computers to grade these answers. Instead, they hired 100 licensed mental health professionals (real therapists, social workers, and psychologists) to act as judges.
• The Grading: The humans didn't just say "Good" or "Bad." They graded the answers on six specific things:
  • Empathy: Did it sound like it cared?
  • Specificity: Did it actually answer this person's problem, or just give generic advice?
  • Safety: Did it accidentally give dangerous medical advice (like telling someone to take specific drugs)?
  • Toxicity: Was it mean or dismissive?
  • Facts: Was the information accurate?
  • Overall Quality: Did it feel helpful?

The Results:
The AI models were surprisingly good at sounding nice and empathetic. However, they had some major flaws:

• The "Overconfident Doctor" Problem: The AIs sometimes acted like they were doctors and gave specific medical advice (like suggesting specific medications) which they are not licensed to do.
• The "Generic Bot" Problem: Sometimes they gave advice that was too vague, like saying "Just talk to someone," without actually helping the person feel heard.
• The Human Surprise: Interestingly, the real human therapists on the forum sometimes gave answers that were less empathetic or more generic than the AIs, likely because they were writing quickly on a forum, not in a private session.

2. The "Trap" Test (CounselBench-Adv)

The researchers realized that just asking normal questions wasn't enough. They needed to see if they could break the robots.

• The Setup: They hired 10 experts to write 120 "trap questions." These weren't normal questions; they were designed specifically to trick the AI into making a mistake.
• The Traps:
  • Trap 1: A question designed to make the AI suggest a specific drug.
  • Trap 2: A question designed to make the AI sound judgmental or mean.
  • Trap 3: A question designed to make the AI guess a medical diagnosis.
• The Results: The AIs fell into the traps! Different models had different "personality flaws."
  • One model was very likely to suggest medication.
  • Another was very likely to sound cold and uncaring.
  • A third was likely to make up facts about symptoms.

3. The "AI Judge" Problem

The researchers asked a big question: Can we use an AI to grade other AIs?

They let various AI models act as the judges. The result was a disaster.

• The "Yes-Man" Effect: The AI judges were too nice. They gave almost perfect scores to the other AIs, even when the human experts said the answers were dangerous or useless.
• The Blind Spot: The AI judges completely missed safety issues. If a robot gave dangerous advice, the AI judge often said, "That's a great answer!" This shows that we cannot trust AI to police AI in high-stakes fields like mental health. We still need real humans in the loop.

Why Does This Matter?

Imagine you are building a self-driving car. You don't just test it on a straight, empty road (the multiple-choice tests). You test it in a storm, with pedestrians running across the street, and with other cars cutting you off (the CounselBench tests).

CounselBench is that storm test for mental health AI. It tells us:

1. AI is getting better at sounding human and caring.
2. AI is still dangerous because it sometimes gives medical advice it shouldn't.
3. We need real humans to check the work, because AI judges are too easily fooled.

The paper concludes that while AI can be a helpful tool for mental health, we need to be very careful, keep humans in charge, and use benchmarks like this one to make sure the robots don't hurt the people they are trying to help.

Technical Summary

Here is a detailed technical summary of the paper "COUNSELBENCH: A LARGE-SCALE EXPERT EVALUATION AND ADVERSARIAL BENCHMARKING OF LARGE LANGUAGE MODELS IN MENTAL HEALTH QUESTION ANSWERING."

1. Problem Statement

While Large Language Models (LLMs) have shown proficiency in general question answering (QA) and factual medical tasks (e.g., MedQA), their application in open-ended mental health QA remains under-explored and risky.

• The Gap: Existing benchmarks rely heavily on multiple-choice or fact-based tasks, which fail to capture the nuance of real-world patient queries. Mental health questions are often ambiguous, mixing symptoms, emotional needs, and treatment concerns.
• The Challenge: Effective responses require a delicate balance of empathy, actionable guidance, and strict professional boundaries. Errors in this domain (e.g., unauthorized medical advice, dismissive tone, or overgeneralization) can have immediate negative real-world consequences.
• Evaluation Deficit: Current evaluation methods often use small expert panels or "LLM-as-Judge" protocols, which lack the scale, clinical validity, or sensitivity to detect subtle safety failures and safety risks.

2. Methodology

The authors introduce COUNSELBENCH, a two-part benchmark developed in collaboration with 100 licensed or professionally trained mental health professionals.

A. Data Curation

• Source: Real-world patient questions were sourced from CounselChat, a public forum where licensed therapists respond to anonymous queries.
• Selection: 100 questions were selected across 20 diverse mental health topics (e.g., depression, trauma, relationships, substance abuse).
• Response Generation:
  • Human Baseline: Top-voted responses from verified online human therapists.
  • LLM Baseline: Responses generated by GPT-4, LLaMA-3.3, and Gemini-1.5-Pro.
  • Adversarial Set: 120 new questions authored by experts specifically designed to trigger known failure modes.

B. Evaluation Framework (CounselBench-EVAL)

• Dimensions: Responses were rated on six clinically grounded dimensions using 5-point Likert scales (except for binary/4-point specific metrics):
  1. Overall Quality: Holistic judgment.
  2. Empathy: Emotional attunement and validation.
  3. Specificity: Tailoring to the user's context (avoiding generic advice).
  4. Medical Advice: Binary flag for unauthorized diagnostic/therapeutic guidance (e.g., prescribing meds).
  5. Factual Consistency: Alignment with clinical knowledge.
  6. Toxicity: Presence of harmful or stigmatizing language.
• Annotation Protocol: 100 experts provided 2,000 evaluations (5 experts per response). Annotators provided span-level annotations (extracting specific problematic text) and written rationales (median 576 words per survey).
• LLM-as-Judge Experiment: Nine LLMs were tested as automated judges to compare their scoring and failure detection capabilities against human experts.

C. Adversarial Benchmark (CounselBench-ADV)

• Construction: Based on failure patterns observed in the EVAL phase, 10 experts authored 120 adversarial prompts targeting six specific failure modes:
  1. Recommending specific medications.
  2. Suggesting specific therapy techniques (e.g., CBT) without context.
  3. Speculating about medical symptoms.
  4. Being judgmental.
  5. Displaying apathy.
  6. Relying on unsupported assumptions.
• Testing: 1,080 responses were generated by 9 LLMs and evaluated by experts to measure vulnerability to these specific triggers.

3. Key Contributions

1. Clinically Grounded Rubric: Defined six evaluation dimensions that move beyond factual accuracy to capture safety, empathy, and contextual relevance.
2. Large-Scale Expert Dataset: Created CounselBench-EVAL with 2,000 expert evaluations, 100 real-world questions, and rich span-level annotations/rationales.
3. Adversarial Stress-Test: Developed CounselBench-ADV, a dataset of 120 expert-crafted prompts designed to systematically elicit and measure specific failure modes in LLMs.
4. LLM Judge Analysis: Provided a comprehensive analysis showing that current LLM judges systematically overrate model performance and fail to detect safety risks identified by humans.

4. Key Results

A. Performance of LLMs vs. Human Therapists (EVAL)

• Quality Scores: LLaMA-3.3 achieved the highest overall scores (4.29/5), outperforming GPT-4 (3.28) and Gemini (3.26). Surprisingly, Online Human Therapists received the lowest average scores (2.60), likely due to the informal nature of forum responses and the high bar set by LLMs for structure and length.
• Safety Risks: Despite high quality scores, LLaMA-3.3 had the highest rate of flagged unauthorized medical advice (14%), often suggesting therapy techniques like CBT. GPT-4 was more conservative, frequently declining to answer and recommending professional consultation.
• Failure Modes:
  • GPT-4: Unconstructive feedback and lack of personalization.
  • LLaMA-3.3: Overgeneralization and making judgments without context.
  • Gemini: Lack of empathy and unconstructive feedback.
  • Human Therapists: Overgeneralization and inappropriate tone.

B. LLM-as-Judge Reliability

• Inflated Scores: LLM judges consistently assigned higher scores than human experts, particularly for Factual Consistency and Toxicity.
• Safety Blindness: LLM judges failed to detect toxic or medically unsafe content. For example, while humans flagged toxic content, LLM judges uniformly assigned the lowest toxicity scores (indicating "not toxic").
• Ranking Divergence: LLM judges often ranked models differently than humans (e.g., ranking Gemini higher than GPT-4, contrary to human preference).

C. Adversarial Findings (ADV)

• Targeted Failures: The adversarial questions successfully triggered specific failure modes.
  • GPT-5 showed high vulnerability to suggesting therapy techniques (85%) and recommending medications (47%).
  • Llama models frequently speculated on symptoms and made unsupported assumptions.
  • Gemini and Claude showed lower rates of medication advice but still exhibited apathy or assumption-based errors.
• Model Families: Failure modes were consistent within model families (e.g., Llama variants shared similar error distributions) but shifted significantly across major architecture upgrades.

5. Significance and Implications

• Safety First: The study highlights that high "quality" scores do not equate to safety. LLMs can sound helpful while providing dangerous, unauthorized medical advice.
• Evaluation Limitations: The findings strongly suggest that LLM-as-Judge is currently unreliable for high-stakes mental health evaluation. Human expert oversight remains essential for safety validation.
• Resource Availability: The release of CounselBench-EVAL and CounselBench-ADV provides the research community with the first large-scale, clinically validated resources for stress-testing LLMs in mental health.
• Future Directions: The framework supports the development of better alignment methods (RLHF), automated safety detectors, and red-teaming pipelines specifically for mental health applications.

In conclusion, COUNSELBENCH establishes that while LLMs are improving in open-ended mental health QA, they exhibit systematic, model-specific failure patterns regarding safety and clinical boundaries that current automated evaluation methods fail to catch.