Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

1. Problem Statement

The deployment of Large Language Models (LLMs) in mental health care offers expanded access to support but introduces critical safety risks, specifically hallucinations (factually incorrect or fabricated information) and omissions (missing critical therapeutic or crisis intervention information).

• Failure of Current Methods: State-of-the-art "LLM-as-a-Judge" approaches (using models like GPT-4 to evaluate outputs) and traditional metrics (BLEU, ROUGE) fail in high-stakes mental health contexts. The paper demonstrates that leading LLM judges achieve only ~52% accuracy on mental health data, with some hallucination detection approaches showing near-zero recall.
• Root Cause: LLMs lack the nuanced understanding of therapeutic appropriateness, clinical accuracy, and professional communication standards required to detect subtle errors. They often misinterpret necessary therapeutic hedging as uncertainty or fail to identify missing crisis resources (e.g., suicide hotlines).
• Data Scarcity: There is a lack of high-quality, domain-specific datasets with rigorous annotations for mental health hallucinations and omissions.

2. Methodology

The authors propose a hybrid framework that integrates human domain expertise with automated machine learning, moving away from relying solely on black-box LLM judgments.

A. Dataset Creation

• Custom Dataset: A new, rigorously annotated dataset of 4,418 prompt-response pairs for hallucination detection and 3,368 pairs for omission detection.
• Annotation Protocol: Multi-stakeholder approach involving clinicians, administrators, patients, and caregivers. Samples required unanimous consensus among three diverse annotators, with disagreements resolved by a meta-expert.
• Class Imbalance: The dataset reflects real-world deployment scenarios with low positive rates: 1.97% for hallucinations and 3.68% for omissions.
• Human Baseline: A study with 233 stakeholders showed that even trained professionals struggle, with an average human F1 score of 0.536.

B. Feature Extraction Framework

Instead of asking an LLM to directly classify a response as "hallucinated," the framework uses LLMs to extract interpretable, domain-informed features across five analytical dimensions:

1. Logical Consistency: Detects internal contradictions and factual inconsistencies.
2. Entity Verification: Assesses the plausibility of named entities and relationships (e.g., detecting fabricated medication names).
3. Factual Accuracy: Evaluates verifiable claims against established medical knowledge.
4. Linguistic Uncertainty: Quantifies hedging, certainty markers, and epistemic stance.
5. Professional Appropriateness: Evaluates tone, empathy, relevance, and adherence to professional boundaries.

C. Ensemble Learning Architecture

The system employs a two-stage pipeline:

1. Feature Generation: An LLM (used as a feature extractor, not a judge) generates scores for the five dimensions above.
2. Supervised Classification: These features, combined with the raw LLM-as-a-Judge scores, are fed into traditional machine learning classifiers (e.g., LightGBM, XGBoost, Random Forest, MLP).
3. Task-Specific Optimization: Separate models are trained for hallucination detection and omission detection to optimize for their specific feature weightings.

3. Key Contributions

1. Empirical Evidence of Failure: Quantified the systematic failure of vanilla LLM-as-a-Judge methods in mental health (54% accuracy, 0% recall for some metrics like SAC3).
2. Novel Dataset: Introduced a high-quality, multi-stakeholder annotated dataset reflecting realistic class imbalances in mental health AI errors.
3. Domain-Grounded Feature Taxonomy: Developed a framework extracting 15+ interpretable features based on clinical expertise, which black-box judges cannot reliably access.
4. Systematic Omission Detection: Addressed the under-explored problem of detecting missing critical information (e.g., crisis resources), which is often more dangerous than hallucinations in this domain.

4. Experimental Results

The proposed framework was evaluated on both the custom dataset and a public Kaggle mental health dataset.

• Hallucination Detection:
  • Custom Dataset: Achieved F1 = 0.717 (vs. 0.169 for GPT-4o-mini baseline).
  • Kaggle Dataset: Achieved F1 = 0.849 (vs. 0.032 for GPT-4o-mini baseline).
  • Improvement: Up to 75% improvement over baselines.
• Omission Detection:
  • Achieved F1 = 0.637 (Custom) and F1 = 0.591 (Kaggle).
  • Outperformed baselines significantly, which often suffered from catastrophic recall issues.
• Human Comparison: The model's performance (F1 ~0.72) is comparable to or better than the average individual human annotator (F1 = 0.536) while offering greater consistency.
• Model Agnosticism: The framework performed robustly across different LLM backends (GPT-4o, GPT-5, O3, Llama-3.3-70B). Notably, the open-source Llama-3.3-70B achieved F1=0.695, matching proprietary models, suggesting viability for privacy-sensitive, on-premise deployment.
• Ablation Studies: Removing the LLM-as-a-Judge scores and relying only on the 13 custom domain features retained 84.8% of the full model's performance, proving that the domain-informed features carry the primary discriminative signal.

5. Significance and Impact

• Safety in Clinical AI: The work demonstrates that combining human expertise with automated methods yields more reliable and transparent evaluation than black-box LLM judging, which is critical for high-stakes mental health applications.
• Interpretability: By decomposing the "hallucination" problem into specific, interpretable sub-tasks (e.g., entity verification, professional tone), the system provides actionable insights rather than opaque scores.
• Deployment Readiness: The framework produces deployment-ready classifiers that can augment, rather than replace, human clinical oversight. It addresses the specific failure modes of current AI in healthcare, such as missing crisis interventions or fabricating medical advice.
• Future Direction: The paper argues that for specialized domains like healthcare, "prompt engineering" alone is insufficient; explicit domain-guided feature engineering is necessary to bridge the gap between general LLM capabilities and clinical safety requirements.