Development and Evaluation of HopeBot: an LLM-based chatbot for structured and interactive PHQ-9 depression screening

Technical Summary: Feasibility and User Evaluation of HopeBot

Problem Statement
Depression screening relies heavily on the Patient Health Questionnaire-9 (PHQ-9). While clinician-administered interviews allow for the clarification of ambiguous responses and detection of suicidality, they are resource-intensive and difficult to scale. Conversely, self-administered versions are widely adopted for digital screening but lack interactivity, offering no opportunity for users to clarify ambiguous answers or receive real-time guidance. This static format may limit user engagement and scoring accuracy, particularly for individuals with lower health literacy or fluctuating emotional states. Existing chatbot-based screening systems often rely on rigid, rule-based architectures (e.g., predefined intents) that restrict conversational flexibility and the ability to handle nuanced emotional states, or they lack transparent, evidence-grounded reasoning.

Methodology
The authors developed HopeBot, a conversational agent powered by GPT-4o designed to administer the PHQ-9 via text or voice while preserving the validated structure and scoring logic of the original instrument.

• System Architecture:

  • Core Engine: Utilizes GPT-4o for natural language understanding and generation, supporting both English and Mandarin natively without translation.
  • Retrieval-Augmented Generation (RAG): To mitigate hallucination risks and ensure safety, the system employs a RAG layer grounded in four curated data sources: (1) 34 anonymized Cognitive Behavioral Therapy (CBT) session transcripts, (2) A Therapist's Guide to Brief CBT, (3) emotional dialogue datasets (ESConv and PsyQA_example), and (4) bilingual helpline directories from the UK and China.
  • Scoring Logic: The LLM interprets user responses and maps them to PHQ-9 categories (A–D). However, numerical scoring (0–3 per item) and total aggregation are executed deterministically by a backend function, ensuring reproducible scores independent of the language model's generation.
  • Interaction Protocol: The system follows a three-phase structure: (1) rapport building (max 20 turns), (2) sequential PHQ-9 administration with dynamic clarification for ambiguous inputs, and (3) personalized feedback and safety guidance.
  • Safety Mechanisms: The system includes crisis detection to redirect users to validated helplines and operates within a secure, pseudonymized research environment compliant with UK GDPR.

• Study Design:

  • A prospective, within-subject feasibility study was conducted with 132 adults recruited from the UK and China.
  • Procedure: Participants completed a standard self-administered PHQ-9 followed immediately by the HopeBot-assisted PHQ-9.
  • Evaluation: The study measured score concordance between the two formats and collected user feedback via a 25-item survey and open-ended questions. The self-administered PHQ-9 served as a pragmatic comparator for scoring fidelity, not as a clinical reference standard for diagnostic validity.
  • Analysis: Quantitative analysis included Intraclass Correlation Coefficient (ICC), Spearman's rank correlation, and Bland–Altman plots. Qualitative analysis followed Braun and Clarke's thematic framework.

Key Contributions

1. Format-Aligned Administration: HopeBot introduces a methodological framework that preserves the validated structure, sequence, and scoring logic of the PHQ-9 while enabling natural, open-ended conversational interaction.
2. Evidence-Grounded Design: By integrating RAG with authoritative psychological sources, the system provides clarifications and safety guidance that are traceable and grounded, reducing the risk of unsafe or hallucinated outputs common in open-ended LLM interactions.
3. Dynamic Clarification: Unlike prior rule-based systems, HopeBot adaptively interprets free-text responses, issues real-time clarification prompts for ambiguous inputs, and explicitly maps user responses to scoring categories.
4. Bilingual Multimodal Capability: The system supports seamless text and voice interaction in both English and Mandarin, leveraging GPT-4o's native multilingual capabilities and low-latency audio processing.

Results

• Score Concordance: HopeBot-assisted scores showed high agreement with self-administered scores.
  • Intraclass Correlation Coefficient (ICC): 0.92 (95% CI: 0.89–0.94).
  • Median Absolute Difference: 1 point.
  • Identical Scores: 44.7% of participants (59/132) received identical scores from both formats.
  • Bias: No systematic tendency to inflate or reduce severity ratings was found (Wilcoxon signed-rank test, p = 0.649).
  • Subgroup Analysis: Concordance remained high across severity levels and recruitment countries, with only minor, likely clinically insignificant variations.
• User Acceptability:
  • Confidence: Among participants who compared the two formats, 71% (53/75) reported greater confidence in the HopeBot-assisted scores, citing clearer structure, interpretive guidance, and a supportive tone.
  • Usability Ratings: Mean ratings (0–10) were 8.4 for comfort, 7.7 for voice clarity, 7.6 for handling sensitive topics, and 7.4 for recommendation helpfulness.
  • Preference: 69.7% of participants preferred the chatbot-assisted format over self-completion, valuing its engagement and real-time clarification.
  • Limitations: Some users noted generic responses, occasional robotic tone, and privacy concerns regarding AI disclosure. Participants with prior mental health service experience rated the helpfulness of recommendations significantly lower than those without such experience.

Significance and Claims
The paper claims that HopeBot demonstrates the feasibility and acceptability of using an LLM-powered conversational agent with RAG-based grounding to administer the PHQ-9. The high score concordance suggests that such systems can faithfully reproduce the scoring accuracy of self-report tools while adding the benefits of interactivity and clarification.

The authors position HopeBot not as a replacement for clinician judgment or a diagnostic tool, but as a scalable adjunct for early, low-burden depression screening. The study establishes a "format-level validation framework," serving as a methodological bridge between conventional self-assessment and clinician-guided evaluation. The findings suggest that conversational delivery can improve user engagement and perceived structure without compromising the psychometric integrity of standardized screening instruments, potentially extending access to structured screening in resource-limited settings. The authors explicitly state that further validation against clinician-administered assessments in real-world workflows is required to establish diagnostic validity.