Training anamnesis interviews with a chatbot virtual patient and integrated feedback: healthcare students’ perspectives

Technical Summary: Training Anamnesis Interviews with a Chatbot Virtual Patient and Integrated Feedback

Problem Statement
Effective anamnesis (history taking) is a fundamental clinical competency across all health professions, yet training opportunities with real or standardized patients are resource-intensive, difficult to scale, and often inaccessible in online learning environments. Traditional methods, such as standardized patients (SPs) or peer role-plays, face limitations regarding scheduling, cost, and scalability. While Large Language Models (LLMs) have shown promise in creating virtual patients (VPs) for medical students, existing evidence is heavily skewed toward medical education. Furthermore, the systematic integration of automated feedback targeting both professional and communication competencies across diverse health professions remains underexplored. There is a specific gap in understanding how non-medical health students perceive LLM-powered VPs and whether these systems can effectively support formative learning through integrated, automated feedback.

Methodology
The study employed a cross-sectional design conducted at IU International University of Applied Sciences between January and March 2025.

• Participants: 60 undergraduate students from eight health-related Bachelor's programs (including nursing, physiotherapy, occupational therapy, speech therapy, and nutritional science). The cohort was predominantly female (83.3%) with a mean age of 37.0 years; most had prior professional healthcare training and work experience.
• Intervention: Participants engaged in an anamnesis interview with a chatbot-based Virtual Patient (VP) powered by GPT-4. The scenario involved a patient ("Karl von Hausen") with a brain hemorrhage, multimorbidity (hypertension, obesity, urinary diversion), and language disorders, simulated alongside his wife. The interaction was conducted via the university's Guided Conversation Designer (GCD) platform.
• Feedback Mechanism: Immediately following the interview, the system generated automated feedback addressing two domains: professional competence (accuracy, completeness, question depth tailored to the student's profession) and communication competence (clarity, empathy, interaction skills).
• Data Collection: Participants completed an online survey comprising:
  • Demographic and background data.
  • The Chatbot Usability Questionnaire (CUQ) for usability assessment.
  • Purpose-developed, pilot-tested 5-point Likert scales measuring response quality, realism, behavioral engagement, feedback quality, and training benefit.
  • Open-ended questions for qualitative feedback.
• Analysis: Quantitative data were analyzed using descriptive statistics, Cronbach's alpha for internal consistency, and Positive Response Proportions (PRP). Qualitative open-ended responses underwent content analysis following Mayring's framework.

Key Results

• Usability: The system received high usability ratings (CUQ mean score: 76.93/100). However, error handling was a weak point, with some participants noting the chatbot appeared "too robotic" or complex.
• Response Quality & Realism: Response quality was rated highly (Cronbach's α = 0.90), with 95% of participants finding the responses credible. Realism received moderate ratings (Cronbach's α = 0.87); while 86% felt the VP used terms like a real patient, only 52% felt the VP reacted unpredictably like a real patient.
• Behavioral Engagement: Despite limited perceived naturalness, 95% of students reported acting respectfully toward the VP, and 83% felt they acted professionally.
• Feedback Quality: This dimension received the highest ratings across all scales (Cronbach's α = 0.86). 97% of participants agreed the feedback was related to the content of the anamnesis, and 82% viewed it as necessary for learning success.
• Educational Value: The training was appraised as highly beneficial (Cronbach's α = 0.89). 85% believed it enhanced professional competence, and 83% felt it improved communication skills.
• Qualitative Insights: Students valued the safe, consequence-free environment for practicing. The most requested improvements included more unpredictable patient behavior, multimodal interaction (e.g., voice, visual avatars), and richer clinical context.

Key Contributions

1. Extension Beyond Medical Education: The study provides empirical evidence that LLM-powered VPs with integrated feedback are valuable not only for medical students but also for diverse health professions, including nursing, physiotherapy, and nutritional science.
2. Integrated Feedback Validation: It demonstrates the feasibility and high perceived value of automated, AI-generated feedback that simultaneously addresses professional and communication competencies, a feature often lacking in pre-LLM simulation systems.
3. Complex Scenario Handling: The system successfully simulated a complex case involving multimorbidity and a family member, suggesting AI can support training for clinically challenging scenarios beyond simple, single-pathology cases.
4. Behavioral Findings: The research indicates that high levels of professional and empathic behavior can be elicited from students even when the simulation lacks full emotional realism, suggesting that high-fidelity immersion is not strictly necessary for formative skill development.

Significance and Claims
The paper claims that AI-based virtual patients with integrated automated feedback represent a scalable, effective formative learning tool for anamnesis training across the health professions. The authors assert that while current systems have limitations in emotional expressiveness and unpredictability, they offer a viable alternative to resource-heavy traditional methods, particularly in online and distance learning contexts.

The study modestly concludes that these systems are best suited for formative learning and iterative practice rather than high-stakes assessment. It emphasizes that the educational value lies in the ability to provide immediate, structured, and personalized feedback at scale. The authors note that future research is required to determine if skills acquired in these virtual environments transfer to real clinical practice and to investigate long-term learning outcomes. They also highlight the need for pedagogical integration and human oversight to mitigate risks associated with AI inaccuracies.