Can LLM-Simulated Practice and Feedback Upskill Human Counselors? A Randomized Study with 90+ Novice Counselors

A randomized study with 94 novice counselors demonstrates that combining LLM-simulated practice with structured feedback significantly improves client-centered microskills and empathy, whereas practice alone fails to produce gains and may even lead to a decline in empathetic performance.

The Gist

Here is an explanation of the paper using simple language, creative analogies, and metaphors.

The Big Picture: Can a Robot Teacher Train a Human Counselor?

Imagine you want to learn how to be a great listener and counselor for people in trouble. Traditionally, you'd need a human teacher, a role-playing partner, and hours of supervision. But there aren't enough human teachers to go around.

So, researchers asked: Can a super-smart AI (a Large Language Model) act as both the "patient" you practice on AND the "teacher" who grades you?

To find out, they built a system called CARE and ran a massive experiment with 94 beginners (novice counselors). They split the group into two teams to see what actually works:

1. Team "Just Practice" (Group P): They talked to the AI patient but got no feedback. It was like playing a video game where you just keep playing without a score or a "Game Over" screen.
2. Team "Practice + Coach" (Group P+F): They talked to the AI patient, and after every few sentences, the AI gave them a coach's note. It said, "Good job asking an open question!" or "Try saying it this way to show more empathy."

The Results: The "Feedback" Factor is Everything

The results were surprising and very clear.

1. The "Just Practice" Group Got Worse at Empathy
Think of this group like someone trying to learn to swim by just jumping into the deep end and flailing around. They didn't get better; they actually got worse at the most important skill: Empathy.

• What happened? Without a coach telling them to slow down and listen, they kept trying to "fix" the patient's problems immediately. They became like a mechanic trying to repair a car that just wants to talk about its feelings. They stopped listening and started offering solutions too fast.
• The Metaphor: It's like trying to learn to dance by just moving your feet. Without a teacher correcting your rhythm, you might end up stepping on your partner's toes more often than before.

2. The "Practice + Coach" Group Got Better
This group was like a student with a personal trainer. The AI coach told them exactly what to improve.

• What happened? They got significantly better at Reflections (repeating back what the patient said to show understanding) and Questions (asking open-ended questions). They learned to pause, listen, and validate feelings instead of rushing to fix things.
• The Metaphor: This is like having a GPS while driving. You still drive the car, but the GPS tells you, "You're going the wrong way; turn left here to get to the destination."

The "Confidence Trap" (The Dunning-Kruger Effect)

One of the most interesting findings was about confidence.

• The Problem: The beginners who were actually the worst at counseling thought they were the best. They were overconfident.
• The Reality: The AI feedback didn't magically fix their confidence immediately, but it did show them the gap between what they thought they were doing and what they were actually doing.
• The Metaphor: Imagine a person who thinks they are a gourmet chef because they can boil water. They taste their food and say, "Perfect!" But a real chef (the AI) tastes it and says, "It's raw." The beginners in the study were like the overconfident cook—they needed an outside voice to tell them they weren't ready yet.

What Did the Beginners Say? (The Human Element)

The researchers also talked to the participants to see how they felt.

• The Good: Most people loved the feedback. They said things like, "Oh, I didn't realize I was being too pushy," or "That alternative phrase the AI suggested was much nicer."
• The Bad: Some people felt overwhelmed. Imagine getting a red "X" on every single sentence you write in a diary. Some participants felt demoralized, thinking, "I can't do anything right." They worried the AI was making them sound robotic and losing their "authentic voice."
• The Resistance: The AI patients were programmed to be stubborn (just like real people in crisis). Some beginners got frustrated when the AI wouldn't listen to their advice. This was actually a good thing—it prepared them for real life, where people often say "No" to help.

The Takeaway: Practice Alone Isn't Enough

The main lesson from this paper is simple: You cannot learn to be a counselor just by "doing it." You need feedback.

• Without Feedback: You might practice the wrong habits over and over, getting worse at the most important skill (empathy) because you think you're helping by fixing problems.
• With Feedback: You learn to slow down, listen, and understand the person's feelings before trying to solve their problems.

The Final Analogy:
Learning to be a counselor without feedback is like trying to learn a new language by only listening to yourself speak. You might think you sound fluent, but you're actually making mistakes you can't hear. The AI feedback is the native speaker who gently corrects your pronunciation, helping you actually become fluent.

In short: AI can be a fantastic training tool, but only if it acts as a coach, not just a partner.

Technical Summary

Here is a detailed technical summary of the paper "Can LLM-Simulated Practice and Feedback Upskill Human Counselors? A Randomized Study with 90+ Novice Counselors."

1. Problem Statement

The demand for mental health support significantly outpaces the supply of qualified counselors. Traditional training methods (role-plays, supervised practice) are resource-intensive, difficult to scale, and rely heavily on human trainers. While Large Language Models (LLMs) offer a potential solution by simulating patients and generating feedback, their actual efficacy in developing novice counselor skills remains unproven. Specifically, it is unknown whether:

• LLM-simulated practice alone improves skills.
• Structured AI feedback is necessary to prevent skill degradation (e.g., loss of empathy).
• LLMs can effectively teach client-centered microskills (empathy, reflections, active listening) versus solution-oriented approaches.

2. Methodology

System Design: CARE

The authors developed CARE, a web-based training platform integrating two core components:

1. LLM-Simulated Patients: Built using GPT-4o, these agents were co-designed with expert counselors to exhibit realistic, challenging behaviors (resistance, ambivalence, vague disclosures) based on specific clinical scenarios (e.g., a veteran seeking custody, a young adult with family issues).
2. LLM-Generated Feedback: A fine-tuned Llama-2 13B model (trained on expert-annotated counseling transcripts) provides post-response feedback. It classifies counselor utterances into eight microskills, identifies strengths/weaknesses, and offers alternative responses with explanatory rationales grounded in client-centered frameworks (e.g., Hill's Helping Skills model).

Study Design

• Participants: $N=94$ novice counselors (undergraduate/master's level, <1 year experience) recruited via Prolific.
• Procedure: A 75-minute randomized controlled trial (RCT) conducted over Zoom.
  • Pre-Intervention: 10-minute chat with an AI patient.
  • Intervention: 20-minute chat with a second AI patient.
    • Group P (Practice Only): Interacted with the AI patient without feedback.
    • Group P+F (Practice + Feedback): Interacted with the AI patient and received structured AI feedback after each response.
  • Post-Intervention: 10-minute chat with a third AI patient.
• Measures:
  • Behavioral Performance: Automatic assessment using fine-tuned RoBERTa classifiers to detect the proportion of utterances demonstrating strong use of Empathy, Reflections, Questions, and inappropriate use of Suggestions.
  • Self-Efficacy: Measured via the Counselor Activity Self-Efficacy Scale (CASES-R).
  • Therapeutic Intentions: Qualitative analysis of self-reflections regarding what participants intended to do differently.
  • Perceptions: Quantitative surveys (Likert scales) and qualitative interviews regarding the realism of patients and helpfulness of feedback.

3. Key Contributions

1. System Development: Creation of CARE, a holistic LLM-based training system that combines expert-validated patient simulations with fine-tuned, explanatory feedback models.
2. Empirical Evidence: The first large-scale randomized evaluation ($N=94$) of LLM-based training for novice counselors, triangulating behavioral data, self-efficacy, and qualitative intentions.
3. Design Implications: Demonstration that structured feedback is critical; practice alone is insufficient and can be detrimental to specific skills like empathy.

4. Key Results

Behavioral Performance (RQ1)

• Practice + Feedback (P+F): Showed significant improvements in Reflections (+3.7%, $p=0.034$) and Questions (+6.6%, $p=0.018$). There was a trend toward improvement in Empathy.
• Practice Only (P): Showed no improvement in Reflections or Questions. Crucially, this group experienced a significant decline in Empathy (-9.6%, $p<0.001$).
• Suggestions: Both groups reduced inappropriate suggestions, but the reduction was driven by the patient's resistance (adaptive simulation) rather than feedback, as the between-group difference was negligible ($d=0.02$).
• Conclusion: Feedback is necessary to cultivate client-centered listening; without it, novices tend to de-prioritize empathy.

Self-Efficacy and Calibration (RQ2)

• Raw Scores: Both groups showed modest increases in self-efficacy, with no significant difference between groups.
• Calibration (Dunning-Kruger Effect): Participants generally misjudged their abilities. The lowest-performing quartile significantly overestimated their skills.
• Impact of Intervention: Neither intervention significantly improved the calibration between self-perception and actual performance. This suggests self-assessment is an unreliable metric for skill development in this context.

Therapeutic Intentions (RQ3)

• P+F Group: Shifted toward client-centered approaches, emphasizing empathy, validation, and exploring client emotions.
• P Group: Remained solution-oriented, focusing on gathering information to provide advice, even after the intervention.

User Experience (RQ4-RQ6)

• Feedback: Participants found the feedback helpful (76%) and comfortable to receive (84%). However, some felt overwhelmed by the frequency of corrections or questioned the AI's ability to handle high-risk scenarios.
• Simulated Patients: Rated as highly authentic (avg. 6.1/7). However, the rapid response speed of AI created artificial time pressure, sometimes disrupting clinical attentiveness.
• Resistance: Participants had mixed reactions to AI resistance; some found it valuable preparation, while others felt discouraged or questioned the realism of the "loop" of refusal.

5. Significance and Implications

• Scalability: LLM-based training offers a scalable alternative to human supervision for foundational counseling skills.
• Feedback Necessity: The study challenges the assumption that "practice makes perfect." In counseling, practice without feedback can degrade empathy. Structured feedback is required to guide novices away from solution-oriented defaults toward client-centered listening.
• Feedback Design: The study highlights the need for balanced feedback. Too much criticism can demoralize learners, while too little fails to correct misconceptions. Future systems should consider clustering errors and modulating feedback based on learner affect.
• Limitations: The study focused on text-based interactions and foundational microskills. Generalizability to complex modalities (CBT, psychodynamic) and non-verbal communication (voice, facial expressions) requires further research. Additionally, the study did not include a "no-AI" control group, though the primary comparison was between practice modes.

Conclusion: The paper concludes that LLM-simulated training is a viable path for upskilling counselors, but only when paired with structured, expert-aligned feedback. Without feedback, simulated practice may reinforce unhelpful habits or erode essential empathetic skills.