Large language models for self-administered conversational vignette assessment of provider competencies: A pilot and validation study in Vietnam with automated LLM-powered transcript classification

This study validates a low-cost, scalable, and open-source platform in Vietnam that uses large language models to simulate patient conversations in local languages for self-administered assessment of primary care provider competencies, demonstrating that automated transcript scoring achieves reasonable agreement with human experts without requiring translation or in-person enumerators.

The Gist

Imagine you want to know if a chef is actually good at cooking. You could send a food critic to their restaurant to taste the dishes (this is the traditional, expensive way). Or, you could ask the chef to cook a specific meal for a robot that tastes everything and writes a report (this is the new, cheap way).

This paper is about building that robot chef-taster for doctors, but instead of food, they are testing how well doctors diagnose and treat patients.

Here is the story of how they did it, broken down simply:

1. The Problem: The "Food Critic" is Too Expensive

Traditionally, to check if doctors in places like Vietnam are doing a good job, researchers have to hire teams of "food critics" (called enumerators). These critics travel to hospitals, sit in waiting rooms, and pretend to be sick patients to see how the real doctors treat them.

• The Catch: This is incredibly expensive. You need to pay for travel, hotels, and salaries for the critics. Because it costs so much, we rarely get to check if doctors are improving their skills. It's like only tasting a restaurant once every ten years because hiring a critic is too pricey.

2. The Solution: The "AI Patient"

The researchers built a digital patient using Artificial Intelligence (specifically, Large Language Models like the ones behind ChatGPT).

• How it works: A doctor logs into a website on their phone or laptop. They start a text conversation with an AI that is pretending to be a patient with a specific problem (like asthma or hepatitis).
• The Twist: The AI patient is very smart. It answers questions, describes symptoms, and reacts to the doctor's orders just like a real human would. The doctor has to ask the right questions, order the right tests, and give the right diagnosis.

3. The Test: Did the Robot Work?

The team tested this in Vietnam with 31 real doctors.

• The Pilot (The Taste Test): First, they asked a few doctors to try it out. The doctors said, "Wow, this feels real!" They liked that they could do it from their own phones and that the AI spoke perfect Vietnamese.
• The Validation (The Scorecard): Then, 22 doctors played through 132 different scenarios.
  • Human Judges: Two expert doctors read the chat transcripts and gave the doctors a score (1 to 5 stars).
  • The AI Judge: A different AI (Claude) read the same chats and tried to grade them automatically.

4. The Results: The Robot is a Good Judge

The results were surprisingly good:

• Realism: The doctors felt the AI patients were realistic enough to be useful.
• Accuracy: The AI's grading matched the human experts' grading quite well. It was like having a robot judge that agreed with the human judges about 85% of the time.
• The Language Trick: Usually, to use AI in a foreign country, you have to translate everything into English first. But this team found they didn't need to! The AI could read and grade the Vietnamese chats directly, just as well as if they had been translated. This saves a huge amount of time and money.
• Cost: The total cost to run 132 of these "simulated patient visits" was less than $2.00. Compare that to the thousands of dollars it would cost to send human critics to do the same job.

5. Why This Matters

Think of this as a video game for doctors.

• Old Way: You hire a coach to watch you play in real life. It's expensive and you can only do it once in a while.
• New Way: You play a high-tech simulation game. The game tracks your moves, grades your performance instantly, and costs almost nothing.

This tool allows health systems to check the skills of thousands of doctors frequently and cheaply. It doesn't replace real doctors, but it gives us a way to see if they are learning and improving without breaking the bank.

The Catch (Limitations)

Just like a video game isn't exactly the same as real life, this system has limits:

• No Body Language: The AI is text-only. In real life, a doctor might look at a patient's pale face or listen to their breathing. The AI can't "see" or "hear" those things yet.
• Small Sample: They only tested it on a small group of doctors in one country. It needs to be tested more to be sure it works everywhere.

The Bottom Line

This study shows that we can use smart computer programs to create "fake patients" that help us test and improve real doctors' skills. It's a low-cost, scalable way to make sure healthcare is better for everyone, especially in places where money is tight.

Technical Summary

1. Problem Statement

Traditional methods for assessing clinical competencies of healthcare providers, specifically clinical vignettes (open-ended roleplay with standardized patients), face significant scalability and cost barriers.

• High Cost & Logistics: Traditional surveys require two trained enumerators per interaction, extensive travel budgets, and in-person coordination, making them prohibitively expensive for large-scale or frequent monitoring.
• Validity Trade-offs: Digital alternatives often rely on multiple-choice questions, which undermine the validity of vignettes by failing to capture the unprompted, open-ended nature of clinical reasoning.
• Gap in LMICs: While Large Language Models (LLMs) show promise in medical education, few have been validated for health workforce assessment in Low- and Middle-Income Countries (LMICs), and automated scoring of clinical interactions remains "noisy" with moderate agreement to human evaluators.

2. Methodology

The study developed and validated a self-administered, LLM-powered conversational vignette platform deployed via web-based SurveyCTO software.

• Study Design & Participants:
  • Pilot (Focus Group): 9 physicians in Phu Tho, Vietnam, tested usability and realism.
  • Validation: 22 providers from the StITCH hepatitis care training program completed 132 interactions across 10 clinical scenarios (5 general primary care, 5 hepatitis-related).
  • Scenarios: Cases included conditions like asthma, pneumonia, diabetes, tuberculosis, hypertension, and various hepatitis presentations.
• System Architecture:
  • Chatbot (Patient Simulation): Powered by OpenAI's gpt-4.1-nano. The system used structured system prompts (1,500–3,000 words) containing patient demographics, history, and pre-scripted responses. It maintained conversational state and responded in Vietnamese via the provider's personal smartphone or laptop.
  • Automated Scoring (Data Extraction): Powered by Anthropic's claude-haiku-4-5. The model was prompted to analyze transcripts against a specific rubric to determine if providers asked checklist questions, ordered tests, and provided correct diagnoses.
• Data Processing & Analysis:
  • Human Coding: Transcripts were translated to English (Google Translate API) and manually coded by researchers to establish a "ground truth" checklist score.
  • LLM Coding: The Claude model scored transcripts in two ways: (1) from English translations and (2) directly from original Vietnamese transcripts.
  • Validation Metrics: The study compared LLM scores against human-coded checklists and holistic expert clinician ratings (1–5 scale) using Pearson correlation, Ordinary Least Squares (OLS) regression, and Receiver Operating Characteristic (ROC) analysis.

3. Key Contributions

• Novel Assessment Framework: Introduced a fully automated, self-administered clinical vignette system that eliminates the need for in-person enumerators while preserving the open-ended format critical for validity.
• Multilingual Direct Coding: Demonstrated that LLMs can score clinical interactions directly in the local language (Vietnamese) without a separate translation step, achieving performance comparable to English-based coding.
• Cost-Effectiveness: Achieved 132 clinical interactions for a total API cost of under USD 2, representing a fraction of the cost of traditional surveys.
• Open-Source Platform: The tool is built on widely used survey software (SurveyCTO) and is designed to be open-source and replicable in other health system contexts.

4. Key Results

• Usability & Realism: In the pilot, providers rated the simulated interactions as realistic (mean 3.78/5) and user-friendly. Qualitative feedback highlighted the system's ability to provide contextually appropriate, localized responses.
• Correlation with Expert Judgment:
  • Human-coded checklist scores correlated well with expert clinician ratings (Pearson's ρ = 0.55–0.60).
  • A 10 percentage point increase in checklist completion corresponded to a ~0.64-point increase in expert ratings.
• Automated Coding Performance:
  • Human vs. LLM (English): Moderate correlation (ρ = 0.53).
  • Human vs. LLM (Vietnamese Direct): Comparable correlation (ρ = 0.51), confirming translation is not a binding constraint.
  • Discriminative Ability: The LLM's ability to distinguish between correct and incorrect checklist items (measured by AUROC) was 0.87 for both English and Vietnamese transcripts, indicating "good" discriminative ability.
• Provider Performance: The average provider completed only 16.25% of the essential diagnostic checklist items, reflecting the comprehensive nature of the checklists and the tendency of providers to focus on clinically relevant rather than exhaustive questioning.

5. Significance and Implications

• Scalable Monitoring: This method enables routine, large-scale monitoring of healthcare quality in resource-limited settings where traditional surveys are too expensive to replicate (e.g., Vietnam's last comprehensive vignette study was in 2015).
• Elimination of Translation Bottlenecks: The finding that direct Vietnamese coding performs as well as English coding suggests a streamlined pipeline for multilingual global deployment, reducing costs and potential translation errors.
• Future of Health Workforce Assessment: As LLM capabilities improve (e.g., reasoning models, multimodal inputs), this platform could evolve to include audio/video cues, further bridging the gap between simulated and real-world clinical assessment.
• Limitations: The study acknowledges a small sample size (22 providers), the lack of non-verbal cues in text-based interactions, and the "know-do gap" (vignette performance may not perfectly mirror real-world practice). However, the low cost and high scalability offer a transformative approach to closing the data gap in global health.