Design and Feasibility of a Large Language Model-Based Teaching Pathway for Clinical Pharmacy Practice Competencies

In the world of healthcare, learning to talk to patients is just as critical as knowing which medicine to prescribe. For new clinical pharmacists, the path to mastering these conversations is often blocked by a simple reality: there are not enough real patients available for practice, and the experts who could guide them are too busy to watch every interaction. Traditionally, schools have used "standardized patients"—actors trained to pretend to be sick with specific symptoms—to create safe practice environments. While effective, these actors are expensive to hire, hard to schedule, and can only repeat a script so many times before they become tired or inconsistent.

To solve this, researchers have begun turning to artificial intelligence, specifically a type of computer program called a large language model. These models are trained on vast amounts of text and can generate human-like responses, making them potential candidates for creating virtual patients that never get tired. However, early attempts at using this technology in medicine often resulted in rigid, robotic conversations that felt unnatural or failed to provide useful feedback. The challenge has been to teach these computer programs not just to talk, but to act like a real person with a medical history, while also teaching them how to grade a student's performance fairly.

A team of researchers at the Affiliated Cancer Hospital of Nanjing Medical University set out to build and test a new way to train pharmacists using this technology. They wanted to see if they could create a virtual teaching system that felt real enough to be useful, but smart enough to give accurate feedback without needing a human teacher present. They focused on oncology, or cancer care, a field where medication management is complex and mistakes can be dangerous. The team created three specific scenarios involving patients with different types of cancer and chemotherapy regimens. They then invited twenty-one trainees, including junior pharmacists and graduate students, to practice their skills by having a conversation with a virtual patient powered by an open-source artificial intelligence model.

The researchers designed the system with two main goals: first, to make the virtual patient sound and act like a real person, and second, to have the computer automatically score the trainee's performance. To achieve the first goal, they used a structured set of instructions, known as a prompt, that told the computer exactly how to behave. The computer was instructed to stay in character as a patient, to answer questions naturally, and to share medical details only when asked or when it felt like a real person would volunteer them. To achieve the second goal, they gave the computer a separate set of instructions to act as an examiner, reviewing the conversation text to see if the trainee asked the right questions and gathered the necessary information.

When the study began, the results were promising. The virtual patients were remarkably convincing. Experts who reviewed the conversations rated the computer's ability to sound like a real human very highly, noting that the speech felt natural and free of robotic stiffness. The trainees agreed, reporting that they felt comfortable and less tense during the interactions than they might have with a human actor. They appreciated that the virtual patient provided a safe space to practice without the fear of judgment, allowing them to focus entirely on the conversation. The system successfully simulated the flow of a real hospital ward round, moving from taking a medical history to counseling the patient and answering their questions.

The computer also proved capable of grading the trainees. When the researchers compared the scores given by the artificial intelligence to the scores given by a panel of human experts, they found a very strong agreement. The computer and the humans generally ranked the trainees in the same order, identifying who performed well and who needed more practice. However, the study also revealed a specific flaw in the computer's grading logic. The artificial intelligence tended to give lower scores than the human experts, a difference of about two and a half points on a standard scale.

The reason for this gap lies in how the computer interprets the rules. In a real conversation, a patient might volunteer important information, such as mentioning a side effect they are experiencing, even if the pharmacist hasn't asked about it yet. Human experts understand this as a successful interaction because the pharmacist now has the information. The computer, however, was programmed to look strictly for a specific question followed by an answer. If the patient gave the information without being asked, the computer sometimes deducted points, thinking the pharmacist failed to ask the right question. This suggests that while the computer is excellent at following strict rules, it still struggles with the subtle, flexible nature of human conversation where context matters more than a checklist.

Despite this scoring limitation, the study concluded that this approach is a powerful tool for training. The system is not yet ready to replace human teachers for high-stakes exams where a single score determines a career, because of the consistent underestimation. But for daily practice, self-assessment, and building confidence, it offers a solution that is low-cost, always available, and highly effective. The researchers showed that by using clear instructions to guide the artificial intelligence, they could create a teaching pathway that is both realistic and scalable. This means that hospitals and schools with limited budgets could potentially adopt this method to give their trainees the practice they need, bridging the gap between classroom learning and real-world patient care. The work suggests that the future of medical training may not rely on hiring more actors, but on teaching computers to listen and respond with the nuance of a real human.