Accuracy and Limitations of AI Scoring for Objective Medical Interview Checklist Items in a Year-One School of Medicine Objective Structured Clinical Examination (OSCE)

In medical schools, learning to talk to patients is just as critical as learning to examine them. To ensure students are ready for the real world, educators use a testing method called the Objective Structured Clinical Examination. In this setting, a student sits down with a trained actor playing a patient. The student must ask the right questions to gather a medical history, and the actor scores the student based on a strict checklist of what was asked and how well it was done. This process is the gold standard for teaching clinical skills, but it is also incredibly labor-intensive. It requires hours of training for the actors, who must remain consistent while also judging the student's performance, and it demands that faculty members spend countless hours reviewing these interactions to ensure fairness. As artificial intelligence becomes more capable of listening to conversations and understanding language, educators have begun to wonder if a computer could take over the scoring. The hope is that a machine could listen to the recording, check off the required questions, and grade the student in a fraction of the time it takes a human, freeing up resources for more teaching.

A team of researchers at the University of Texas Medical Branch at Galveston set out to test this idea with a specific goal: to see if artificial intelligence could accurately score the objective parts of a medical interview. They focused on a first-year medical school exam involving 234 students. In this exam, students were recorded while interviewing a standardized patient, and human actors had already graded them on twelve specific checklist items, such as asking about the location of pain, the severity of symptoms, or the patient's family history. The researchers took the video recordings of these encounters and turned them into text transcripts. They then fed these transcripts into a software platform equipped with artificial intelligence, programming it with prompts designed to look for those same twelve checklist items. The computer was told to act like a grader, assigning a point for each item it detected in the text. The team then compared the scores the computer generated against the scores the human actors had given, looking for differences and trying to understand why they occurred.

The results showed that the artificial intelligence was a capable, though not perfect, assistant. When the researchers compared the computer's grades to the human grades, they found a moderate level of agreement. The average total score given by the computer was nearly identical to the average given by the humans, with the computer awarding an average of 11.51 points out of a possible 12, while the humans awarded 11.60. In terms of time, the difference was stark. The human scoring process, which included training the actors and reviewing the exams, took roughly 56 hours to complete for this group of students. The computer, once the system was set up, analyzed all 234 interviews and generated scores in just under three hours. This suggests that for the specific task of checking off factual questions in a medical interview, the machine can do the heavy lifting with remarkable speed and a high degree of consistency.

However, the study also revealed exactly where the machine stumbled. Out of nearly 2,800 individual scoring decisions, there were 83 instances where the computer and the human actor disagreed. When the researchers looked closely at these errors, they found four main reasons for the mismatch. First, the computer is entirely dependent on the quality of the text it reads. If the audio recording was quiet, if two people spoke at the same time, or if background noise interrupted the conversation, the transcription software might miss a word or mishear it. For example, if a student asked if pain "radiated" to another part of the body but the computer transcribed it as "resonated," the AI would not recognize the correct medical term and would mark the question as missing.

Second, the computer struggles with the way humans actually speak. Students often phrase their questions in unique or indirect ways that a human actor can understand immediately, but a computer might miss if it is looking for a specific set of words. If a student asked, "Does the pain stay in one spot?" instead of the expected "Does the pain radiate?", a human might give credit for understanding the concept, but the computer, following rigid rules, might not. Third, the computer cannot see or hear nonverbal cues. In a real interview, a patient might nod or point to their stomach to answer a question, and a human actor would give the student credit for that interaction. Since the computer only reads the text transcript, it cannot see the nod or the pointing, so it marks the question as unanswered. Finally, there were differences in judgment. Sometimes the human actor would give credit for an implied meaning that the student intended, while the computer required the student to say the exact words written on the checklist.

The researchers concluded that while artificial intelligence is a powerful tool for speeding up the grading of structured, objective checklists, it is not ready to replace human judgment entirely. The computer is excellent at finding specific words in a transcript, but it lacks the ability to understand context, tone, or the subtle intent behind a student's question. The study suggests that the best use of this technology is as a support tool. It can handle the initial pass of grading, saving educators dozens of hours, but a human must still review the results to catch the errors that arise from poor audio, unique phrasing, or nonverbal communication. Until the technology can reliably interpret the full nuance of a human conversation, including what is left unsaid, the final say in assessing a student's clinical skills must remain with a person. The future of medical education assessment may involve a partnership where the machine handles the volume and the human handles the judgment, ensuring that students are graded fairly and efficiently.