Automating QuAL Scoring of Entrustable Professional Activity Narrative Feedback Across Residency Programs: A Locally Deployed Natural Language Processing Model
This study demonstrates that a locally deployed DeBERTa-v3 natural language processing model can accurately and scalably automate the Quality of Assessment for Learning (QuAL) scoring of Entrustable Professional Activity narrative feedback across multiple residency programs, achieving substantial agreement with human raters while preserving data privacy.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of medical training, learning is no longer just about memorizing facts; it is about proving you can do the job. Modern programs rely on a system where doctors-in-training are constantly observed performing real tasks, from managing a patient's care to handling a crisis. These observations are recorded not just with a number, but with written comments. These notes are the lifeblood of the process. They tell the trainee what they did well, what they need to fix, and how their actions connect to the standards of the profession. Without these specific, high-quality notes, a trainee might get a score but remain unsure of how to improve. The problem is that writing these notes is hard work, and the quality of what gets written varies wildly. Some notes are rich with detail and clear advice; others are vague or missing the point entirely. To fix this, educators have developed a way to measure the quality of these comments, looking for three specific things: proof of what happened, a clear suggestion for improvement, and a link to a learning goal. But checking every single note by hand is slow and exhausting, making it difficult to ensure every trainee gets the feedback they deserve.
A team of researchers at the University of Alberta set out to see if a computer could help with this heavy lifting. They wanted to know if a specialized computer program, one that understands human language, could read these thousands of written notes and grade them just as a human expert would. The goal was not to replace the teachers, but to build a tool that could handle the volume of work while keeping the data safe within the hospital and university walls. They focused on a specific type of medical training called Entrustable Professional Activities, where residents are assessed on their ability to perform specific duties. The researchers gathered over 5,000 written comments from 24 different residency programs at a single Canadian institution. These comments came from a year of training between 2019 and 2020. Before the computer ever saw the data, the notes had already been carefully read and scored by human experts using a standard checklist. This human scoring served as the gold standard, the correct answer against which the computer's performance would be measured.
The researchers built a computer model that they ran entirely on their own local servers, ensuring that no sensitive patient or trainee information ever left their secure environment. They taught this model to look for the three key elements in the notes: evidence, suggestions, and connections. For the evidence part, the model had to decide how well the writer described what actually happened, ranging from no description at all to a very detailed account. For the other two parts, it simply had to decide if a suggestion for improvement was present and if the note connected the observation to a broader learning goal. To make sure the test was fair and realistic, they split the data so that notes about the same trainee never appeared in both the training group and the testing group. This prevented the computer from simply memorizing a specific trainee's writing style and pretending it understood the rules.
When the researchers let the model loose on the unseen test data, the results were encouraging. The computer became very good at spotting whether a note contained a clear suggestion for improvement, getting it right most of the time. It also did a strong job of identifying the evidence section, correctly judging the quality of the description in the vast majority of cases. The model was less perfect at finding the connection between the observation and the learning goal, but it still performed well enough to be useful. When the researchers looked at the total score the model gave for each note compared to the human score, they found that in more than 86 percent of cases, the computer's score was within just one point of the human expert's score. The errors it did make were usually small, such as giving a score of two instead of three, rather than large mistakes that would completely misrepresent the quality of the feedback.
The study suggests that this kind of locally run computer tool could become a practical partner for medical educators. It offers a way to monitor the quality of feedback across many programs without the massive cost of hiring people to read every single note. Because the model runs on local computers, it respects the privacy of the trainees and the institution, avoiding the risks of sending sensitive data to outside companies. The researchers are careful to note that this is a starting point, not a finished solution. The model still needs to be tested in other hospitals and in real-time situations to see if it helps teachers and trainees in the long run. But the findings show that it is possible to build a system that understands the nuance of medical feedback, offering a scalable way to ensure that every trainee receives the clear, actionable guidance they need to become a skilled doctor.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.