Accuracy of AI-Generated Procedural Descriptions in Peripheral Nerve Surgery: Implications for Surgical Training
This study demonstrates that while AI models (GPT-5.2 and GPT-5.5) can generate generally accurate procedural descriptions for peripheral nerve surgery, their reliability is significantly compromised by prompting strategies and they consistently struggle with judgment-dependent steps, indicating they should serve as supplementary educational tools rather than replacements for expert guidance.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Surgery is a discipline built on the precise execution of steps, where the difference between a successful outcome and a complication often lies in the details of how a surgeon moves their hands and makes decisions in the moment. For decades, the training to master these movements has relied on a slow, deliberate process of watching experts, practicing under supervision, and absorbing the unwritten rules that separate a competent procedure from an expert one. Recently, a new tool has entered this landscape: artificial intelligence capable of generating text that describes complex medical procedures. These systems, often called large language models, can read vast amounts of information and write clear, organized instructions. As medical schools and hospitals look for ways to help students learn more efficiently, a critical question has emerged: can a computer program accurately describe the intricate steps of surgery, or does it miss the subtle, human elements that make the difference?
A team of researchers at the University of British Columbia set out to answer this question by testing the capabilities of a sophisticated artificial intelligence model on the delicate field of peripheral nerve surgery. This branch of medicine involves repairing and reconnecting the tiny nerves that control movement and sensation in the arms and hands, a task that requires extreme precision. The researchers chose fifteen different nerve procedures, ranging from releasing a compressed nerve in the wrist to transferring a nerve from one muscle to another. They used a high-quality educational website featuring videos and transcripts from expert surgeons as the "gold standard" for what a correct description should look like. They then asked the artificial intelligence to generate step-by-step descriptions for each of these fifteen procedures, running the same request six times for each surgery to see if the answers were consistent.
The results showed that the artificial intelligence was surprisingly good at the basics but struggled with the nuances. On average, the AI's descriptions were about 81 percent accurate when compared to the expert standard. This means that for most of the time, the computer correctly identified the major steps of the surgery, such as where to make the incision or how to move the nerve. However, the accuracy was not perfect, and it varied significantly from one attempt to the next. In some runs, the AI got nearly 92 percent of the steps right, while in others, it dropped to just under 69 percent. This inconsistency means that a student asking the same question twice could receive two different versions of the same surgery, with no way of knowing which one was more complete.
When the researchers looked closely at what the AI missed, they found a clear pattern. The most common errors were not in the big, obvious steps, but in the smaller, judgment-based decisions that experienced surgeons make. These are the moments where a surgeon might choose to add a specific maneuver to improve exposure, or decide to use a particular technique based on how the tissue looks in front of them. The AI tended to describe the core framework of the surgery but often skipped these optional, expert-level details. More concerning were the instances where the AI failed to mention critical safety steps, such as identifying a specific blood vessel or nerve that must be protected to prevent injury. While these safety steps were missed less often than the judgment calls, the researchers noted that missing even one of them could have serious consequences for a patient.
The study also tested how the way a question is asked affects the answer. When the researchers used a simple, casual prompt—something like "tell me how to do this surgery"—the accuracy of the AI's response plummeted to about 40 percent. In these cases, the AI missed entire major steps of the procedure, not just the subtle details. However, when the researchers used a carefully crafted, structured prompt that asked for specific details like landmarks and instruments, the accuracy jumped back up to over 80 percent. This suggests that the quality of the AI's output depends heavily on how the user interacts with it. A well-designed question can guide the computer to provide a much more useful and accurate answer, while a vague question leads to a poor one.
The researchers also checked if a newer version of the AI model performed better. They found that the newer model was just as accurate as the older one, but it was even more variable in its answers. This indicates that simply waiting for a newer version of the software will not solve the problem of missing details or inconsistent answers. The core limitation appears to be in how these systems generate text: they are excellent at recalling facts and organizing information, but they struggle to replicate the conditional reasoning and personal experience that guide a surgeon's choices.
Ultimately, the study concludes that these artificial intelligence tools can be helpful for giving students a general overview of a surgery, acting as a starting point for orientation. However, they should not be used as a replacement for expert guidance, especially for the safety-critical steps or the nuanced decisions that define expert practice. The technology is a powerful assistant that can summarize information quickly, but it cannot yet replace the years of mentorship and observation required to learn the art of surgery. For now, the best approach is to use these tools with caution, always verifying the information against trusted human experts and understanding that the computer may miss the very details that matter most.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.