📄 medicine

Harnessing Artificial Intelligence (AI) for Surgical Education: Comparing AI- and Surgeon-Generated Feedback in Informed Consent Training

While AI-generated feedback for informed consent training was rated as higher quality and more comprehensive than surgeon-generated feedback, the AI systematically assigned more favorable performance scores, indicating that high feedback quality does not guarantee evaluative validity and highlighting the need for human oversight in nuanced surgical assessments.

Original authors: Catherine Gbekie, Brianna M. Peet, Mary Grace Whelan, Grace B. Simmons, Daniel P. Pacella, Viemma Nwigwe, Angel Rosario

Published 2026-10-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Catherine Gbekie, Brianna M. Peet, Mary Grace Whelan, Grace B. Simmons, Daniel P. Pacella, Viemma Nwigwe, Angel Rosario

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes world of surgery, the moment a patient agrees to an operation is as critical as the procedure itself. This agreement, known as informed consent, is not merely a signature on a form but a complex conversation where a doctor must explain the risks, benefits, and alternatives of a surgery in a way the patient truly understands. It requires a delicate balance of medical knowledge, empathy, and the ability to navigate a patient's fears, cultural beliefs, and personal values. For decades, teaching future surgeons how to master this conversation has relied on human teachers observing and critiquing their students. Yet, in busy hospitals, these teachers often struggle to find the time to provide detailed, personalized feedback to every trainee. As artificial intelligence becomes more capable of understanding human language, a new question has emerged: could a computer program learn to teach these vital communication skills as well as, or perhaps even better than, a human surgeon?

Researchers at Columbia University set out to test this idea by pitting a sophisticated computer program against human experts in a simulated training exercise. They asked six medical students to practice a difficult conversation: explaining the need for a laparoscopic appendectomy to a virtual patient who held specific religious beliefs and cultural restrictions. The students spoke with a computer program designed to act as the patient, navigating a scenario that required them to be both medically accurate and culturally sensitive. After each student finished their conversation, the computer generated a detailed report on their performance. Simultaneously, four human surgeons, who were not present during the simulation, read the transcripts of the same conversations and wrote their own evaluations using the exact same scoring guide. The goal was to see if the computer's feedback matched the human experts' views and to determine if the computer could offer high-quality advice for improvement.

The results revealed a striking contrast between the quality of the feedback and the accuracy of the grading. The computer program produced feedback that was significantly more comprehensive, balanced, and constructive than the notes written by the human surgeons. In a standardized rating of the feedback reports, the computer's suggestions scored much higher, offering specific examples of what went well and what needed work, along with clear, actionable steps for improvement. The human surgeons, by comparison, often provided shorter, more direct comments that sometimes focused heavily on errors without always acknowledging the student's strengths. The computer consistently used supportive language and justified its scores with detailed reasoning, whereas the human feedback varied more in its depth and tone.

However, when the researchers looked at the actual scores the computer gave versus the scores the surgeons gave, a different picture emerged. The computer was consistently more generous, awarding higher performance marks than the human experts for the same conversations. In several key areas, such as explaining the risks of the surgery, discussing alternatives, and addressing the patient's emotions, the computer rated the students significantly higher than the surgeons did. The gap was widest in areas requiring deep clinical judgment, such as assessing whether the patient truly understood the situation or integrating the patient's personal preferences into the plan. While the computer often gave near-perfect scores, the human surgeons identified subtle deficiencies in the students' understanding and nuance that the machine missed.

This study suggests that while artificial intelligence is an excellent tool for generating detailed, encouraging, and structured feedback, it is not yet ready to replace human judgment in evaluating a surgeon's true competence. The computer excelled at providing the kind of rich, educational commentary that helps a learner practice and improve, but it tended to overlook the deeper, more subtle flaws in a conversation that an experienced surgeon would catch. The findings indicate that the best approach for training future surgeons is likely a partnership: using artificial intelligence to provide frequent, immediate, and high-quality practice sessions, while relying on human experts to provide the final, nuanced assessment of whether a student has truly mastered the difficult art of informed consent.

Technical Summary: Harnessing AI for Surgical Education in Informed Consent Training

Problem Statement
High-quality, timely, and individualized feedback is essential for developing clinical acumen and communication skills in surgical trainees. However, the delivery of such feedback is often hindered by the competing clinical, research, and educational demands placed on surgical faculty. Consequently, trainee assessments are frequently incomplete or delayed. While informed consent (IC) is a critical competency requiring the integration of procedural knowledge, patient-centered decision-making, and nuanced risk-benefit communication, significant gaps exist in resident and student performance. Current literature indicates that trainees often fail to disclose all essential elements of IC, lack adequate practice, and struggle with standardized education. Although Large Language Models (LLMs) offer a scalable solution for simulated training and feedback, their ability to accurately evaluate learner performance with fidelity comparable to human experts remains insufficiently validated.

Methodology
The study employed a comparative design involving six medical students who completed a simulated informed consent encounter for a laparoscopic appendectomy. The simulation utilized a programmed LLM-based chatbot (ChatGPT, GPT-4) acting as a synthetic patient, incorporating a culturally sensitive scenario involving religious beliefs to test communication beyond basic procedural knowledge.

Evaluation Protocol: Each student's encounter transcript was evaluated twice: once by the LLM and independently by four human surgeons (residents and attendings).
Scoring Instrument: Both AI and human evaluators utilized a standardized 17-item informed consent rubric adapted from the New Mexico Clinical Communication Scale (NMCCS). Items were scored on a 0–3 scale, yielding a maximum total of 51. The rubric covered components such as addressing patient emotions, explaining risks/alternatives, assessing capacity, and avoiding coercion.
Feedback Quality Assessment: Narrative feedback generated by both the AI and the surgeons was evaluated using a modified Completed Clinical Evaluation Report Rating (mCCERR) framework. This 8-item instrument assessed three dimensions: Comprehensiveness and Balance, Specificity and Justification, and Educational Utility and Constructiveness. Scores ranged from 0 to 40.
Statistical Analysis: Scoring alignment was defined as the difference between AI-assigned and the median surgeon-assigned scores. Differences were analyzed using the Wilcoxon rank-sum test. Feedback quality scores were aggregated and compared similarly.

Key Results
The study revealed a distinct divergence between the quality of feedback generated and the validity of the performance assessment.

Performance Scoring Discordance: The AI consistently assigned higher performance scores than human surgeons. Statistically significant differences ( $P=0.03$ ) were found in five specific rubric items: addressing patient emotions, explaining the nature of the procedure, explaining risks, explaining alternatives, and avoiding coercion. The largest absolute score differences occurred in explaining contraindications (1.75 points), assessing capacity (1.5 points), and integrating patient preferences (1.25 points). The AI frequently awarded maximum or near-maximum scores, suggesting limited discrimination across varying levels of learner performance.
Feedback Quality Superiority: In contrast to scoring alignment, AI-generated narrative feedback received significantly higher quality ratings than surgeon-generated feedback. The median overall mCCERR score for AI was 38.5 out of 40, compared to 24.75 for surgeons ( $P=0.04$ ). AI feedback scored significantly higher across all three quality dimensions ( $P \leq 0.04$ ). Qualitative analysis showed AI feedback was more comprehensive, provided explicit justifications for scores, offered actionable recommendations, and utilized supportive, empathetic language. Surgeon feedback was often more concise, variable in scope, and occasionally focused disproportionately on errors without balancing strengths.

Key Contributions

Validation of Feedback Quality vs. Assessment Validity: The study demonstrates that high-quality, comprehensive, and constructive feedback generated by LLMs does not necessarily correlate with accurate performance assessment. The AI provided superior narrative feedback but failed to align with human experts in scoring learner proficiency.
Identification of Systematic Bias: The research identifies a systematic tendency for LLMs to assign more favorable performance scores than human experts, particularly in nuanced areas requiring contextual judgment (e.g., assessing patient capacity or integrating cultural preferences).
Scalability of Formative Feedback: The findings suggest LLMs can effectively generate structured, detailed, and immediate formative feedback, potentially alleviating the burden on faculty while providing learners with consistent guidance.

Significance and Claims
The authors conclude that while Generative AI is a promising tool for providing scalable, individualized formative feedback in surgical education, it should not currently be viewed as a standalone evaluator of learner performance. The study posits that "high feedback quality should not be viewed as a proxy for greater evaluative validity."

The paper advocates for a hybrid, adjunctive model of AI-assisted training. In this model, LLMs are best suited for repeated, low-stakes simulated practice and the delivery of immediate, constructive feedback. However, human faculty oversight remains essential for the nuanced assessment of communication skills and for consequential evaluations of proficiency, particularly where contextual judgment and deep clinical understanding are required. The authors caution that relying solely on AI for assessment could lead to false reassurance and the reinforcement of suboptimal informed consent practices.

Technical Summary: Harnessing AI for Surgical Education in Informed Consent Training

More like this