💬 NLP

A Multi-Agent LLM Framework for Rating the Quality of Surgical Feedback

This paper introduces a two-stage, multi-agent LLM framework that discovers interpretable surgical feedback quality criteria to automatically score live operating room interactions, demonstrating superior performance over prior methods in predicting feedback effectiveness and trainee behavioral adjustments.

Original authors: Rafal Kocielnik, J. Everett Knudsen, Steven Y. Cen, Jasmine Lin, Cherine H. Yang, Atharva Deo, Ujjwal Pasupulety, Peter Wager, Anima Anandkumar, Andrew J. Hung

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rafal Kocielnik, J. Everett Knudsen, Steven Y. Cen, Jasmine Lin, Cherine H. Yang, Atharva Deo, Ujjwal Pasupulety, Peter Wager, Anima Anandkumar, Andrew J. Hung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a surgical operating room as a high-stakes orchestra. The surgeon in training is the lead violinist, and the attending surgeon (the teacher) is the conductor. For the music to be perfect, the conductor needs to give feedback. But here's the problem: sometimes the conductor's feedback is a gentle whisper, sometimes a sharp shout, and sometimes it's just a confusing murmur.

For years, researchers have tried to grade these musical directions by listening to what the conductor says (e.g., "fix the note" vs. "play louder"). But this new paper argues that how the feedback is delivered matters just as much as the words themselves. Did the teacher sound urgent? Was the instruction clear? Did they sound encouraging?

The authors, a team from Caltech and Cedars-Sinai, built a "smart robot listener" (a Large Language Model framework) to solve this. Here is how they did it, broken down into simple steps:

1. The "Brainstorming Party" (Discovery)

Instead of guessing what makes good feedback, the researchers asked a team of AI agents (think of them as five different expert music critics) to listen to thousands of real surgical conversations.

The Setup: They gave these AI agents a list of what "good" looks like (like "the student fixed their mistake") and asked them to figure out why some feedback worked and some didn't.
The Result: The AI agents came up with a long, messy list of ideas. The researchers then had a "consolidation meeting" where they grouped similar ideas together and distilled them down to six golden rules for good feedback:
1. Encouraging: Does it boost confidence? (e.g., "Great job!")
2. Urgent: Does it scream "Stop right now!"? (e.g., "Don't coagulate!")
3. Actionable: Is it a specific step? (e.g., "Move the needle 2cm left.")
4. Timely: Was it said while the action was happening, or hours later?
5. Clear: Is it easy to understand, or confusing?
6. Reflective: Does it make the student think? (e.g., "Why did you choose that?")

2. The "Robot Judge" (Scoring)

Once they had these six rules, they turned the AI into a judge. They fed it 4,200 real surgical feedback snippets and asked it to score each one from 1 to 5 on all six rules.

The Analogy: Imagine a teacher grading a student's essay. Instead of just giving a letter grade, this robot gives a detailed report card: "Your urgency was a 5/5, but your clarity was only a 2/5."

3. The Proof (Does it work?)

The team tested if these robot scores actually predicted what happened next in the operating room.

The Test: They looked at whether the student changed their behavior (like moving a tool correctly) or simply said "Okay" to acknowledge the teacher.
The Result: The AI's six rules were better at predicting student reactions than previous methods that just looked at the topic of the conversation.
- Example: They found that "Urgent" and "Actionable" feedback made students move faster. But "Reflective" and "Clear" feedback made students talk back more.
- Surprise: "Encouraging" feedback actually made students less likely to change their behavior or speak up. The paper suggests this is because encouragement is often used when the student is already doing a good job, so no change is needed!

4. The Human Check (Reliability)

To make sure the robot wasn't just hallucinating, they asked real human surgeons to grade the same feedback using the AI's six rules.

The Match: The humans and the AI agreed with each other quite well (about 60-70% agreement). This proves the rules are clear enough that both humans and machines can understand them.

Why This Matters (According to the Paper)

This framework is like giving every surgical teacher a "quality control dashboard."

It doesn't just tell you what was taught; it tells you how well it was taught.
It can automatically score thousands of hours of surgery without needing a human to sit there and take notes for days.
It helps identify if a teacher is being too vague, too late, or too confusing, which could help improve how surgeons are trained.

In short: The paper built an AI system that learned to grade surgical teachers not on their vocabulary, but on their delivery style. It found that being clear, urgent, and actionable is the secret sauce for getting students to actually fix their mistakes in real-time.

Technical Summary: A Multi-Agent LLM Framework for Rating the Quality of Surgical Feedback

Problem Statement

Verbal feedback from attending surgeons to resident trainees in the operating room (OR) is critical for skill acquisition and surgical outcomes. However, systematically assessing the quality of this feedback—specifically how it is delivered rather than just its content—remains a significant challenge. Prior research has largely focused on categorizing the content of feedback (e.g., anatomical references, instrument handling) using manual annotation or automated methods like keyword analysis and topic modeling. These existing approaches fail to capture nuanced delivery attributes such as clarity, urgency, timeliness, and emotional tone, which are essential for understanding feedback effectiveness. Furthermore, manual annotation is labor-intensive and not scalable, while existing automated methods lack the sensitivity to clinical language and the ability to distinguish between identical instructional tasks delivered with different tones or urgencies.

Methodology

The authors propose a two-stage, Large Language Model (LLM)-based framework designed to discover interpretable feedback quality criteria and automatically score surgical feedback. The pipeline integrates multi-agent prompting, domain knowledge injection, and an "LLM-as-a-judge" approach.

1. Criteria Discovery Phase

The discovery process utilizes a multi-agent strategy to generate and consolidate evaluation criteria:

Domain-Guided Initiation: GPT-4o agents are seeded with formal definitions of surgical feedback outcomes (Behavioral Adjustment, Verbal Acknowledgment, Trainer Approval) and a definition of feedback itself. They are provided with a small subset of unlabeled feedback examples (<1% of the dataset) to infer relevant evaluative dimensions through analogical reasoning.
Multi-Agent Generation: Five independent GPT-4o agents, configured with a high sampling temperature ( $T=1.0$ ) to encourage diversity, independently propose candidate quality dimensions. Each agent is tasked with defining abstract, generalizable criteria that can be scored on a 5-point Behaviorally Anchored Rating Scale (BARS) based solely on a single transcribed feedback line.
Criteria Consolidation: The candidate criteria from all agents are applied to the full dataset. A hierarchical clustering algorithm (single linkage, Euclidean distance) groups semantically similar criteria. The optimal number of clusters ( $k=6$ ) is determined via the silhouette coefficient. A final GPT-4o instance (in deterministic mode, $T=0.0$ ) synthesizes a unified definition and BARS anchors for each cluster, ensuring reproducibility and non-overlap.

2. Criteria Consolidation and Validation

The process yielded six interpretable feedback quality dimensions:

Encouraging: Supportive tone boosting confidence.
Urgent: Communication indicating immediate action is required.
Actionable: Clear, specific steps the trainee can implement.
Timely: Feedback provided during or immediately after the action.
Clear: Straightforward, unambiguous messaging.
Reflective: Prompts for self-assessment or deeper insight.

3. Automated Scoring and Evaluation

LLM-as-a-Judge: The final six-dimension rubric is used to score 4,210 live surgical feedback instances using GPT-4o. The model assigns scores from 1 to 5 for each dimension.
Predictive Modeling: The authors evaluated the ability of these AI-derived scores to predict four behavioral outcomes: Trainee Behavior Change, Trainee Verbal Response, Trainer Approval, and Trainer Disapproval. They compared their model against prior topic modeling approaches and manually annotated human categories using Random Forest classifiers with fivefold stratified cross-validation.
Human-AI Alignment: Two human raters with domain knowledge scored a subset of 30 instances using the AI-discovered rubric. Inter-rater reliability was assessed using Quadratic Weighted Kappa ( $\kappa$ ) to measure agreement between humans, between AI runs, and between humans and AI.

Key Results

Predictive Performance

The AI-discovered quality criteria demonstrated superior predictive power compared to prior methods:

Standalone Performance: Using only the six quality scores, the model achieved AUROC scores of 0.75 for Trainee Behavior Change, 0.71 for Verbal Response, 0.66 for Trainer Approval, and 0.60 for Trainer Disapproval.
Comparative Improvement: When combined with prior topic modeling features, the AI scores improved prediction of Trainee Behavior Change by 12% (AUROC 0.77) and Trainer Disapproval by 11% (AUROC 0.63). When combined with prior human-defined categories, improvements ranged from 3% to 12% across all outcomes.
Statistical Significance: DeLong's test confirmed that adding AI-derived quality scores resulted in statistically significant improvements ( $p < 0.05$ ) over both topic modeling and human-defined categories for most outcomes.

Human-AI Alignment

Consistency: AI runs showed near-perfect internal consistency ( $\kappa = 0.92–1.00$ ).
Alignment: Human-AI agreement was substantial ( $\kappa = 0.60–0.79$ ) for five of the six criteria (Encouragement, Urgency, Actionability, Clarity, Reflection), approaching inter-human agreement levels. Agreement was lower for "Timeliness" ( $\kappa = 0.54$ ), reflecting the inherent difficulty in judging temporal aspects.

Association with Outcomes

Analysis of the relationship between specific quality dimensions and outcomes revealed distinct patterns:

Behavioral Adjustment: Strongly associated with Actionable (RR=1.22), Timely (RR=1.24), and Urgent (RR=1.11) feedback. Conversely, Encouraging and Reflective feedback were associated with reduced behavioral adjustment, likely because they affirm correct behavior or stimulate long-term insight rather than immediate correction.
Verbal Acknowledgment: Strongly associated with Reflective (RR=1.34) and Clear (RR=1.13) feedback. Encouraging and Timely feedback were associated with reduced verbal acknowledgment.

Significance and Claims

The paper claims to advance the field of surgical education and communication analysis in several ways:

Novel Discovery Mechanism: Unlike prior work relying on predefined taxonomies or unsupervised clustering, this framework uses a two-stage, LLM-guided strategy to discover novel, human-interpretable scoring criteria grounded in real-world clinical interactions.
Scalability and Interpretability: The framework provides a scalable, automated method for assessing feedback delivery quality without the need for extensive manual annotation. The use of BARS ensures the criteria are interpretable by both humans and machines.
Clinical Relevance: The discovered dimensions are not merely linguistic features but are behaviorally grounded, showing statistically significant associations with trainee behavioral adjustments and trainer reactions.
Practical Application: The authors posit that this framework can underpin point-of-care dashboards, longitudinal curriculum analytics, and automated quality-assurance audits. It offers a pathway to improve intraoperative teaching effectiveness and supports trainer development.
Generalizability: While developed for the operating room, the authors suggest the pipeline could generalize to other clinical settings relying on spoken guidance (e.g., ICU rounds, telemedicine), provided the input is text-based.

The authors maintain a modest stance regarding limitations, acknowledging that the analysis relies solely on text (excluding prosodic or visual cues), that the evaluation was retrospective, and that the rubric discovery relied on a proprietary model (GPT-4o), though they have published the final rubric to enable re-implementation with open-source models.