📄 medicine

Examining Tools for Assessing Technical Skills in Robotic Surgery: A Validity- Focused Systematic Review

This systematic review evaluates the validity of current assessment tools for robotic surgery, revealing that while existing methods like GEARS and OSATS have strong internal structure evidence, there is a significant gap in validated tools covering most ACS/APDS skill modules and a lack of evidence regarding the consequences of assessment.

Original authors: Naimisha Badri, Amber Chen-Goodspeed, Folarin Adeyemi, Iman Ghaderi

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Naimisha Badri, Amber Chen-Goodspeed, Folarin Adeyemi, Iman Ghaderi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching someone to drive a car. In the old days, you might have just said, "You're ready when I say you are," based on how many miles they drove with you. But today, we use driving tests with specific checklists: "Did they check the mirrors? Did they signal? Did they stop at the red light?" This ensures the driver is actually safe, not just lucky.

This paper is about doing the same thing for robotic surgery, but it finds that we are still missing a lot of the checklists.

Here is the breakdown of what the researchers found, using simple analogies:

1. The Problem: We Have a New Car, But No Driver's Ed Manual

Robotic surgery is like a high-tech, futuristic car. It's different from regular surgery (laparoscopy) because the surgeon sits at a console, uses wristed instruments, and doesn't have the same "feel" (haptic feedback) as using their hands directly.

Because this technology is so new and different, the researchers asked: "How do we know a surgeon is actually good at using the robot?"

They looked at the standard "driving curriculum" used for all surgeons (created by the American College of Surgeons). This curriculum has 45 specific skills a surgeon must learn, ranging from basic things like tying knots to complex procedures like removing a gallbladder.

2. The Search: Looking for the "Test Questions"

The researchers went on a hunt (a systematic review) to find existing "tests" or "scorecards" that measure these specific robotic skills. They wanted to see if there was a valid way to grade a surgeon on every single one of those 45 skills.

The Result: It was a bit of a disaster.

Out of the 45 skills surgeons need to learn, they only found valid tests for 13 of them.
That means for 32 skills (about 71%), there is no test at all. It's like having a driving school where you have to learn how to parallel park, but there is no test to see if you can actually do it. You just have to hope you're good.

3. The Tools: Using Old Maps for New Territory

For the 13 skills that did have tests, the researchers noticed something else. Most of the time, they weren't using special "robotic" tests. Instead, they were taking tests designed for regular surgery or laparoscopy and just trying to use them on robots.

The Main Tools: The most common tools found were GEARS (Global Evaluative Assessment of Robotic Skills) and OSATS (Objective Structured Assessment of Technical Skills).
The Analogy: It's like trying to test a Formula 1 driver using a test designed for a standard sedan. It might catch some basic driving errors, but it might miss the specific high-speed skills needed for the F1 car.

4. The "Report Card" Check: Do the Tests Actually Work?

The researchers didn't just count the tests; they checked the "report cards" (validity evidence) to see if the scores these tools give are actually trustworthy. They looked at five different types of proof:

Content: Does the test cover the right stuff? (Mostly okay).
Response Process: Do the graders understand what they are looking at? (Okay).
Internal Structure: Do the different parts of the test fit together logically? (Strong).
Relationships: Do the scores match up with how experienced the surgeon is? (Strong).
Consequences: Does using this test actually make patients safer or help surgeons learn better?

The Big Gap:
The researchers found that while the tests were good at measuring skills (like "did they tie the knot?"), they were terrible at proving Consequences.

Analogy: We have a test that says, "This student passed the driving test." But we have almost no proof that passing this test actually means they won't crash their car in real life. The link between the test score and real-world safety is the weakest part of the evidence.

5. The Conclusion: We Need to Build More Tests

The paper concludes that while we have started building the "driving school" for robotic surgery, we are missing most of the curriculum.

What we have: A few tests for basic skills (like tying knots or simple tissue handling) and some tests for specific procedures (like removing a colon).
What we are missing: Tests for the majority of complex procedures and a lack of proof that these tests actually predict patient safety.

The Bottom Line:
The authors say we need to stop guessing. We need to build specific, high-quality tests for the many robotic skills that currently have no assessment, and we need to prove that these tests actually help surgeons become safer and better. Until then, we are flying blind on a huge chunk of robotic surgical training.

Technical Summary: Examining Tools for Assessing Technical Skills in Robotic Surgery

Problem Statement
Objective assessment of operative performance is a cornerstone of competency-based surgical education. While structured tools like the Objective Structured Assessment of Technical Skills (OSATS) are well-established for open and laparoscopic training, equivalent assessment methods for robotic surgery remain fragmented. Robotic surgery demands distinct cognitive and psychomotor abilities—such as console ergonomics, wristed instrument control, and navigation without direct haptic feedback—that differ from conventional laparoscopy. Despite the rapid integration of robotic surgery into training, technical proficiency is often inferred from case volume rather than objectively demonstrated skills. Furthermore, the validity evidence supporting existing robotic assessment tools is not comprehensively characterized. Under the contemporary unitary framework of validity, an instrument's scores must be supported by evidence across five domains: content, response process, internal structure, relationships to other variables, and consequences. This study addresses the gap in understanding the current landscape of robotic technical assessment and the strength of validity evidence supporting these tools.

Methodology
The authors conducted a systematic review via a PubMed search to identify current robotic surgical technical skills assessment tools. The search utilized keywords including "Objective," "Assessment," "Tool," "Skill," "Evaluation," "Proficiency," and "Robot," combined with procedure-specific terms.

Framework Mapping: The study mapped identified tools to the 45 procedural modules of the American College of Surgeons (ACS) and Association of Program Directors in Surgery (APDS) skills curriculum. Specifically, 23 robotic equivalents of these modules were identified; modules without a recognized robotic equivalent were classified as unstudied domains.
Validity Rating: An established rating scale, adapted from Ghaderi et al., was applied to rate the strength of evidence for each study based on the five sources of validity:
- Content: Representation of the construct and domain.
- Response Process: Alignment of rater actions and scoring with the construct.
- Internal Structure: Statistical alignment of components (reliability, factor analysis).
- Relationships to Other Variables: Correlation with external variables (e.g., novice vs. expert discrimination).
- Consequences: Intended benefits and unintended harms of assessment use.
- Scores ranged from 0 (no evidence) to 3 (comprehensive evidence).
Data Analysis: Two independent reviewers scored each study, reaching a consensus. Interrater reliability (IRR) was calculated using weighted Cohen's $\kappa$ and the Intraclass Correlation Coefficient (ICC).

Key Contributions

Systematic Mapping: The study provides a structured mapping of existing robotic assessment tools against the standard ACS/APDS 45-module curriculum, identifying exactly which skill domains have corresponding literature and which remain unstudied.
Validity Evidence Quantification: It is the first study to comprehensively evaluate the strength of validity evidence for robotic assessment tools using the unitary framework, assigning numerical scores to the five sources of validity across the identified literature.
Reliability Metrics: The study establishes the reliability of the review process itself, demonstrating substantial interrater agreement in scoring validity evidence.

Results

Module Coverage: Of the 45 ACS/APDS skill modules, only 13 (28.9%) had corresponding published papers. The majority of identified papers utilized previously studied assessment tools, most notably the Global Evaluative Assessment of Robotic Skills (GEARS) and OSATS.
Validity Evidence Strength:
- Highest Evidence: "Internal structure" and "Response process" demonstrated the highest levels of evidence (average scores of 2.53 and 2.63, respectively).
- Moderate Evidence: "Content" and "Relationships to other variables" showed moderate to strong supporting evidence (average scores of 2.47).
- Lowest Evidence: "Consequences" demonstrated the weakest level of evidence (average score of 1.32), indicating a lack of data on how assessments impact outcomes or decision-making.
Interrater Reliability: The overall IRR was substantial, with a weighted $\kappa$ of 0.754 and an ICC of 0.757. Reliability was highest for "Content" ( $\kappa=0.746$ ) and lowest for "Consequences" ( $\kappa=0.510$ ) and "Response process" ( $\kappa=0.514$ ).

Significance and Claims
The authors claim that the findings indicate a significant lack of available assessment tools for robotic surgery, with the majority of skill modules having no corresponding papers. The review highlights that while tools like GEARS and OSATS are widely used, they often rely on adaptation rather than novel, procedure-specific development.

The paper concludes that the literature evaluating these instruments remains limited, and relatively few studies have examined robotic surgical skill assessment using the contemporary unitary framework of validity. Consequently, many tools currently used in robotic surgical training possess only partial evidence supporting the interpretation and appropriate application of their scores. The authors assert that future work must focus on developing and validating global and procedure-specific assessment tools, particularly as robotic surgery becomes increasingly integrated into practice. They emphasize that strengthening validity evidence is essential to ensure these instruments can be confidently used for training, feedback, and competency evaluation, moving beyond case volume to objective demonstration of skill.