Automated Assessment of Kidney Ureteroscopy Exploration for Training

This paper presents a novel, automated ureteroscope video-based framework that generates reference reconstructions to localize camera poses and identify missed calyces during kidney phantom explorations, thereby enabling effective out-of-the-operating-room training with objective feedback without requiring expert supervision.

The Gist

Imagine you are learning to drive a very tricky, winding mountain road. The road is inside a giant, dark cave (your kidney), and your goal is to visit every single little side-cave (called a calyx) to make sure you don't leave any hidden treasures (kidney stones) behind.

Right now, learning to drive this road is hard. You need a driving instructor sitting right next to you in the car, watching every move, and only giving you feedback after you've finished the whole trip. This is expensive, scary (because it happens on real patients), and there aren't enough instructors for everyone.

This paper introduces a "Smart Driving Coach" that works on a practice model (a fake kidney) and gives you instant, automatic feedback without needing a human to watch you.

Here is how it works, broken down into simple steps:

1. The "Perfect Map" (Stage 1)

Before any student tries to drive the road, the experts take a slow, super-careful tour of the fake kidney. They record this perfect journey.

• The Analogy: Think of this like a cartographer slowly walking through a cave with a high-end 3D scanner, creating a perfect, digital "Google Earth" map of every nook and cranny.
• The Tech: The computer uses this slow video to build a 3D model of the kidney's inside. It also marks exactly where the camera was at every second. This becomes the "Reference Model." Once this map is made for a specific fake kidney, it can be used forever for any student.

2. The "Student's Drive" (Stage 2)

Now, a trainee (a student surgeon) takes the camera and explores the same fake kidney at a normal, real-world speed. They might move fast, the camera might get blurry, or they might shake the camera.

• The Problem: Usually, if the video is shaky or blurry, computers get lost and can't tell where they are.
• The Solution: The system doesn't try to build a new map from the student's messy video. Instead, it acts like a GPS that matches your current view to the Perfect Map.
  • Even if the student's video is blurry, the computer grabs a clear frame, looks at the Perfect Map, and says, "Ah, that blurry spot looks like the entrance to the 'Left Upper Cave' on the map!"
  • It does this frame-by-frame, stitching the student's messy journey onto the perfect map.

3. The "Report Card" (The Result)

Once the student finishes, the system compares their journey against the Perfect Map.

• The Output: It generates a visual report showing the kidney. It lights up the caves the student visited in Green and leaves the missed ones Red.
• The Score: It tells the student exactly which "side-caves" they missed. No human needs to watch the video to tell them this; the computer does it automatically in about 10 minutes.

Why is this a big deal?

• No Extra Hardware: You don't need expensive sensors attached to the camera (which can make the tool too thick to use). It just uses the video from the camera itself.
• Safe Practice: Students can practice as much as they want on the fake kidney without risking a real patient.
• Instant Feedback: Instead of waiting for a boss to say, "You missed a spot," the student gets a score immediately, like a video game.

Did it work?

The researchers tested this with 15 different student explorations.

• Accuracy: The system correctly identified missed caves 93% of the time.
• Precision: It knew where the camera was within 4 millimeters (about the width of a pencil eraser).
• Speed: It took about 10 minutes to grade a 2-minute video.

The Bottom Line

This paper presents a new way to train surgeons. Instead of relying on a human teacher to watch every second of a practice session, this system acts like an automatic tour guide. It builds a perfect map of the terrain once, and then checks every student's journey against that map to tell them exactly where they went wrong. This could help more surgeons learn faster, safer, and more effectively.

Technical Summary

1. Problem Statement

Kidney ureteroscopy is a complex procedure with a steep learning curve. A significant challenge is navigating the kidney's collecting system to ensure every cavity (calyx) is fully visited; up to 20% of patients require a second operation due to missed stones.

• Current Limitations: Traditional training relies on apprenticeship models in the Operating Room (OR), which are constrained by time, safety, and the availability of expert supervision. Feedback is often subjective, verbal, and provided only after the procedure.
• Existing Solutions: While anatomical phantoms exist for out-of-OR training, they typically lack automated feedback. Previous attempts to automate feedback using electromagnetic (EM) tracking increase hardware costs and complexity. Furthermore, standard computer vision methods (SLAM/SfM) often fail on the poor-quality, motion-blurred videos generated by trainees.

2. Methodology

The authors propose a purely video-based, two-stage framework that requires no additional hardware (e.g., EM sensors) other than a consumer-grade computer. The system uses a "reference reconstruction" approach to overcome the limitations of standard SfM on low-quality query videos.

Stage 1: Reference Model Generation

• Input: Two slow, thorough exploration videos of a specific kidney phantom, performed by experts.
• Process:
  • Frames from the two videos are concatenated.
  • A Structure from Motion (SfM) pipeline (using hloc toolkit: NetVLAD for retrieval, ALIKED for features, LightGlue for matching, COLMAP for reconstruction) generates a high-quality 3D point cloud of the phantom's collecting system.
  • The reconstruction is registered to a pre-existing CT segmentation of the phantom using Iterative Closest Point (ICP).
  • Individual calyces are manually annotated on the CT segmentation to serve as ground truth for visitation.
• Output: A reusable 3D reference model (point cloud + camera poses) and a segmented CT model with annotated calyces.

Stage 2: Query Localization and Assessment

• Input: Normal-speed exploration videos from trainees (query videos).
• Process:
  • Frame Localization: For each frame in the query video, the system performs a two-stage image retrieval against the reference model:
    1. Global retrieval (NetVLAD) to find candidate covisible reference images.
    2. Local feature matching (ALIKED/LightGlue) and RANSAC filtering to verify matches and estimate camera pose.
  • Consistency Filtering: Spatio-temporal filters remove outliers (e.g., frames localized outside the mesh or with unrealistic motion speeds).
  • Visitation Scoring:
    • The system renders the view from the estimated camera pose within the CT mesh.
    • Ray casting identifies which vertices of the CT segmentation are visible.
    • A visitation score is calculated for each calyx: $Score = \frac{\text{Viewed Vertices}}{\text{Total Vertices in Calyx}}$.
  • Classification: A threshold ($V S_{thd}$) determines if a calyx is "visited" or "missed." This threshold is determined via 5-fold cross-validation on trainee data.

3. Key Contributions

1. Hardware-Free Automation: The first framework to provide automated, objective feedback for kidney phantom training using only ureteroscope video, eliminating the need for expensive EM tracking hardware.
2. Reference-Based Localization: A novel two-stage approach that uses a high-quality "slow" reference reconstruction to localize challenging, low-quality "fast" trainee videos. This allows the system to function even when individual query frames cannot be reconstructed independently due to motion blur.
3. Calyx-Level Granularity: The system provides binary classification (visited/missed) for individual calyces, offering specific feedback on anatomical coverage rather than just general scope position.
4. Open Source: The codebase is made publicly available to facilitate further research.

4. Results

The system was evaluated using 4 silicone kidney phantoms and 15 trainee exploration videos.

• Reference Reconstruction Accuracy:
  • Mean Euclidean distance between the SfM point cloud and CT segmentation: < 2 mm.
  • 99th percentile Hausdorff distance: < 6.5 mm.
  • Coverage varied (43–66%) but successfully reconstructed clinically relevant calyces.
• Camera Pose Localization:
  • Using EM-tracked ground truth for validation, the mean localization error was < 4 mm for both reference and query videos.
  • Qualitative review confirmed that rendered views matched real anatomical landmarks accurately.
• Visitation Classification:
  • Accuracy: 69 out of 74 calyces were correctly classified across 15 videos.
  • Overall Accuracy: 92.8% (CI: 91.6%–94.0%).
  • Threshold Stability: The visitation threshold remained stable ($0.45 \pm 0.06$) across cross-validation folds.
• Runtime:
  • Reference generation: ~40 minutes per phantom.
  • Query processing: ~10 minutes for a 1-2 minute video (1000 frames), enabling semi-real-time feedback.

5. Significance and Future Implications

• Scalable Training: By removing the need for expert supervision and expensive hardware, this system enables scalable, out-of-OR training, allowing trainees to practice repeatedly with immediate, objective feedback.
• Objective Metrics: It shifts surgical training from subjective judgment to quantitative, data-driven assessment of skill (specifically, exploration completeness).
• Preoperative Planning: Since the phantoms can be fabricated from patient-specific CTs, the system could be used pre-operatively to identify difficult-to-reach calyces, helping surgeons plan strategies to avoid missed stones.
• Limitations & Future Work: The system occasionally misclassifies calyces if the view is extremely brief or if the entire sequence exploring a region is blurred. Future work aims to incorporate view duration estimation and higher-level geometric features to improve robustness in edge cases.

In conclusion, this work demonstrates a viable path toward automated, objective surgical training for ureteroscopy, potentially reducing the learning curve and improving patient outcomes by minimizing missed stones.