💻 computer science

Sample-Efficient Learning with Online Expert Correction for Autonomous Catheter Steering in Endovascular Bifurcation Navigation

This paper presents a sample-efficient reinforcement learning framework enhanced by online expert correction and a fuzzy controller, which significantly improves the convergence speed and positional accuracy of autonomous catheter steering in complex endovascular bifurcation navigation compared to baseline methods.

Original authors: Hao Wang, Tianliang Yao, Bo Lu, Zhiqiang Pei, Liu Dong, Lei Ma, Peng Qi

Published 2026-02-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Wang, Tianliang Yao, Bo Lu, Zhiqiang Pei, Liu Dong, Lei Ma, Peng Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to navigate a tiny, flexible snake (a catheter) through a complex, twisting maze of pipes (your blood vessels) to reach a specific destination deep inside your body. This is what doctors do during minimally invasive heart or kidney surgeries.

Doing this by hand is hard. The pipes are squishy, the view is blurry (like looking through a foggy window), and if you push too hard or turn the wrong way, you could damage the pipe walls. Plus, the doctor has to stand under bright X-ray lights for hours, which is bad for their health.

This paper introduces a robotic "co-pilot" that learns to steer this snake-catheter automatically, but with a special twist: it learns by doing, but it also has a human expert hovering over its shoulder to give it a gentle nudge when it gets confused.

Here is how the system works, broken down into simple concepts:

1. The Problem: Learning by Trial and Error is Slow

Usually, if you want a robot to learn a new skill, you let it try, fail, and try again (this is called Reinforcement Learning).

The Analogy: Imagine teaching a toddler to walk through a dark forest. If you just say, "Go find the treasure," they might wander in circles for days, trip over roots, and get lost. They need a lot of tries to figure it out.
The Issue: In surgery, you can't let a robot wander around blindly. It needs to learn fast, and it needs to be safe. Standard AI methods are too slow and rely on "static maps" (like a GPS that doesn't update when the road changes).

2. The Solution: The "Expert Co-Pilot" System

The authors created a smart system called SAC-EIL-GAIL. Let's break that scary name down into three friendly parts:

Part A: The "Eyes" (Seeing the Snake)

Before the robot can steer, it needs to see where the catheter tip is.

The Analogy: Think of this like a high-tech video game where the computer draws a glowing line right down the center of the snake's body. Even if the image is fuzzy, the robot uses a special "skeleton" trick to trace the exact path of the catheter in real-time.

Part B: The "Brain" (Learning the Rules)

This is the Reinforcement Learning part.

The Analogy: The robot is playing a video game. It gets points for moving closer to the goal and loses points if it hits the wall.
The Innovation: Instead of just guessing randomly, the robot is taught using Generative Adversarial Imitation Learning (GAIL).
- Imagine a Teacher (the AI) and a Student (the Robot). The Teacher tries to trick the Student into thinking it's an expert. The Student tries to act so much like a human expert that the Teacher can't tell the difference. This forces the robot to learn the style of a pro surgeon, not just the math.

Part C: The "Safety Net" (Online Expert Correction)

This is the most important part. When the robot reaches a bifurcation (a fork in the road where the vessel splits into two), things get tricky.

The Analogy: Imagine you are driving a car approaching a fork. You aren't sure which way to turn. Suddenly, a human expert in the passenger seat says, "No, turn left! That's the right path," and gently steers the wheel for you.
How it works: The robot detects the fork. If it looks unsure, it pauses and asks the human expert for the "target pose" (where the tip should be). The robot then uses a Fuzzy Controller (a smart logic system that handles "maybe" and "sort of" rather than just "yes" and "no") to adjust its movement to match the expert's suggestion perfectly.

3. The Results: Faster, Smarter, Safer

The researchers tested this on a realistic plastic model of a kidney artery.

The Race: They compared their new system against older AI methods.
The Winner: The new system learned to navigate the maze 26% faster than the standard method.
The Accuracy: It made fewer mistakes. While other robots were wandering off course, this one stayed on the "expert path," reducing the error by about 16%.

Why This Matters

Think of this technology as training wheels for robotic surgery.

Safety: It reduces the time doctors spend under X-ray lights.
Precision: It can navigate tiny, tricky forks in the blood vessels better than a human hand can sometimes.
Adaptability: Because it learns from a human expert while it's working (online), it can handle unexpected changes in the patient's anatomy, unlike older robots that just follow a pre-written script.

In a nutshell: This paper presents a robot that doesn't just "learn by doing" but "learns by doing and listening to a master." It combines the speed of AI with the wisdom of a human surgeon to navigate the complex, winding roads of our bodies with incredible precision.

1. Problem Statement

The paper addresses the challenges of autonomous catheter steering during endovascular interventions, specifically at vascular bifurcations (where blood vessels split). While robot-assisted systems reduce radiation exposure and operator fatigue, achieving full autonomy remains difficult due to:

Sparse Rewards: Traditional Reinforcement Learning (RL) struggles with sparse terminal rewards (only knowing if the goal was reached at the end), leading to inefficient exploration.
Static Models: Many methods rely on preoperative static models that fail to adapt to intraoperative variations, imaging ambiguities, and patient-specific anatomy.
Complex Dynamics: The interaction between flexible catheters and deformable vessels involves friction, backlash, and 2D imaging foreshortening, making precise real-time control difficult.
Safety Risks: Pure exploration in physical systems is risky; the system requires mechanisms to ensure safe navigation while learning.

2. Methodology: The SAC-EIL-GAIL Framework

The authors propose a hybrid framework integrating Soft Actor-Critic (SAC), Generative Adversarial Imitation Learning (GAIL), and Online Expert Correction (EIL). The system operates on a robotic platform with a transparent vascular phantom.

A. System Architecture

Hardware: A dual-arm robotic setup where one arm controls catheter insertion/rotation and the other provides stabilization.
Perception: A YOLO-based vision module detects the catheter tip, while a U-Net model segments the catheter and vessels from intraoperative fluoroscopic images.
State Representation: The state includes the skeletonized centerline, tip position, and orientation derived from image segmentation.

B. Core Components

Constant-Curvature Kinematic Modeling:
- The catheter is modeled using a constant-curvature approximation.
- A sub-pixel skeleton extraction algorithm (thinning + Savitzky-Golay smoothing) fits the catheter trajectory to a geometric model, enabling precise pose estimation.
Hybrid Reward Scheduling (SAC + GAIL):
- SAC (Exploration): Uses maximum entropy RL to encourage exploration.
- GAIL (Imitation): Uses a discriminator to distinguish between expert trajectories and agent trajectories, generating an imitation reward.
- Dynamic Weighting: A sigmoid-based mechanism smoothly transitions the reward weights from SAC-dominated (early training) to GAIL-dominated (later training). This allows the agent to explore initially and then refine its policy using expert priors.
- Total Reward: $R_t = w_{SAC}(t) \cdot r_{SAC} + w_{GAIL}(t) \cdot r_{GAIL} + \epsilon_t$ (where $\epsilon_t$ adds stochasticity).
Online Expert Correction (EIL) & Fuzzy Control:
- Trigger: When a bifurcation is detected, the system pauses autonomous RL and engages an "Expert-in-the-Loop" module.
- Correction: An expert (or pre-defined expert path) defines a target pose ( $D_{target}, P_{target}$ ).
- Fuzzy Logic Controller: Calculates translation ( $e_{trans}$ ) and rotation ( $e_{rot}$ ) errors between the current and target poses. A fuzzy controller maps these errors to robot actuation commands (translation/rotation) to correct the catheter's path in real-time, compensating for modeling errors and image noise.

3. Key Contributions

Unified SAC-EIL-GAIL Framework: A novel integration of maximum-entropy RL, adversarial imitation, and online expert supervision. This design significantly improves sample efficiency and training stability while reducing reliance on hand-crafted reward functions.
Sub-Pixel Kinematic Modeling: Coupling a constant-curvature model with skeleton-based centerline extraction allows for precise, sub-pixel trajectory fitting and control input mapping.
Robust Bifurcation Navigation: The introduction of a fuzzy logic controller driven by expert-in-the-loop corrections enables the system to handle complex bifurcation scenarios, compensating for image-derived modeling errors in real-time.

4. Experimental Results

Experiments were conducted on a 3D silicone renal artery phantom using a custom robotic platform. The proposed method was compared against TD3, standard SAC, SAC-GAIL, and SAC-EIL.

Convergence Efficiency:
- The proposed SAC-EIL-GAIL converged in 123 episodes, a 25.9% reduction compared to the baseline SAC (166 episodes) and TD3 (175 episodes).
- Convergence time was reduced to 59.41 seconds.
Success Rate:
- Achieved a 59.00% success rate, outperforming SAC (44.67%), TD3 (41.67%), and ablation variants (SAC-GAIL: 53.67%, SAC-EIL: 51.33%).
Accuracy (Error Reduction):
- Achieved the lowest average positional error of 82.58 pixels.
- This represents an 83.8% relative error compared to the baseline SAC (98.55 pixels) and a significant reduction compared to TD3 (208.76 pixels).
Statistical Significance: Boxplot analysis confirmed that SAC-EIL-GAIL has lower variance and tighter error distribution, with statistically significant improvements ( $p < 0.05$ ) over other methods.

5. Significance and Future Work

Clinical Impact: The framework demonstrates that combining sample-efficient RL with online expert correction can achieve reliable, high-precision catheter steering in anatomically challenging bifurcations, a critical step toward fully autonomous endovascular surgery.
Safety & Adaptability: The hybrid approach ensures safety during exploration by leveraging expert priors and fuzzy correction, addressing the "sim-to-real" gap in dynamic vascular environments.
Future Directions: The authors plan to extend the system to multi-instrument collaboration (guidewires and catheters), develop adaptive online expert feedback mechanisms, and enhance generalization through simulation-to-real transfer for patient-specific anatomies.

In summary, this paper presents a robust solution to the sample inefficiency and safety challenges of autonomous catheter navigation by creating a synergistic loop between reinforcement learning, adversarial imitation, and real-time expert-guided fuzzy control.