Toward AI Autonomous Navigation for Mechanical Thrombectomy using Hierarchical Modular Multi-agent Reinforcement Learning (HM-MARL)

1. Problem Statement

Mechanical Thrombectomy (MT) is the standard treatment for acute ischemic stroke caused by large vessel occlusions. However, its accessibility is limited by geographic barriers, logistical delays, and a shortage of highly trained operators. While robotic systems offer a solution, autonomous navigation of endovascular devices remains a significant challenge due to:

• "Long-Horizon" Complexity: Navigating from the femoral artery to the internal carotid artery (ICA) requires a sequence of complex maneuvers over a long distance, which standard Reinforcement Learning (RL) agents struggle to learn effectively.
• Generalization: Most existing RL studies are limited to in silico (simulation) environments using single patient anatomies, failing to generalize to diverse patient vasculatures or in vitro (physical) testbeds.
• Multi-Device Coordination: MT requires the simultaneous or sequential manipulation of two devices (a guide catheter and a guidewire), a task rarely addressed by autonomous agents in realistic settings.

2. Methodology

A. Hierarchical Modular Multi-Agent RL (HM-MARL)

The authors propose a novel framework that decomposes the complex navigation task into manageable sub-tasks, rather than training a single agent to solve the entire path.

• Task Decomposition: The navigation from the femoral artery to the ICA is split into five distinct sub-tasks (e.g., navigating the aortic arch, entering the carotid artery).
• Modular Agents: Five specialized RL agents are trained, one for each sub-task.
• Hierarchical Control: A Task Selection Module (TSM) acts as a high-level controller. It observes the current state (device position and target) and selects the appropriate specialized agent to execute the next action. This avoids the complexity of concurrent multi-agent collaboration, instead using a sequential, rule-based strategy.
• Algorithm: All agents are trained using Soft Actor-Critic (SAC), a model-free RL algorithm known for sample efficiency and stability.
• Reward Function: A dense reward function is used, penalizing path length changes and providing a reward upon reaching the target.

B. Environment and Simulation (In Silico)

• Framework: The stEVE (simulated EndoVascular Environment) framework based on SOFA (Simulation Open Framework Architecture) was used.
• Data: Patient-specific vasculature was reconstructed from Computed Tomography Angiography (CTA) scans (neck and body). Ten distinct patient anatomies were generated by scaling and rotating a base dataset to ensure diversity.
• Physics: The simulation models a Navien 0.058" support catheter and a 0.035" guidewire, accounting for friction, stiffness, and vessel wall interactions.

C. Physical Testbed (In Vitro)

• Phantom: A 3D-printed transparent vessel phantom (Clear V4 resin) submerged in glycerine to match refractive indices for optical tracking.
• Robotics: A custom modular robotic manipulator using concentric tube robots (CTR) and stepper motors to independently translate and rotate the catheter and guidewire.
• Tracking: A 4K overhead camera tracks the device tip coordinates, providing the state input to the RL agent without visual feedback of the vessel geometry (simulating fluoroscopic constraints).

D. Experimental Design

Three models were compared:

1. HM-MARL-1: Trained on a single patient anatomy.
2. HM-MARL-10: Trained on ten diverse patient anatomies (to test generalization).
3. SA-RL-1: A Single-Agent RL baseline trained on the single anatomy (State-of-the-Art comparison).

3. Key Contributions

1. First In Vitro Autonomous MT Navigation: This is the first demonstration of fully autonomous navigation of two devices (catheter and guidewire) in a realistic in vitro MT vasculature testbed.
2. HM-MARL Framework: Introduction of a hierarchical, modular multi-agent approach that successfully breaks down long-horizon endovascular tasks, outperforming single-agent baselines.
3. Generalization Pipeline: Development of a scalable pipeline to generate diverse, patient-specific MT environments from CTA scans, bridging the gap between in silico training and in vitro validation.
4. Superhuman Challenge: The study includes a "superhuman" task (navigating the left common carotid artery with a straight catheter), which is notoriously difficult even for expert human operators.

4. Results

In Silico Performance

• Success Rates (SR): HM-MARL models achieved 92–100% SR on right-sided tasks and 56–80% SR on left-sided tasks across multiple anatomies.
• Comparison: HM-MARL significantly outperformed the Single-Agent (SA-RL) baseline. For example, on the long-horizon task (Femoral to ICA), SA-RL achieved 0% SR, while HM-MARL-1 achieved 100% (Right) and 84% (Left).
• Generalization: While HM-MARL-10 (trained on 10 anatomies) showed slightly lower SR than HM-MARL-1 (trained on 1) due to the bias-variance trade-off, it successfully generalized across diverse patient geometries.

In Vitro Performance

• Right-Sided Navigation: Both HM-MARL models achieved 100% SR navigating from the femoral artery to the Right Common Carotid Artery (RCCA) and 70–80% SR to the Right ICA.
• Left-Sided Navigation: Both models failed to navigate the left-sided vessels (0% SR). This was attributed to the acute angle of the left carotid take-off and the use of a straight catheter, which even expert human operators struggled with in the testbed (Expert SR: 20% on left).
• Simulation-to-Reality Gap: Procedure Times (PT) increased significantly in in vitro settings (e.g., from ~40s in simulation to ~250s in in vitro) due to robotic step delays and mechanical friction differences.

5. Significance and Future Work

• Clinical Impact: This work advances the Technology Readiness Level (TRL) of autonomous MT systems toward Level 4 (validation in relevant environments). It demonstrates that AI can learn complex, long-horizon endovascular tasks that were previously intractable for single agents.
• Clinical Translation: The authors advocate for Shared Autonomy as the immediate path forward, where the AI assists the operator while the clinician retains final decision-making authority.
• Future Directions:
  • Refining RL strategies using World Models to better simulate dynamics and improve generalization.
  • Validating on unseen in vitro data and expanding the dataset to include more complex anatomies.
  • Investigating image-based RL (using raw fluoroscopy) instead of state-based tracking.
  • Testing with specialized catheters (e.g., Simmons 2) to improve success rates on difficult left-sided anatomy.

In conclusion, the paper establishes a foundational framework for autonomous endovascular surgery, proving that hierarchical multi-agent reinforcement learning can effectively navigate complex vascular networks, though challenges in simulation-to-real transfer and difficult anatomical variations remain.