🤖 machine learning

Towards Real-Time Autonomous Navigation: Transformer-Based Catheter Tip Tracking in Fluoroscopy

This paper presents a real-time, multi-threaded catheter tip tracking pipeline using deep learning segmentation models, specifically demonstrating that a two-class SegFormer architecture outperforms existing methods in accuracy and robustness to provide a reliable foundation for reinforcement learning-based autonomous mechanical thrombectomy navigation.

Original authors: Harry Robertshaw, Yanghe Hao, Weiyuan Deng, Benjamin Jackson, S. M. Hadi Sadati, Nikola Fischer, Tom Vercauteren, Alejandro Granados, Thomas C. Booth

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Harry Robertshaw, Yanghe Hao, Weiyuan Deng, Benjamin Jackson, S. M. Hadi Sadati, Nikola Fischer, Tom Vercauteren, Alejandro Granados, Thomas C. Booth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to "Drive" a Catheter

Imagine a stroke is like a traffic jam in a tiny, winding city inside your brain. To fix it, doctors use a long, flexible tube (a catheter) to navigate through blood vessels and clear the blockage. This is called a Mechanical Thrombectomy (MT).

Right now, a human doctor has to steer this tube manually while looking at X-ray movies (fluoroscopy). It's hard work, exposes the doctor to radiation, and takes a long time. The goal of this research is to build a robot that can steer the catheter autonomously (by itself) using Artificial Intelligence.

However, for a robot to drive, it needs to know exactly where the tip of the steering wheel is. In this case, the "steering wheel" is the tip of the catheter. The problem is that in X-ray movies, the catheter often looks faint, gets blurry, or hides behind other tools.

This paper asks: Can we build a computer system that can instantly and accurately find the tip of the catheter in these messy X-ray videos, even when it's hard to see?

The Solution: A "Smart Camera" Pipeline

The researchers built a four-step assembly line (a pipeline) to solve this. Think of it like a high-speed factory that processes every single frame of an X-ray video:

The Reader: Grabs the video frame.
The Preparer: Cleans up the image and gets it ready for the computer.
The Brain (Inference): A special AI model looks at the image and tries to draw a mask around the catheter and the wire inside it.
The Refiner: This is the clever part. The AI doesn't just guess a single dot. It:
- Merges broken pieces of the image.
- Turns the thick catheter shape into a thin, one-pixel-wide line (like tracing a path with a pencil).
- Finds the very end of that line (the tip).
- Crucially: To avoid getting confused by motion blur, it doesn't just look at the tip. It looks at the tip plus two points just behind it (like looking at the nose and two points on the forehead to know which way a face is turning). This helps the system guess the direction even if the tip is blurry.

The "Brains" Tested: Three Different AI Models

The team tested three different types of AI "brains" to see which one was best at drawing the catheter:

U-Net: A classic, reliable workhorse model.
U-Net + Transformer: The classic model with a "super-vision" upgrade that helps it understand the big picture.
SegFormer: A modern model that uses "Transformers" (the same tech behind advanced chatbots) to understand context and details simultaneously.

They tested these models in two ways:

Two-Class Mode: Just finding "The Tool" vs. "The Background."
Three-Class Mode: Finding "The Catheter," "The Wire," and "The Background" separately.

The Results: Who Won the Race?

The researchers tested the models on three types of "tracks":

The Practice Track (Phantom): A fake plastic artery in a lab.
The Live Lab Track (Live Camera): A real-time video of a catheter moving in a plastic model.
The Real Road (Human Patients): Actual X-ray videos from human surgeries.

The Winner:
The SegFormer model (the modern Transformer-based one) was the champion.

Accuracy: It made the fewest mistakes in finding the exact tip location. In the "moderate difficulty" human videos, it was off by an average of 4.44 millimeters.
Robustness: It handled the messy, low-contrast, and blurry parts of the X-rays better than the older models.
Speed: It was fast enough to run in real-time (about 30 frames per second), which is fast enough for a robot to react instantly.

A Surprising Twist:
While finding both the catheter and the wire separately (Three-Class) sounds more helpful for a robot, it actually made the tracking less accurate in some cases. The AI got confused by the extra details, causing the "line" it drew to branch out incorrectly. The simpler "Two-Class" approach (just finding "the tool") was often more stable and accurate for pinpointing the tip.

The Bottom Line

This paper proves that we can build a system that watches X-ray videos and instantly tells a robot where the catheter tip is, even when the image is noisy or the tool is hiding.

What they achieved: A stable, real-time tracking system that outperforms previous methods on standard benchmarks.
What they didn't achieve (yet): They didn't reach "perfect" sub-millimeter precision on real human patients (the error was a few millimeters, not less than 1). They noted this is because the AI was trained on plastic models and had to guess on real humans without extra training.
The Goal: This system provides the "eyes" needed for a robot to eventually drive the catheter on its own, potentially making stroke treatment faster and safer for doctors and patients.

In short: They built a very sharp pair of glasses for a robot, allowing it to see the steering wheel of a catheter clearly, even in a foggy X-ray room.

Technical Summary: Towards Real-Time Autonomous Navigation: Transformer-Based Catheter Tip Tracking in Fluoroscopy

Problem Statement
Mechanical thrombectomy (MT) is a critical intervention for large vessel occlusion strokes, yet its efficacy is time-sensitive and limited by a lack of local treatment access. While endovascular robotic platforms and reinforcement learning (RL)-based autonomous navigation offer potential solutions, current RL approaches require precise, real-time tracking of device tip coordinates (catheter and guidewire) to function. Existing device tracking methods struggle under clinically realistic conditions characterized by low image contrast, noise, motion blur, and device occlusion. Furthermore, it remains unclear whether deep learning models trained on synthetic or phantom data can generalize to in vivo clinical fluoroscopy, particularly when devices are partially hidden. The absence of robust, real-time tracking pipelines capable of operating across variable imaging modalities hinders the transition of autonomous navigation from in silico simulations to clinical application.

Methodology
The authors propose a multi-threaded, real-time catheter and guidewire tip-tracking pipeline designed to operate under challenging fluoroscopic conditions. The system integrates four asynchronous stages:

Reader: Captures raw fluoroscopy or video frames independently of inference speed to prevent dropped frames.
Preprocessor: Resizes, normalizes, and converts frames to tensor format.
Inference: Passes frames through trained segmentation models. For large images, a sliding-window approach with overlapping patches is used to balance quality and memory.
Post-processor: Refines segmentation masks through component merging, artifact removal, and skeletonization. Crucially, it employs a two-step component filtering and one-pixel medial skeletonization to identify endpoints. To stabilize tracking against motion blur and occlusion, the system samples three points along the skeleton: the primary tip ( $T_0$ ) and two downstream points ( $T_1, T_2$ ) at 5 and 10 pixels, respectively.

Model Architectures
Three deep learning segmentation architectures were trained and benchmarked using the CathAction dataset (25,000 pixel-level masks from in vitro phantoms and in vivo porcine subjects):

Standard U-Net: A modified 16-channel base U-Net optimized with AdamW to meet real-time memory constraints, differing from the original 64-channel SGD implementation in prior literature.
U-Net + Transformer: Replaces the U-Net bottleneck with a 16-layer, 16-head self-attention transformer module to capture broader context.
SegFormer: Utilizes a Mix Transformer B5 encoder with a symmetric U-Net decoder, incorporating spatial and channel squeeze & excitation (scSE) attention modules.

The study evaluated both two-class (instrument vs. background) and three-class (catheter vs. guidewire vs. background) formulations to determine their impact on downstream tracking accuracy.

Key Contributions

Novel Tracking Pipeline: Introduction of a skeletonization and multi-point sampling pipeline that improves localization stability in the presence of low contrast, partial occlusion, and overlapping instruments.
Integrated Benchmarking: Demonstration that modern segmentation architectures, when embedded within a tracking framework, outperform existing benchmarks on the CathAction dataset, extending evaluation beyond standalone segmentation metrics to include tip-localization error.
Cross-Domain Validation: Validation of the method across in vitro phantom, in vivo animal, and in vivo clinical fluoroscopic data without in vivo fine-tuning, assessing generalization across imaging modalities.

Results

CathAction Benchmark: The two-class SegFormer achieved the highest performance (Dice: 0.809, IoU: 0.722), outperforming U-Net and U-Net+Transformer. In three-class segmentation, SegFormer also led with a Dice score of 0.667, surpassing prior benchmarks (e.g., U-Net at 0.517).
In Vitro Offline RGB: SegFormer achieved the lowest Mean Absolute Error (MAE) in two-class tracking (0.47 mm). However, in three-class tracking, U-Net performed best (0.60 mm), while SegFormer suffered higher errors (1.21 mm) due to misclassification-induced skeleton branches.
In Vivo Fluoroscopy: On clinical data stratified by complexity, SegFormer (two-class) consistently demonstrated the most stable and accurate tip localization. In the "Moderate Complexity" group (G2), it achieved a mean absolute error of 4.44 mm, outperforming U-Net (4.60 mm) and U-Net+Transformer (6.20 mm). Three-class models generally exhibited higher errors and more artifacts in complex anatomical backgrounds.
Speed: The system maintained real-time performance, with inference speeds ranging from ~21 to ~30 FPS depending on the model and dataset.

Significance and Claims
The paper claims that the proposed multi-threaded tracking framework maintains stable performance under challenging imaging conditions, providing a reliable foundation for RL-based autonomous MT navigation. The authors emphasize that while the system outperforms prior benchmarks and demonstrates feasibility for real-time tracking, it does not yet achieve clinical-grade sub-millimeter precision (<1 mm) in in vivo settings. The observed errors (4–7 mm) are attributed to domain shifts (contrast, noise, motion), occlusion, and the lack of in vivo training data.

The study concludes that two-class segmentation offers the most reliable trade-off between speed, stability, and tip-localization accuracy for current tracking needs. However, three-class segmentation remains valuable for autonomous RL navigation, as it enables the simultaneous tracking of both catheter and guidewire, providing richer input for decision-making agents. The work positions itself as a step toward clinically deployable systems, highlighting that future improvements in domain adaptation and the inclusion of diverse in vivo training data are necessary to bridge the gap between phantom performance and clinical reality.