S3-Tracker: Self-Supervised Surgical Tissue Tracking With Contrastive Random Walks

Inside the human body, the view from a surgical camera is often a shifting landscape of soft, wet tissue that stretches, folds, and slides under the pressure of instruments. For surgeons and the robots that assist them, keeping a steady mental map of where every piece of tissue is located is a constant challenge. If a surgeon needs to match what they see on a screen with a pre-operative scan, like a CT or MRI, the task becomes nearly impossible when the tissue deforms in real time. To solve this, computers need to track specific points on the tissue from one video frame to the next, maintaining a continuous connection between the live view and the patient's internal anatomy. While modern artificial intelligence has become quite good at following objects in standard videos, the messy reality of surgery—filled with blood, smoke, and highly reflective surfaces—has made it difficult to teach computers how to do this reliably without massive amounts of human-labeled data.

A team of researchers has developed a new way to teach computers to track surgical tissue without needing those difficult-to-get labels. Instead of showing the computer thousands of videos where humans have painstakingly drawn dots and lines to show movement, the new method, called S3-Tracker, learns by watching the video itself and checking its own work. The system treats the video as a series of connected moments and asks a simple question: if I move from point A in one frame to point B in the next, and then try to move back from B to A, do I end up exactly where I started? By forcing the computer to answer this question correctly over and over again, it learns to understand how pixels move and deform, even when the tissue looks very different from one moment to the next. This approach allows the system to learn from vast amounts of unlabeled surgical footage, bypassing the bottleneck of needing human experts to annotate every single frame.

The researchers built their system to look at pairs of video frames and figure out how the pixels in one relate to the pixels in the next. They use a process that creates a map of connections between these points, essentially asking the computer to guess the most likely path a piece of tissue took. To make sure the computer isn't just guessing randomly, the system checks its own logic by running the movement forward and then backward. If the path forward and the path backward do not meet up at the same spot, the system knows it made a mistake and adjusts its understanding. This self-checking mechanism, which the authors call a contrastive random walk, allows the model to learn the complex ways tissue stretches and bends without ever seeing a single correct answer provided by a human. The system also includes a way to decide when a point is no longer visible, perhaps because it has been covered by blood or an instrument, and stops tracking it until it reappears, preventing the computer from getting confused and drifting off course.

When tested on real surgical video datasets, the new method proved to be a strong contender against existing systems that rely on partial human labeling. The researchers found that their self-supervised approach could track points with an accuracy that rivals methods trained with some human guidance, while being significantly faster to run in real time. In tests measuring how far off the predicted points were from the actual ground truth, the system performed well enough to be considered viable for clinical use, achieving an average accuracy score of 0.708 across various error thresholds. While fully supervised models, which use complete human labels, can sometimes be slightly more precise in terms of raw error distance, they are often too slow to work frame-by-frame during a live surgery. The new method, by contrast, operates efficiently enough to keep up with the video stream, processing frames at a rate of three times per second. This speed, combined with its ability to handle the visual chaos of surgery without needing pre-labeled data, suggests that self-supervised tracking could become a practical tool for guiding robotic surgery and helping surgeons navigate the complex, moving terrain of the human body.