💻 computer science

RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination

This paper introduces RAG-RUSS, an interpretable, retrieval-augmented robotic framework that autonomously performs and explains a full carotid ultrasound examination by mimicking clinical workflows while overcoming data scarcity and the "black box" limitations of existing methods.

Original authors: Dianye Huang, Ziping Cong, Nassir Navab, Zhongliang Jiang

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dianye Huang, Ziping Cong, Nassir Navab, Zhongliang Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to drive a car, but instead of a human instructor sitting next to you, you have a robot. The problem with most current "robot drivers" for medical scans is that they are either too rigid (following a strict, dumb rulebook like "if you see a bump, turn left") or too mysterious (they use complex AI that acts like a "black box," where you can't tell why they are making a move). If a robot makes a mistake during a delicate ultrasound of your neck, a doctor needs to know why it did that to trust it.

Enter RAG-RUSS. Think of this system as a super-intelligent, talking robot sonographer that doesn't just "do" the scan, but explains what it's doing and why, just like a human expert would.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Black Box" vs. The "Rulebook"

The Old Way: Imagine a robot that only knows a strict rulebook: "Move left until you hit the thyroid." If the patient's anatomy is slightly different, the robot gets confused and stops.
The "Black Box" Way: Imagine a robot that learns by watching thousands of videos of doctors. It gets very good at moving the probe, but if you ask, "Why did you stop there?" it can't answer. It just says, "Because the math said so." This makes doctors nervous.

2. The Solution: The "Smart Intern" with a Library

RAG-RUSS is like a medical intern who has two superpowers:

It can see and understand the ultrasound images (like a human eye).
It has a "Retrieval-Augmented Generation" (RAG) brain. This is the magic sauce.

What is RAG?
Think of RAG as giving the robot a digital library of past successful scans right next to its brain.

When the robot is scanning a patient's neck, it pauses and asks its library: "Hey, I'm looking at this specific part of the neck. Has anyone else scanned this before? What did they see? What did they do next?"
It finds similar cases from its "library" (which was built from 32 real volunteers).
It uses those past examples to figure out: "Okay, I see the thyroid gland here. In similar past cases, the next step was to rotate the probe. I will do that, and I will tell the doctor why."

This means the robot doesn't need to memorize every possible situation (which requires massive data). Instead, it looks up the answer in its library, just like a student checking a textbook during an exam.

3. The Workflow: A Conversation with the Robot

The system is designed to follow the exact steps a human doctor uses to scan the carotid artery (the main blood vessel in the neck). It breaks the job down into 8 stages, like a checklist:

Find the start: "I see the common artery and the thyroid gland nearby."
Move down: "The thyroid is gone, so I'm moving further down."
Find the split: "Here is where the artery splits into two."
Rotate: "Time to turn the probe to see the length of the vessel."

The "Talking" Part:
At every single step, the robot speaks up.

Doctor: "What are you doing?"
Robot: "I am currently at the 'proximal common artery' stage. I can see the thyroid gland next to it. Based on similar scans I found in my library, the next logical step is to track forward."

This constant explanation builds trust. The doctor isn't just watching a robot move; they are watching a partner who explains its reasoning.

4. Why is this a big deal?

It's Transparent: You know exactly why the robot is moving. No more "black box" mysteries.
It's Data-Smart: Medical data is hard to get (you can't scan 10,000 people easily). Because RAG-RUSS can "look up" similar cases instead of needing to have seen every case before, it works well even with a smaller dataset.
It's Safe: By explaining its steps, a human doctor can intervene if the robot gets confused, making it safer for real-world use.

The Analogy Summary

If traditional robotic ultrasound is like a blindfolded person following a tape measure, and standard AI is like a genius who won't tell you how they solved the puzzle, then RAG-RUSS is like a skilled apprentice.

The apprentice has a reference book (the RAG library) full of photos and notes from previous successful exams. When they encounter a new patient, they look at the photo, flip through the book to find a similar case, say, "Ah, this looks like Case #42. In that case, we saw the thyroid here, so we rotated the probe. I will do the same," and then they do it.

This paper proves that by giving robots the ability to look up context and explain their thoughts, we can finally make autonomous medical robots that doctors actually trust and want to use.

1. Problem Statement

Robotic Ultrasound (RUSS) systems aim to overcome the operator-dependence and variability of conventional ultrasound examinations. However, existing autonomous RUSS methods face two critical limitations hindering clinical adoption:

Black-Box Decision Making: Current approaches rely heavily on rule-based policies or end-to-end learning models (Imitation Learning/Reinforcement Learning) that lack transparency. They cannot explain why a specific action is taken or what the current scanning stage is, raising safety concerns for clinicians.
Data Scarcity: Medical data is expensive and difficult to collect. Large-scale training datasets required for deep learning models (like Vision-Language-Action models) are often unavailable, leading to poor generalization.

2. Methodology: RAG-RUSS Framework

The authors propose RAG-RUSS, an interpretable framework that combines a Vision-Language Model (VLM) with Retrieval-Augmented Generation (RAG) to perform autonomous carotid artery scanning.

A. Clinical Workflow & Data Preparation

Workflow: The system mimics the standard clinical workflow for carotid examination, divided into 8 sequential stages:
1. Examine CCA proximal (Common Carotid Artery near thyroid).
2. Examine CCA distal.
3. Examine bifurcation.
4. Transverse scan completed.
5. Return to carotid bulb.
6. Return completed.
7. Rotate to longitudinal view.
8. Longitudinal scan completed.
Dataset: Due to safety constraints, the authors created a simulation environment using 32 human volumetric ultrasound scans (reconstructed from 32 healthy volunteers).
- A virtual probe was used to generate paired B-mode images and probe trajectories.
- The dataset includes 15,459 sliding-window entries with visual data and textual annotations (stage name, anatomical explanation, next API).
- Split: 28 volunteers for training, 4 unseen volunteers for testing.

B. System Architecture

RAG-RUSS consists of three main components:

Vision Encoder: A frozen PubMedCLIP-ViT-B/32 model pretrained on biomedical data (ROCO) to extract visual features from current and previous US frames.
Large Language Model (LLM): Vicuna v1.5 (7B) serves as the backbone. It processes visual embeddings, system prompts, and retrieved context to generate:
- The current scanning stage.
- An explanation of the stage (e.g., "Thyroid is near the CCA").
- The next API command (e.g., transverse_tracking, rotate).
Retrieval-Augmented Generation (RAG) Module:
- Context Retrieval: A ResMLP projects historical scanning contexts (two US frames, previous stage, VQA annotations) into an embedding space.
- Mechanism: During inference, the system retrieves the top- $k$ (usually 2) most similar historical scanning contexts from a database.
- In-Context Learning (ICL): These retrieved examples are fed to the LLM as additional context, allowing the model to reason based on similar past cases rather than relying solely on training weights.

C. Training Pipeline

The model is trained in three sequential stages:

Retrieval Engine Training: The ResMLP is trained using Triplet Loss to ensure that embeddings of the same scanning stage (from different volumes) are closer than those of different stages.
Cross-Modality Pre-training: A two-layer MLP projector aligns visual features from the frozen encoder with the LLM's text embedding space.
Instruction Fine-tuning: The LLM backbone (using LoRA adapters) and the projector are fine-tuned on multi-turn Question-Answer pairs (Stage, Explanation, Next API) using Next-Token Prediction loss.

3. Key Contributions

Interpretability: Unlike black-box RL/IL systems, RAG-RUSS explicitly outputs the current stage, a natural language explanation of the anatomical features, and the next action, enabling clinicians to monitor and trust the system.
Data Efficiency via RAG: By integrating RAG, the system reduces dependence on massive training datasets. It leverages retrieved similar cases to generalize to unseen patients, addressing the medical data scarcity bottleneck.
Unified Framework: It is the first robotic US system to integrate perception, explanation, and action in a unified VLM-RAG framework for full carotid artery examination (both transverse and longitudinal planes).

4. Experimental Results

The system was evaluated on 4 unseen volunteer volumes in a closed-loop simulation.

Retrieval Accuracy: The RAG module achieved 70.0% Top@1 and 76.2% Top@2 accuracy in retrieving relevant scanning contexts. Stages with distinct visual cues (e.g., "Examine CCA proximal") performed best (>95%), while ambiguous stages (e.g., "Transverse scan completed") were more challenging.
Stage Recognition Accuracy (Ablation Study):
- RAG-RUSS@2 (retrieving 2 examples) achieved the highest average accuracy of 79.1%.
- It significantly outperformed VLM Only (71.7%) and RAG Only (64.7%).
- Specific high performance was noted in anatomically clear stages: Examine CCA proximal (97.8%), Examine CCA distal (95.0%), and Transverse scan completed (100.0%).
Failure Analysis: The "Return completed" stage showed a performance drop when using 2 retrieved examples (25.0%) compared to 1 example (75.0%), suggesting that in cases of high temporal ambiguity, excessive retrieved context can introduce noise.

5. Significance and Conclusion

Clinical Trust: By providing step-by-step explanations, RAG-RUSS bridges the gap between autonomous robots and human clinicians, a crucial step for regulatory approval and routine clinical use.
Generalization: The RAG approach demonstrates that high performance can be achieved with limited medical data (32 volunteers) by leveraging retrieval-based reasoning, making it a viable solution for other data-scarce medical domains.
Future Outlook: While the system currently operates in simulation and uses discrete API controls, it establishes a foundational architecture for trustworthy, autonomous medical robotics that can adapt to patient variability through evidence-based reasoning.