💻 computer science

Attribute Retrieving for Open-Vocabulary Endoscopic Compositional Referring Segmentation

This paper introduces ReferEndoscopy, a large-scale benchmark for referring image segmentation in endoscopy, and proposes the AR-ERIS framework, which leverages attribute retrieval to achieve state-of-the-art open-vocabulary compositional segmentation with strong generalization across simulated and real-world data.

Original authors: Shun Liu, Nan Xi, Yang Liu, Tianyu Luan, Xuan Gong, David Doermann

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shun Liu, Nan Xi, Yang Liu, Tianyu Luan, Xuan Gong, David Doermann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a live feed from inside a human body, like a tiny camera exploring a dark, squishy cave. This is an endoscope. Now, imagine a surgeon needs to grab a specific tool or point out a specific piece of tissue, but instead of pointing with a finger, they just say, "Grab the red, long thing on the left."

In the past, computer programs trying to understand these requests were like a clumsy robot that only knew the word "tool" or "organ." If you asked for "the red tool," the robot might grab the wrong one because it didn't understand that "red" and "long" were important clues. This paper introduces a new, super-smart system called AR-ERIS that acts like a detective, piecing together tiny clues (attributes) to find exactly what the surgeon is talking about.

The Big Problem: The "Blind" Robot

The authors found that existing computer models were struggling in the messy, tricky world of surgery. They argued against the idea that just showing a picture and a simple name (like "forceps") is enough. In the real world, tools move, tissues get squished, and lighting changes. The paper explicitly rules out the idea that current models can handle complex descriptions like "the instrument which is red, located in the bottom-left, touching the tissue." Without special training, these models get confused and fail to generalize to new, unseen situations.

The Solution: A New "Library" and a "Clue Hunter"

To fix this, the team built a massive new library called ReferEndoscopy. It's not just a few pictures; it's a huge collection of 65,964 endoscopic images with 242,055 detailed masks (outlines) and over 1.4 million text-image pairs. They didn't just label them "liver" or "tool." They broke every object down into specific clues: its color, size, texture, shape, and exactly where it is sitting relative to other things.

Think of this library as a giant database where every object has a "ID card" filled with these specific details.

Then, they built the AR-ERIS framework. This is the detective. Here is how it works, using a fun analogy:

The Frequency Filter: Imagine looking at a photo through two different pairs of glasses. One pair blurs the colors but highlights the sharp edges (like the outline of a metal tool). The other pair highlights the smooth, soft areas (like a piece of fat or skin). The paper suggests that endoscopic images are special because they have these distinct "frequency" parts. The new model puts on both pairs of glasses at once to understand the scene better than models that only look at one type of detail.
The Attribute Retrieval (The Clue Hunter): This is the magic part. When the surgeon says, "Find the red thing," the model doesn't just guess. It goes to its memory bank (the Attribute Database) and pulls up the "ID cards" for everything it knows. It checks: "Which tool is red? Which one is elongated? Which one is on the left?" It retrieves the specific clues that match the description.
The Match: If the surgeon asks for something the model has never seen before (like a new type of robotic arm), the model uses its training to find the most similar clues from its memory bank. It's like saying, "I've never seen this exact tool, but I know it looks like a 'probe' that is 'round' and 'black,' so I'll look for those features."

What the Numbers Say

The paper tested this detective system against other top models. The results were pretty clear:

When the instructions were simple (just the name of the object), the new model got a score of 73.76% (mIoU), beating the next best model which only got 39.21%.
When the instructions got harder (adding more clues like color and location), the new model stayed strong. For example, with "hard" instructions, it scored 74.23%, while other models dropped significantly.
In a "stress test" using a completely new dataset (SAR-RARP50) that the model had never seen before, the new system achieved a 21.32% score, while a famous competitor (GroundedSAM) only managed 9.25%. This suggests the model is actually learning the rules of the game, not just memorizing the answers.

What the Paper Doesn't Claim

It's important to note what the paper doesn't say. The authors are careful to state that this is a new benchmark and a new framework, but they don't claim it's a perfect, finished product ready for every hospital tomorrow. They suggest that this approach could help in real-time navigation and robotic assistance, but they present these as potential future steps, not current realities. They also note that while their model handles "open-vocabulary" (new words) well, it still relies on the specific attributes it was trained to recognize.

The Bottom Line

This paper suggests that by teaching computers to look for specific, tiny details (attributes) and by giving them a massive, organized library of examples, we can build systems that understand complex, natural language commands in surgery. It's a step toward a future where a surgeon can just talk to the robot, and the robot knows exactly which red, squishy, left-side tool to grab, even if it's never seen that exact tool before. The authors show that this method works better than current state-of-the-art models, but they frame it as a strong foundation for future research rather than a final solution to all medical imaging problems.

Technical Summary: Attribute Retrieving for Open-Vocabulary Endoscopic Compositional Referring Segmentation

Problem Statement

Referring Image Segmentation (RIS) aims to segment image regions based on natural language descriptions, enabling fine-grained visual understanding. While successful in natural images, extending RIS to endoscopic imagery presents unique challenges:

Data Scarcity and Quality: There is a lack of high-quality, instruction-grounded annotations in the medical domain.
Complex Domain Specificity: Endoscopic images feature complex textures, occlusions, dynamic interactions between instruments and tissues, and challenging conditions like low lighting and smoke.
Fine-Grained Requirements: Medical descriptions require the interpretation of subtle anatomical and instrumental variations (e.g., specific instrument parts, tissue types) that go beyond typical object descriptions.
Class Imbalance: Endoscopic datasets often exhibit long-tail distributions where specific organs or instruments appear far more frequently than others.
Generalization Gap: Existing methods rely heavily on image-only data within limited domains, lacking the textual guidance necessary for robust open-vocabulary generalization.

Methodology: AR-ERIS Framework

The authors propose Attribute Retrieval-based Endoscopic-RIS (AR-ERIS), a framework designed for open-vocabulary endoscopic compositional referring segmentation. The core components include:

1. ReferEndoscopy Benchmark

To address the data scarcity, the authors constructed ReferEndoscopy, a large-scale benchmark comprising:

65,964 endoscopic images from 10 diverse datasets (including AutoLaparo, DSAD, Cholecseg8k, EndoVis-17/18, Kvasir-Instrument, RoboTool, SAR-RARP50, SISVSE, and LaparoI2I).
242,055 instruction-grounded masks.
1,452,330 image-mask-instruction triplets.
59 categories covering instruments (parts), organs, and tissues.

Attribute-Guided Annotation Pipeline:

Step 1 (Collection): Aggregating data from diverse domains and preprocessing (cropping black backgrounds, resizing to 448×448).
Step 2 (Attribute Recognition): Using OpenCV to automatically extract attributes for each mask, including:
- Color: Mean color mapped to labels (e.g., red, silver).
- Size: Mask area mapped to qualitative labels (small, medium, large).
- Spatial Location: Centroid calculation for relative position (e.g., bottom-left).
- Reference/Relation: Distance calculations between contours to determine relationships (e.g., touch, near, overlap).
Step 3 (Instruction Design): Generating adaptive instructions using a template engine that combines attributes into prompts of varying complexity:
- Naive: Class name only.
- Simple/Medium/Hard: Increasing numbers of attributes (1, 3, or 5) describing color, texture, location, shape, and size.

2. AR-ERIS Architecture

The framework consists of a dual encoder, Feature Pyramid Network (FPN), cross-model attention, and a projector.

Frequency-Aware Referring Network (FAR-Net):
- Frequency Decomposition: Unlike standard methods, FAR-Net applies a 2D Fast Fourier Transform (FFT) to decompose input images into low-frequency (texture/organ body) and high-frequency (boundaries/edges) components.
- Freq-Fusion: The model separates these components using frequency masks, applies inverse FFT, and concatenates them with the original RGB image ( $I_{combined} = \text{Concat}(I, I_{low}, I_{high})$ ). This allows the model to perceive class rigidity and deformation independently of color.
- Feature Fusion: A mixture-of-experts network fuses these multi-frequency features with original visual features.
- Cross-Modal Alignment: Text prompts are embedded via a CLIP text encoder and aligned with image features through a cascaded Transformer decoder and cross-attention mechanisms.
Attribute Retrieval Module (ARM):
- Designed to handle open-vocabulary and novel attributes.
- During inference, if a testing class is novel, ARM computes similarity scores between the target class and pre-training classes to retrieve the most relevant attributes (color, size, texture, location) from a class-attribute memory bank.
- This enables the generation of accurate instructions for unseen classes based on pre-trained attribute co-occurrences.
Training Strategy:
- Loss Functions: The model is optimized using a combination of Binary Cross-Entropy (BCE), Dice Loss ( $L_{Dice}$ ), and a novel Frequency Consistency Loss ( $L_{freq}$ ).
- $L_{freq}$ specifically penalizes discrepancies between the high-frequency image component and the segmentation mask, forcing the model to capture sharp edge details crucial for surgical instruments.

Key Contributions

ReferEndoscopy Benchmark: The creation of a large-scale, instruction-grounded dataset covering diverse endoscopic domains, instruments, and anatomical structures, addressing the lack of fine-grained text-image data in surgery.
AR-ERIS Framework: A novel architecture that:
- Decomposes RGB inputs into distinct frequency components to capture boundary, shape, and texture features.
- Utilizes a mixture-of-experts network for adaptive feature selection.
- Employs an Attribute Retrieval Module (ARM) to handle open-vocabulary settings and attribute retrieval for novel classes.
State-of-the-Art Performance: Demonstrated superior performance over baseline models (e.g., GroundedSAM, LAVT, CRIS) across various instruction complexities and datasets.

Experimental Results

Benchmark Performance: On the ReferEndoscopy dataset, AR-ERIS achieved state-of-the-art results. The "medium" variant (pre-trained with 3 attributes) achieved an mIoU of 73.98% and DICE of 85.04% on the overall benchmark, outperforming baselines like LAVT (38.75% mIoU) and GroundedSAM (9.84% mIoU).
Robustness to Complexity: While natural image baselines suffered significant performance drops as instruction complexity increased (e.g., "naive" to "hard"), AR-ERIS variants pre-trained on attributes maintained robustness, showing gains of 2.50% and 1.61% in mIoU for "medium" and "hard" settings respectively compared to naive baselines.
Generalization (Zero-Shot): On the unseen SAR-RARP50 dataset (robotic-assisted radical prostatectomy), AR-ERIS achieved a 21.32% mIoU (hard variant), significantly outperforming GroundedSAM (9.25% mIoU). This demonstrates the model's ability to generalize to new surgical domains and anatomies without specific training.
Component Analysis:
- Dice Loss ( $L_{Dice}$ ): Improved performance on soft tissue segmentation (e.g., liver ligaments, abdominal wall).
- Frequency Loss ( $L_{freq}$ ): Crucial for instrument segmentation, yielding massive IoU gains (e.g., +21.46% for electric hooks) by preserving sharp, high-frequency edges.
- ARM Module: Enabled effective attribute retrieval, improving mIoU from 72.35% to 74.85% when moving from naive to hard instruction subsets.

Significance

The paper claims that this work establishes a foundational approach and benchmark for vision-language grounding in the medical domain. By leveraging attribute retrieval and frequency-aware feature fusion, AR-ERIS addresses the specific challenges of endoscopic imagery, such as class imbalance, complex textures, and the need for fine-grained, open-vocabulary segmentation. The authors posit that this framework has the potential to enhance real-time navigation and provide context-aware robotic assistance in minimally invasive procedures, moving beyond static object detection to dynamic, instruction-driven surgical support.