💻 computer science

ISRS-DETR: Detection-Guided Click Propagation for Remote Sensing Interactive Segmentation

The paper proposes ISRS-DETR, a detection-guided interactive segmentation framework that leverages strong inter-object correlations in remote sensing imagery to propagate a single user click across all instances of a class, thereby achieving state-of-the-art accuracy while significantly reducing the number of clicks required per image.

Original authors: Thanh Duc Pham, Anh Nguyen, Duong Duc Hieu, Minh-Tan Pham

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thanh Duc Pham, Anh Nguyen, Duong Duc Hieu, Minh-Tan Pham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive puzzle, but instead of finding clues on a table, you are looking at a giant, high-resolution photograph taken from a satellite. This photo shows a whole city or a vast ocean, packed with thousands of tiny details: cars, ships, buildings, and trees. In the world of computer science, this is called "remote sensing." The goal is to teach a computer to recognize and outline every single one of these objects. Usually, to teach a computer this, humans have to draw a line around every single car or building pixel by pixel. It's like trying to color a coloring book where every page has a million tiny dots; it takes forever and is incredibly boring.

To make this easier, scientists invented "interactive segmentation." Think of it as a game of "Hot and Cold." Instead of drawing the whole picture, you just click on an object, and the computer guesses where it is. If it's wrong, you click again, and it gets better. It's much faster than drawing everything by hand. However, when you try this game on satellite photos, it gets tricky. The photos are so huge and the objects (like tiny cars or distant ships) are so small and scattered that the computer gets confused. It often needs you to click dozens of times just to get one picture right, which defeats the purpose of making things easy.

This is where a new idea called ISRS-DETR comes in. The researchers behind this paper noticed something interesting about satellite photos: objects of the same type usually hang out together. If you see one school bus in a parking lot, there's a good chance there are ten more nearby, looking almost exactly the same. They realized that instead of treating every single bus as a separate mystery to solve one by one, the computer should realize, "Hey, I just found a bus! I bet all these other bus-shaped things are buses too!"

The paper proposes a smart new system that acts like a "click multiplier." Here is how it works: When you click on just one object, the system doesn't just look at that one spot. It quickly scans the whole image to find other objects that look like the one you clicked. It then creates "fake" clicks for all those other similar objects automatically. So, your single click on one car instantly helps the computer outline hundreds of other cars in the same image. It's like having a magic wand that, when you point it at one apple in a basket, instantly highlights every other apple in the basket without you having to touch them.

The researchers tested this idea on three different sets of satellite images, including ones with thousands of buildings and ships. They found that their new method was a huge improvement. In the past, to get a good outline of all the buildings in a city, a user might have had to click up to 40 times per image. With this new system, they only needed about 10 clicks to get the same result. In fact, for some images, the system reduced the number of clicks needed by nearly 28 clicks per image compared to the previous best methods. The computer didn't just get faster; it also got better at drawing the outlines, especially for tricky, thin shapes like airplane wings or distant ships.

The team showed that this "class-aware" approach—where the computer understands that objects of the same kind are related—is the key to making interactive segmentation work for massive satellite photos. While the system still relies on the computer's ability to find the objects in the first place (if the computer misses a bus, it can't highlight it), the results suggest that this method is a major step forward. It turns a tedious task of clicking one by one into a much more efficient process, making it possible to map our world from space with far less human effort.

Technical Summary: ISRS-DETR

Problem Statement
Interactive segmentation aims to reduce the labor-intensive cost of pixel-level annotation by allowing users to delineate objects with minimal clicks. While effective for natural images, applying this paradigm to Remote Sensing Images (RSI) presents unique challenges: ultra-high resolutions, small object sizes, and sparse spatial distributions often degrade segmentation quality.

Recent frameworks, such as CrossCut, have addressed resolution barriers but suffer from a critical limitation: they treat all instances of a specific class within an image as a single segmentation target. Consequently, a user interaction on one object provides no benefit to other same-class instances, despite strong semantic and visual correlations between them. This forces users to provide up to 40 clicks per image to achieve satisfactory masks, hindering practical scalability. Furthermore, existing methods fail to leverage the "macroscopic correlation" observed in remote sensing scenes, where the presence of one object strongly informs the location and nature of its same-class neighbors.

Methodology
The authors propose ISRS-DETR, a detection-guided interactive segmentation framework designed to inject object-level evidence into both training and inference. The architecture integrates a segmentation branch with a detection guide branch:

Dual-Branch Architecture:
- Segmentation Branch: Follows the CrossCut framework, utilizing a plain ViT-B encoder and a Feature Pyramid Network (FPN) to process image patches. It generates per-patch masks which are reassembled into a final prediction.
- Detection Guide Branch: Built upon the RF-DETR (Robinson et al. 2026) decoder, this branch localizes bounding boxes for co-occurring same-class objects. It reuses the fused features from the segmentation branch (which encode both image content and click semantics) to predict object proposals.
Simulated Click Propagation:
- Upon a user clicking an object, the detector identifies the target class and generates bounding boxes for all other instances of that class in the scene.
- Dynamic Top-K Box Selection: To ensure quality, the system filters proposals based on confidence thresholds and minimum size. It then employs a Dynamic Top-K Simulated Click Selection strategy. Instead of using a fixed number of clicks, the algorithm analyzes the similarity distribution between the user's click and candidate locations across all proposals. It identifies a "sharp transition" in the sorted similarity scores (using the second-order discrete derivative) to determine the optimal number ( $k^*$ ) of high-confidence candidates to retain.
- These selected candidates are converted into simulated clicks and injected into the next interaction round as positive prompts. Crucially, these simulated clicks are not counted toward the user's interaction budget, allowing a single user click to propagate supervision across an entire class.
Training:
- The model is trained end-to-end using a simulated interaction sequence. The loss function combines the segmentation loss (normalized focal loss) and the standard DETR detection loss (classification, $\ell_1$ regression, and GIoU).
- Techniques such as denoising queries and position-relation attention refinement (from Relation-DETR) are employed to stabilize training and accelerate query convergence.

Key Contributions

Identification of a Fundamental Limitation: The paper empirically characterizes that existing ISRS methods collapse same-class instances into a single target, failing to exploit inter-object relationships.
Detection-Guided Framework: ISRS-DETR introduces a novel mechanism to propagate a single user interaction to all reliable same-class instances via a detector-guided simulated click strategy.
Dynamic Top-K Selection: A strategy that adaptively retains only high-confidence proposals, converting them into simulated clicks to suppress error accumulation while maximizing propagation.
New Evaluation Protocol: The authors adopt Number of Clicks per Image (NoC-I) as a primary metric. Unlike traditional per-object metrics (NoC@70), NoC-I measures the total clicks required to segment all target instances in a scene, better reflecting the practical annotation cost in remote sensing.

Results
Experiments were conducted on three standard benchmarks: iSAID, WHU-Building, and NWPU VHR-10.

Performance: ISRS-DETR achieves state-of-the-art accuracy across all datasets, outperforming baselines like CrossCut, SimpleClick, and MFP in both mIoU and AUC (Area Under the Curve).
Efficiency: The method substantially reduces the interaction cost. On the WHU-Building dataset, ISRS-DETR requires only 10.71 clicks to achieve 70% IoU, compared to 38.69 clicks for CrossCut—a reduction of nearly 28 clicks per image. On average, the method saves 9.64 clicks per image across all settings.
Propagation Effectiveness: The detector branch successfully propagates supervision, with the "click amplification" factor varying by dataset (e.g., 8.33x on iSAID), demonstrating the ability to leverage scene-scale correlations.

Significance and Claims
The paper claims that ISRS-DETR establishes a new state-of-the-art framework for interactive segmentation in remote sensing. Its significance lies in shifting the paradigm from per-object interaction to class-aware, scene-scale interaction. By acknowledging and exploiting the strong inter-object correlations inherent in remote sensing imagery, the framework significantly reduces the human annotation burden without sacrificing segmentation quality. The authors note that while the method's performance is bounded by the quality of the underlying detector, the results suggest that detector-guided interaction is a promising direction for scalable remote sensing annotation. They also acknowledge that the advantage is most pronounced in low-click regimes and for dense scenes, while boundary refinement at very high IoU targets remains a challenge.

Technical Summary: ISRS-DETR

More like this