🤖 AI

NegROI: Click-Centric Uncertainty-Guided Refinement with Scene-Conditioned Negative Prompts for Robust Interactive 3D Segmentation

NegROI is a novel transformer-based framework for interactive 3D segmentation that enhances click efficiency and robustness across diverse datasets by combining uncertainty-guided local multi-resolution refinement with scene-conditioned negative prompts to effectively suppress false positives and sharpen object boundaries.

Original authors: Shuheng Zhang, Feng Wu

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shuheng Zhang, Feng Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to draw a perfect outline around a specific object in a messy, 3D room filled with furniture, walls, and clutter. You can't see the whole room perfectly; you only have a low-resolution, blurry map made of tiny blocks (voxels). To fix the map, you can tap on the screen with your finger (a "click") to say, "This is the object," or "This is not."

The paper introduces a new system called NegROI that acts like a super-smart assistant helping you draw that outline with very few taps. Here is how it works, broken down into simple concepts:

1. The Problem: The "Blurry Map" and the "Confusing Clutter"

Current systems have two main headaches:

The Blur: Because the map is made of big blocks, the edges of objects look fuzzy. If you tap once, the system might guess the shape, but the boundary is often too thick or inaccurate.
The Confusion: The room is full of things that look similar to your target (like a chair leg that looks like a table leg). The system gets confused and accidentally highlights the wrong things (false positives), even after you tell it what not to pick.

2. The Solution: NegROI's Two Superpowers

The authors designed NegROI to solve these problems using two main tricks:

Trick A: The "Zoom-In" Tool (Click-Centric Refinement)

Imagine you are looking at a blurry photo. Instead of trying to sharpen the entire photo (which takes forever and uses too much computer power), you just zoom in on the specific spot where you just tapped your finger.

How it works: When you click, NegROI doesn't re-calculate the whole room. It creates a tiny, high-definition "micro-map" just around your finger.
The Result: It fixes the blurry edges right where you are looking, then pastes that sharp little piece back into the big, blurry map. This makes the edges crisp without slowing down the computer.
Smart Saving: It only zooms in on the "confusing" parts of that micro-map (where the computer is unsure), saving even more energy.

Trick B: The "Negative List" (Scene-Conditioned Negative Prompts)

Imagine you are playing a game of "I Spy." Instead of just saying, "Find the red chair," you also give the computer a list of things that look like a red chair but aren't (like a red lamp or a red cushion on a sofa).

How it works: The system learns to recognize the "background noise" of the specific room you are in. It creates a set of "Negative Prompts"—essentially, a list of "Imposters" or "Distractors" that are common in that specific scene.
The Result: When the system sees something that looks like your target but is actually a confusing background object, it checks this "Negative List" and says, "No, that's just a distractor," preventing it from making a mistake.
The "Hard" Lesson: The system is specifically trained to pay extra attention to the tricky spots right next to the object's edge, where mistakes usually happen, ensuring it doesn't accidentally grab the neighbor's furniture.

3. Why It's Better

The paper tested this system on three different types of 3D environments (indoor rooms, crowded halls, and outdoor street scans).

Fewer Clicks: Because it fixes the edges so well and ignores the confusing background so effectively, you need fewer taps to get a perfect result.
Better at Guessing: Even if the system has never seen that specific type of room before (like moving from a ScanNet dataset to a KITTI dataset), it still works well because it knows how to handle the "confusing background" generally.
Less Mistakes: It makes far fewer errors where it highlights the wrong object (false positives), especially near the edges of things.

Summary Analogy

Think of old systems as a painter who tries to fix a blurry painting by repainting the whole canvas every time you point at a spot. NegROI is like a painter who has a magnifying glass and a list of "fake objects." When you point, they zoom in only on that spot to fix the details perfectly, and they use their list to ignore the things that look similar but aren't the real target. This makes the job faster, cleaner, and much more accurate.

Technical Summary: NegROI

Problem Statement
Interactive 3D segmentation aims to extract object masks from point clouds using minimal user clicks. Despite recent advances, existing methods face two primary challenges:

Boundary Under-segmentation: Single-resolution voxel representations often blur fine object boundaries, particularly when the user click budget is small.
Hard False Positives (FPs): Confusing background structures frequently trigger high-confidence false positives, even with user feedback.
These issues are exacerbated by distribution shifts across datasets (e.g., dense RGB-D reconstructions vs. sparse LiDAR scans), where fixed refinement heuristics and purely click-driven decoding fail to generalize.

Methodology
The authors propose NegROI, a novel transformer-based interactive framework that integrates click-centric multi-resolution refinement with scene-conditioned negative prompts. The architecture consists of three coupled components:

Backbone & Interaction:
- The system utilizes a sparse voxel backbone and a two-way transformer decoder (adapted from Easy3D) to encode the scene and integrate click tokens.
- Clicks are mapped to coarse voxel coordinates, and interaction tokens (foreground, background, and click-specific) are injected into the decoder.
Scene-Conditioned Negative Prompts:
- To explicitly model confusing background structures, the method introduces $K$ learnable negative prompt queries.
- These queries cross-attend to scene tokens to generate scene-conditioned negative prototypes, which are appended to the interaction tokens before decoding.
- Supervision:
  - Boundary-Aware Hard Negative Mining ( $L_{hn}$ ): The attention weights of the negative prompts are supervised to focus on high-confidence false positives located near object boundaries.
  - Diversity Regularizer ( $L_{div}$ ): A penalty on the cosine similarity between negative prototypes prevents "prompt collapse," ensuring diverse background modeling.
Click-Centric ROI Refinement:
- Instead of refining the entire scene at high resolution, the system identifies a local Region of Interest (ROI) around the current click.
- Adaptive Radius: The ROI size is determined by an adaptive radius predicted from click context and local point density.
- Selective Refinement: An uncertainty-driven mechanism prioritizes ambiguous regions within the ROI for fine-grid processing.
- Refinement Process: The ROI is re-voxelized at a finer resolution ( $v_f = v/\eta$ ), processed through a shared encoder and a dedicated ROI decoder, and the resulting fine logits are max-aggregated and fused back into the coarse logits via a residual update.

Key Contributions
The paper outlines four primary contributions:

Click-Centric Multi-Resolution Refinement: A module that performs efficient fine-grid refinement within a local ROI and fuses these predictions back to coarse logits, addressing boundary blurring without dense global decoding.
Uncertainty-Driven Selective Refinement: A mechanism that focuses computational resources on ambiguous regions within the ROI, improving both efficiency and robustness.
Scene-Conditioned Negative Prompts: A structured approach to explicit background modeling using negative prototypes learned via cross-attention, augmented by a diversity regularizer to prevent collapse.
Boundary-Aware Hard Negative Mining: A supervision strategy that directs negative-prompt attention specifically toward boundary-proximal, high-confidence false positives.

Experimental Results
The authors evaluated NegROI on three benchmark datasets: ScanNet40, S3DIS, and KITTI-360.

In-Domain Performance: On ScanNet40, NegROI consistently outperformed state-of-the-art baselines (including InterObject3D, AGILE3D, and Easy3D) across all click budgets ( $k \in \{1, 2, 3, 5, 10\}$ ), with the most significant gains observed under sparse interactions (1–3 clicks).
Out-of-Domain Generalization: When trained on ScanNet40 and tested on S3DIS and KITTI-360 without target-domain training, NegROI demonstrated superior cross-dataset robustness compared to baselines, maintaining strong performance particularly at low click counts.
Single-Click Evaluation: In a setting trained only on ScanNet20 and tested on ScanNet20/40, NegROI improved mAP and AP metrics compared to non-interactive and open-vocabulary baselines, suggesting the benefits of the proposed prompting and refinement extend to single-click scenarios.
Qualitative Analysis: Visualizations confirmed that NegROI effectively suppresses attached or visually similar distractors and recovers sharper boundaries, resulting in fewer boundary-proximal false positives compared to the Easy3D baseline.

Significance
The paper claims that NegROI advances the state-of-the-art for robust interactive 3D segmentation. By coupling click-centric local refinement with explicit background modeling via negative prompts, the method achieves improved click efficiency and reduced false positives. Crucially, it addresses the challenge of generalizing across datasets with varying densities and scales, offering a solution that is less reliant on fixed heuristics and more adaptable to distribution shifts. The authors position this work as a step toward more reliable interactive pipelines for 3D applications where dense labeling is expensive.