GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentation

Technical Summary: GenSeg-R1

GenSeg-R1 is a novel framework designed for fine-grained referring image segmentation, a task where a system must identify and mask a specific object based on a complex natural language query. The paper addresses the limitations of current vision-language models (VLMs) in spatial reasoning and their tendency to "hallucinate" masks when presented with negative or non-matching queries.

1. The Problem: Reliability and Reasoning in Grounding

Existing models for referring segmentation typically follow a "grounding-then-segmenting" pipeline. However, they suffer from two major flaws:

• Lack of Robustness to Negative Queries: Many models fail to respect "no-target" prompts. For example, if asked to find a "person in a blue shirt" in an image where everyone is wearing red, current models often incorrectly localize a person anyway.
• Sub-optimal Prompting: Models often optimize for bounding box overlap (IoU) rather than the actual quality of the final mask. Since the segmentation is performed by a downstream model (like SAM 2), a box that is mathematically close to the ground truth might still be a poor prompt for the segmenter.

2. Methodology: The "Reason-then-Segment" Pipeline

The authors propose a decoupled architecture consisting of two main stages:

• Stage 1: Reasoning & Grounding (Trainable VLM): They finetune Qwen3-VL (4B and 8B parameters) using Group Relative Policy Optimization (GRPO).
  • Output Format: Instead of just a box, the model outputs a structured JSON containing a bounding box plus two interior keypoints. These keypoints provide richer spatial cues to the segmenter.
  • Emergent Reasoning: The model is trained to use <think> tags for reasoning and <answer> tags for spatial prompts. Notably, no supervised "chain-of-thought" data is used; the reasoning capability emerges through reinforcement learning.
• Stage 2: Promptable Segmentation (Frozen SAM 2): A frozen SAM 2 (Hiera-Large) takes the box and keypoints from Stage 1 to generate the final high-quality binary mask.

Reinforcement Learning via GRPO:
The core innovation is the reward design. The authors introduce two variants:

1. Distance-based Reward (GenSeg-R1-4B/8B): A fast, lightweight reward based on bounding box and point distance.
2. SAM 2-in-the-loop Reward (GenSeg-R1-G): An "outcome-based" reward where the actual SAM 2 mask is generated during training. The reward is calculated based on the final mask IoU, directly optimizing the VLM to produce prompts that the segmenter can actually use. This variant also includes a high-weight reward for correctly identifying "no-target" scenarios.

3. Key Contributions

• SAM 2-in-the-loop Training: Closing the loop between the VLM's textual/spatial output and the final segmentation quality.
• Negative Prompt Handling: Introducing a variant (GenSeg-R1-G) specifically trained on GRefCOCO to handle queries where no target exists, achieving high accuracy (82.40%) with a very low false-negative rate (0.5%).
• Improved Backbone: Leveraging the superior spatial reasoning of the Qwen3-VL architecture.
• Efficient Training: Using GRPO to avoid the need for expensive supervised reasoning-chain annotations.

4. Results

The model demonstrates state-of-the-art (SOTA) performance across multiple benchmarks:

• RefCOCOg (Validation): The GenSeg-R1-8B model achieved 0.7127 cIoU, significantly outperforming the Qwen3-VL Instruct baseline (+15.3 points) and surpassing previous RL-based models like Seg-Zero-7B and Seg-R1-7B.
• GRefCOCO (Robustness): The GenSeg-R1-G variant achieved 82.40% accuracy on negative prompts, whereas previous models (Seg-R1 and Seg-Zero) had 0% accuracy (they hallucinated masks for every negative query).
• ReasonSeg (Complex Reasoning): The 4B model reached 68.40% mIoU, proving that the model can handle compositional and implicit reasoning (e.g., "the object used to cut paper").
• Ablation: Results confirmed that adding two interior keypoints provides monotonic improvements in mask quality over using a bounding box alone.

5. Significance

GenSeg-R1 represents a shift from optimizing for "proxy" metrics (like box coordinates) to optimizing for end-to-end task success (mask quality). By integrating the segmenter into the reinforcement learning loop, the authors have created a more reliable and "intelligent" grounding agent capable of both complex reasoning and knowing when not to segment, which is critical for real-world applications in robotics and interactive AI.