Technical Summary: DRAgent – Discriminative Reasoning Agent for Referring Expression Segmentation

1. Problem Statement

Referring Expression Segmentation (RES) aims to generate pixel-level masks for objects specified by natural language expressions. While recent approaches leverage Multimodal Large Language Models (MLLMs) combined with foundation segmentation models (e.g., SAM), they face two critical limitations:

1. Localization Bias in Generative Approaches: Current methods often treat visual localization as a generative coordinate prediction problem. MLLMs serialize continuous spatial coordinates into discrete text tokens (autoregressive generation). This representation fails to preserve the continuity of spatial relationships, leading to coarse or biased localization. In complex scenes with dense objects or similar appearances, this "one-pass" generation is prone to confusion with distractors, propagating erroneous spatial prompts to the downstream segmentation module.
2. Unreliable Reasoning-Chain Supervision: Methods utilizing reasoning chains (e.g., Chain-of-Thought) for training often rely on data that is either manually annotated (costly, non-scalable) or generated by external MLLMs. The latter often produces logically inconsistent reasoning or paths that are linguistically plausible but visually ungrounded, causing models to learn unstable patterns.

2. Methodology: DRAgent

The authors propose DRAgent, an MLLM-driven framework that reformulates visual localization from coordinate generation to discriminative target selection within a detector-generated candidate space. The pipeline consists of three stages:

A. Candidate Object Detection

Instead of asking the MLLM to generate coordinates, an open-vocabulary detector (Grounding DINO) first constructs a high-recall candidate set $O = \{o_1, o_2, \dots, o_n\}$. The target category is extracted from the text query $Q$ by an LLM and fed to the detector to generate bounding boxes.

B. Two-Stage Discriminative Reasoning (DR)

The MLLM acts as a visual-semantic target discriminator rather than a coordinate generator. It selects the best candidate through a cascaded two-stage mechanism:

1. Candidate-Level Preliminary Screening:

   • Detected candidates are visualized on the image using indexed Set-of-Mark (SoM) bounding boxes.
   • The MLLM selects the top-$K$ candidates that best match the query $Q$.
   • To ensure recall, additional candidates with an Intersection over Union (IoU) exceeding a threshold $\gamma$ with the top-$K$ boxes are added to form the second-stage candidate set $O_{top}$.

2. Instance-Wise Verification:

   • Each candidate in $O_{top}$ is rendered individually with an SoM box.
   • The MLLM is prompted with a specific instruction (e.g., "Look at the red bounding box. Does it tightly enclose the object described as '{Q}'?") to perform a visual question answering task.
   • Verification Score: Instead of relying on a single generated token, the system calculates a verification score $S_i$ based on the relative logits of positive ("Yes") vs. negative ("No") answer token sets. The candidate with the highest score is selected as the target.

C. Pixel-Level Segmentation

The selected bounding box serves as a spatial prompt for a foundation segmentation model (SAM) to generate the final pixel-level mask.

D. Training Strategy: Self-Consistency-Filtered Data Pipeline

To enhance the MLLM's discriminative reasoning capability, the authors introduce a data pipeline for LoRA-based Supervised Fine-Tuning (SFT):

1. Self-Consistency Generation: The MLLM generates a structured response (reasoning chain, description, answer) for a candidate region.
2. Self-Verification (Closed-Loop): The generated description is fed back into the MLLM as a new query to verify consistency.
3. Filtering: A reasoning chain is retained for training only if the initial answer matches the ground truth AND the closed-loop verification confirms the reconstructed description. This filters out linguistically plausible but visually incorrect reasoning.

3. Key Contributions

1. Reformulation of Visual Localization: DRAgent shifts the MLLM's role from generating coordinates to acting as a discriminative selector over a detector-generated candidate space, mitigating localization bias.
2. Two-Stage DR Mechanism: A novel inference strategy combining candidate screening with instance-wise visual verification to improve selection stability in complex scenes.
3. Self-Consistency-Filtered Data Pipeline: A method to construct reliable reasoning-chain data for SFT by enforcing closed-loop consistency between reasoning, description, and visual evidence, addressing the issue of unstable training signals.

4. Experimental Results

Experiments were conducted on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks using cumulative Intersection over Union (cIoU) as the metric.

• Performance: DRAgent achieved competitive results, notably attaining the best performance on RefCOCOg (76.78% on val(U) and 77.54% on test(U)), outperforming both implicit MLLM-based methods and explicit visual localization generators.
• Ablation Studies:
  • The two-stage DR mechanism significantly improved performance over a single-stage baseline (from 66.84% to 75.42% average cIoU on RefCOCOg).
  • Self-consistency-filtered reasoning data yielded better results than raw CoT data or SFT without CoT, confirming the value of reliable supervision.
  • Parameter analysis indicated optimal performance with $K=4$ (top candidates) and $\gamma=60\%$ (IoU threshold).

5. Significance and Claims

The paper claims that the primary limitation of recent MLLM-based RES methods lies in treating localization as a generative coordinate problem. By introducing a discriminative target selector, DRAgent leverages the MLLM's reasoning capabilities for verification rather than generation.

The authors argue that this approach is particularly beneficial for datasets with long, compositional expressions (like RefCOCOg), where fine-grained visual-semantic comparison among candidates is crucial. Furthermore, the proposed data pipeline offers a scalable solution for training discriminative reasoning without relying on costly manual annotation or unreliable external generation, providing a more robust signal for visual-semantic alignment. The framework demonstrates that decoupling candidate generation from target discrimination can effectively reduce alignment errors and improve segmentation accuracy.