Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-Thought

Technical Summary: SegWorld – Towards Proactive Affordance Reasoning via Visual Chain-of-Thought

1. Problem Definition: The Gap Between Intent and Target

Current language-driven segmentation models, which integrate Large Language Models (LLMs) with mask decoders, operate primarily under a query-conditioned paradigm. In this setting, instructions are "target-referential," explicitly naming, describing, or constraining the specific region to be segmented (e.g., "segment the red cup"). While recent works have introduced visual chain-of-thought (CoT) to handle implicit reasoning, the visual scene remains irrelevant until a specific query is posed.

However, in real-world embodied interaction, human instructions are often intent-level and pragmatic. They describe a desired outcome without specifying the target region (e.g., "I want to drink water" in a scene containing a mug, a kettle, and a plate). Fulfilling such an instruction requires the agent to:

1. Infer the relevant object (the mug).
2. Determine the necessary action (grasping).
3. Identify the specific affordance-bearing part that enables the action (the handle), rather than the whole object.

Existing models struggle with this "intent-to-part" gap because they lack the proactive reasoning to bridge high-level goals with low-level physical interaction sites.

2. Methodology: The SegWorld Framework

The authors propose SegWorld, a framework that replaces direct instruction-to-mask mapping with a proactive, multi-level reasoning process. The core innovation is a two-pass, three-stage architecture that formalizes segmentation as probabilistic inference where proactive observation serves as a prior.

A. Probabilistic Formulation
SegWorld approximates the direct estimation $P_\theta(M | I, q)$ by marginalizing over an instruction-independent scene context $c$:
$$P_\theta(M | I, q) \approx \mathbb{E}_{c \sim P_\theta(c|I)} [P_\theta(M | I, q, c)]$$
Here, $I$ is the image, $q$ is the instruction, and $c$ is a linguistic scene context generated before the instruction is known.

B. Architecture and Stages
The model utilizes a shared Vision-Language backbone (fine-tuned with LoRA) and a SAM2 mask decoder. The process unfolds in two forward passes:

1. Stage 0: Proactive Observation (Instruction-Free)

   • Input: Image $I$ only.
   • Output: A linguistic scene context $c = (s, o_{list}, r, e)$.
   • Process: The model generates four levels of reasoning:
     • Scene ($s$): General description of the setting.
     • Objects ($o_{list}$): List of salient visible objects.
     • Relations ($r$): Spatial and functional relationships.
     • Events ($e$): Plausible interactions the scene affords (e.g., "pouring," "sipping").
   • Goal: To establish a "linguistic scene context" that acts as an informative prior, independent of the specific user query.

2. Stage 1: Instruction-Conditioned Reasoning

   • Input: Image $I$, Instruction $q$, and Context $c$.
   • Process: The model traverses an eight-level visual chain-of-thought organized into three sub-stages:
     • Intent Resolution: Identifies the relevant object ($o$) and the action ($a$) required to fulfill the intent.
     • Affordance Grounding: Identifies the specific object part ($p$) that supports the action and names its affordance ($f$).
     • Segmentation: The chain terminates in a special [SEG] token. Its hidden state is projected into a prompt embedding for the SAM2 mask decoder, which fuses this with image features to predict the binary mask $M$.

C. Training Strategy

• Objective: Joint training with mask loss (binary cross-entropy + Dice) and language modeling losses for both the observation chain and the reasoning chain.
• Supervision: The observation levels are synthesized offline using vision-language models to ensure they are instruction-agnostic. The reasoning levels use ground-truth annotations from the new benchmark.
• Scheduled Sampling: To bridge the train-test gap, the model is trained to condition on synthesized contexts initially, gradually transitioning to self-generated contexts.

3. Key Contributions

1. Intent-Level Segmentation Setting: The paper identifies and defines a new segmentation paradigm where instructions specify a goal rather than a target, requiring the recovery of an affordance-bearing object part through latent reasoning steps ($q \to o \to a \to (p, f) \to M^\star$).
2. SegWorld Framework: A proactive segmentation model that performs an eight-level visual CoT. It is formally grounded in probabilistic inference, using proactive observation to supply a scene context that sharpens the posterior distribution for intent-level queries.
3. Intent2Part Benchmark: A new dataset constructed based on InstructPart, adding intent-level instructions to 1,800 training and 600 test samples. It includes high-level goals (e.g., "I want to relax") paired with the corresponding affordance-bearing parts (e.g., the handle of a wine glass).

4. Experimental Results

Experiments were conducted on the Intent2Part benchmark, comparing SegWorld against baselines like LISA, Sa2VA, Affordance-R1, and SAM3-I.

• Target-Referential Performance: SegWorld matches or exceeds strong baselines on standard referring and reasoning instructions, demonstrating that the proactive reasoning mechanism does not degrade performance on conventional tasks.
• Intent-Level Performance:
  • Zero-Shot Generalization: SegWorld v1 (trained only on target-referential instructions) achieves a 0.514 mIoU on the intent-level test_clean split, outperforming the strongest baseline (Affordance-R1) by 3.4 points. This suggests the model learns a reusable intent-to-part inference pattern.
  • Supervised Improvement: SegWorld v2 (trained with intent-level instructions) further improves to 0.612 mIoU on test_clean, significantly outperforming all baselines.
  • Token Emission: Unlike off-the-shelf models (e.g., Sa2VA) that often fail to emit a segmentation token for intent queries (responding conversationally instead), SegWorld consistently commits to a mask.
• Ablation Studies: Removing the proactive context or the Stage-1 reasoning chain causes significant performance drops, confirming that explicit reasoning about objects, actions, and affordances is critical for intent grounding.

5. Significance and Claims

The paper claims that SegWorld represents a step toward segmentation systems that reason about what a scene affords rather than merely following explicit instructions. By shifting from a reactive, query-conditioned paradigm to a proactive one, the model can handle the underspecified nature of human intent in embodied interaction.

The authors emphasize that the visual chain-of-thought makes the prediction process interpretable and traceable, exposing where a prediction originates (e.g., the reasoning chain leading to the "handle" rather than the whole "bottle"). They acknowledge limitations, noting that the current work is restricted to static images and single-step intents, and that the explicit language-based reasoning makes the system sensitive to errors in intermediate textual predictions. However, the results suggest that proactive observation is most valuable precisely where traditional query-conditioned models are weakest: in recovering actionable parts from high-level goals.