Technical Summary: Vision Harnessing Agent for Open Ad-hoc Segmentation (VASA)

Problem Definition: Open Ad-hoc Segmentation

Current segmentation foundation models (e.g., SAM, SAM3) excel at retrieving known, visually coherent concepts (e.g., "cat," "car") because these concepts are well-represented in training data. However, segmentation remains difficult for open ad-hoc concepts: arbitrary, task-contingent visual definitions constructed on the fly that involve parts, relations, exclusions, and collections (e.g., "the cat's right paw and the stick she is reaching for," or "the cat's head without the ears and eyes").

Existing approaches, including reasoning-based segmentation (e.g., LISA, Seg-Zero) and agentic methods (e.g., SAM3 Agent), typically rely on refining text prompts or predicting intermediate tokens. These methods often fail in open ad-hoc settings because:

1. Lack of Visual Persistence: They treat each prompt revision as a fresh retrieval attempt, discarding intermediate visual evidence.
2. Shallow Reasoning: They search over text queries rather than constructing a visual solution through compositional operations (e.g., "head minus ears minus eyes").
3. Inability to Handle Exclusion/Composition: They struggle to combine parts or exclude specific regions based on detailed long-form instructions.

The core challenge is that the required visual grounding often does not exist as a single learned mask and must be constructed from image evidence during inference.

Methodology: The VASA Framework

The authors propose VASA (Vision-guided Ad-hoc Segmentation Agent), a training-free framework that functions as a "vision harnessing agent." Instead of training a new segmenter, VASA orchestrates a Vision-Language Model (VLM) agent and a segmentation foundation model (SAM3) through a structured, persistent visual workflow.

Core Components

1. Persistent Working Mask: Unlike prompt-refinement agents that restart with every text change, VASA maintains a persistent working mask throughout the interaction. This mask records visual progress, allowing the agent to add, remove, or replace regions based on previous steps.
2. Visual Harness Workflow: The framework couples a VLM agent with SAM3 via a harness that manages:
   • State Management: Tracks the input image, query, current strategy, working mask, and action history.
   • Tool Invocation: Uses structured tools: segment_phrase (invokes SAM3 with a text prompt), examine_each_mask (presents candidates as visual overlays for inspection), and update_working_mask (applies Boolean operations: ADD, REMOVE, REPLACE).
   • Long-Horizon Planning: The VLM analyzes the query structure to select strategies (e.g., "over-segment and remove," "under-segment and add") rather than predicting a final mask directly.
   • Visual Scrutiny & Error Recovery: The agent inspects intermediate masks against the query's inclusion/exclusion constraints. If progress stalls or errors occur, the agent can retry with different prompts or reinitialize local steps while preserving the global history.

Inference Process

Given an image and a query, VASA:

1. Selects an initial segmentation strategy.
2. Iteratively invokes SAM3 to generate candidate masks based on the current strategy.
3. Inspects candidates visually against the query and the current working mask.
4. Updates the working mask using Boolean operations (ADD/REMOVE/REPLACE) to progressively construct the target.
5. Verifies correctness and terminates when the mask satisfies the query, progress stalls, or the interaction budget is exhausted.

Key Contributions

1. Open Ad-hoc Segmentation: The paper defines and formalizes a new segmentation setting for open-ended, task-contingent visual concepts that require construction rather than retrieval.
2. VASA Agent: The first vision harnessing agent for this setting, featuring persistent visual state, long-horizon planning, and iterative mask editing without task-specific training.
3. PARS Benchmark: A new benchmark (PartImageNet Ad-hoc Referring Segmentation) derived from PartImageNet. It converts part-level labels into open ad-hoc concepts using detailed, long-form definition queries that specify inclusion, exclusion, and structural context.
4. Vision Harness Engineering: The work demonstrates that programming agents with task knowledge, visual routines, working memory, and failure-aware workflows yields significant gains over simply wrapping foundation models as tools.

Experimental Results

The authors evaluated VASA on PARS and the standard multi-granularity referring segmentation benchmark, RefCOCOm.

• Performance on PARS: VASA significantly outperforms open-vocabulary, reasoning-based, and agentic baselines.
  • It surpasses SAM3 Agent by 13.5%–25.3% in gIoU/cIoU on ad-hoc concepts.
  • It reduces cross-concept confusion (xIoU) by 25.3% compared to SAM3 Agent.
  • Notably, VASA achieves performance comparable to VLPart, a fully supervised method trained specifically on PartImageNet, despite being training-free.
• Performance on RefCOCOm: VASA improves over SAM3 Agent by 4.8%–8.8% in gIoU and outperforms other agentic baselines by up to 20%, demonstrating generalization to fine-grained referring segmentation.
• Impact of Long-Form Queries: Detailed queries improve VASA's performance (+5.9% gIoU) but degrade SAM3 Agent's performance (-22.1% gIoU), highlighting VASA's ability to use detailed descriptions as step-by-step construction guidance rather than simple retrieval prompts.
• Reasoning Steps: Analysis shows a positive correlation between the number of additional reasoning steps taken by VASA (compared to SAM3 Agent) and performance gains, validating the utility of long-horizon visual construction.

Significance and Claims

The paper claims that VASA validates the concept of agentic visual construction for open ad-hoc segmentation. The primary significance lies in shifting the paradigm from "retrieving established concepts" to "constructing arbitrary concepts" through persistent visual state and iterative refinement.

The authors position this work as a path forward for AI agents beyond merely wrapping foundation models as tools. They argue that effective agents require:

• Programming with task knowledge.
• VLM behavior tailored for visual reasoning.
• Visual routines and working memory (persistent masks).
• Failure-aware workflows.

The results suggest that the bottleneck in open ad-hoc segmentation is not the capability of the underlying segmentation foundation model, but rather how it is harnessed to perform compositional, exclusion-based, and long-horizon visual reasoning.