💻 computer science

SAM3-I: Segment Anything with Instructions

This paper introduces SAM3-I, an instruction-following extension of the Segment Anything Model 3 that unifies concept-level grounding and instruction-level reasoning through a cascaded adaptation mechanism and a new large-scale dataset (HMPL-Instruct), enabling the direct interpretation of complex natural-language instructions without sacrificing the model's original concept recall capabilities.

Original authors: Jingjing Li, Yue Feng, Yuchen Guo, Jincai Huang, Wei Ji, Qi Bi, Yongri Piao, Miao Zhang, Xiaoqi Zhao, Qiang Chen, Shihao Zou, Huchuan Lu, Li Cheng

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingjing Li, Yue Feng, Yuchen Guo, Jincai Huang, Wei Ji, Qi Bi, Yongri Piao, Miao Zhang, Xiaoqi Zhao, Qiang Chen, Shihao Zou, Huchuan Lu, Li Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant named SAM (Segment Anything Model). For a long time, SAM was like a very obedient but literal-minded librarian.

SAM 1 & 2: If you pointed at a specific book and said, "Find this one," it would find it. But if you asked for "a book," it wouldn't know which one to pick.
SAM 3: This version got smarter. If you said, "Find all the soccer players," it could instantly find every single soccer player in a photo, even if there were ten of them. It was great at finding things based on concepts (categories).

The Problem:
Real life isn't just about categories. Imagine you are in a messy room and you tell your robot: "Please pick up the blue soccer ball that is rolling toward the door and hitting the cat."

The old SAM 3 couldn't handle this. It only understood "soccer ball." To make it work, developers had to build a complicated "middleman" system (an AI agent) to translate your complex sentence into a simple keyword like "blue ball," and then try to guess which one you meant. It was slow, clunky, and often got the details wrong (like picking up the red ball instead).

The Solution: SAM3-I
The authors of this paper created SAM3-I (Instruction-following). Think of this as giving the librarian a PhD in human conversation while keeping their amazing memory.

Here is how it works, using some fun analogies:

1. The "Cascaded Adapter" (The Layered Detective)

Instead of trying to learn everything at once, SAM3-I uses a two-step detective team inside its brain:

The "Simple" Detective (S-Adapter): This detective is great at spotting obvious clues. If you say, "The red car," this detective immediately grabs the color and the object.
The "Complex" Detective (C-Adapter): This detective is a genius at reading between the lines. If you say, "The car that looks like it's about to crash," the first detective might get confused. But the second detective understands the action and the context.

They work together in a cascade. The simple clues are passed down to the complex detective, who adds the deep reasoning. This way, the robot understands both "The red car" and "The car that looks like it's about to crash" equally well.

2. The "Alignment Losses" (The Quality Control Team)

When you teach a robot two different ways to think (simple vs. complex), they might start talking past each other. To fix this, the researchers added a "Quality Control Team" (mathematical rules called Losses):

The "Same Target" Rule: If you describe the same object in two different ways (e.g., "The big dog" and "The animal barking"), the robot must realize they are the same dog.
The "Concept Guardrail": If you say "The dog," the robot shouldn't accidentally pick up a cat, even if the cat is barking. It keeps the robot grounded in the right category.
The "Hard Mode" Trainer: The robot is forced to pay extra attention to the tricky parts of the image (like a ball hidden behind a chair) so it doesn't get lazy.

3. The "HMPL-Instruct" Dataset (The Training School)

You can't teach a robot to understand complex instructions without good textbooks. The authors built a massive new library called HMPL-Instruct.

Imagine a school where students practice giving instructions.
Level 1: "Find the apple." (Concept)
Level 2: "Find the shiny apple on the left." (Simple Instruction)
Level 3: "Find the fruit that someone dropped and is rolling near the dog." (Complex Reasoning)
Level 4: "Find three of the apples that are rotting." (One-to-Many reasoning)

They used humans and AI together to create hundreds of thousands of these examples, ensuring the robot learns to handle every type of request.

Why is this a big deal?

Before this, if you wanted a robot to follow complex instructions, you needed a slow, expensive chain of different AI models talking to each other. It was like hiring a translator, then a manager, then a worker.

SAM3-I is like hiring one super-employee who can listen to your complex request, figure out exactly what you mean, and do the job in one smooth motion. It keeps the speed and accuracy of the old version but adds the ability to understand human nuance, humor, and context.

In short: SAM3-I turns a robot that only understands "keywords" into a robot that understands "stories," making it ready for real-world tasks like helping in a home, driving a car, or working in a factory where instructions are never just simple words.

1. Problem Statement

The Segment Anything Model 3 (SAM3) introduced "Promptable Concept Segmentation" (PCS), allowing users to segment all instances of a concept using short noun phrases (e.g., "soccer player"). While effective for open-vocabulary concept grounding, SAM3 struggles with complex natural language instructions common in real-world scenarios. These instructions often involve:

Rich attributes and relations: (e.g., "the soccer player on the far right wearing a blue jersey").
Implicit reasoning and functions: (e.g., "the object used to hold drinks" instead of naming the object).
Multi-instance selection: (e.g., "the two white cups behind the pizza").

Current workarounds rely on external Multi-Modal Large Language Models (MLLMs) to convert complex instructions into short noun phrases, followed by iterative mask filtering. This approach has three major limitations:

Information Loss: Complex instructions are oversimplified into coarse noun phrases, losing critical details.
Computational Overhead: It requires multiple inference passes (agent loop) and external models.
Decoupled Reasoning: Linguistic reasoning is separated from visual segmentation, preventing end-to-end optimization.

2. Methodology: SAM3-I

The authors propose SAM3-I, a unified framework that extends SAM3 to directly interpret free-form natural language instructions while preserving its original concept-recall capabilities.

A. Core Architecture

SAM3-I builds upon the frozen SAM3 backbone (dual vision-language encoder and detector-based head) and introduces an Instruction-Aware Cascaded Adaptation Mechanism.

Frozen Backbone: The original SAM3 parameters remain frozen to prevent catastrophic forgetting of concept-level grounding.
Cascaded Adapters: Two lightweight adapter modules are inserted into the text encoder and detector layers:
- S-Adapter (Simple): Learns attributes, positions, and relations. It handles instructions with explicit noun phrases plus conditions (e.g., "red car").
- C-Adapter (Complex): Built upon the S-Adapter, it handles instructions lacking explicit noun phrases, requiring functional or contextual reasoning (e.g., "the vehicle used for transport").
Progressive Training: The model uses a three-stage curriculum:
1. Train S-Adapter on simple instructions.
2. Freeze S-Adapter, initialize and train C-Adapter on complex instructions.
3. Jointly fine-tune both with alignment losses.

B. Distribution Alignment Losses

To ensure the two branches (Simple and Complex) produce consistent results and remain grounded in the original concept space, four specific loss functions are introduced:

Instruction Contrastive Loss ( $L_{inst}$ ): Ensures that different phrasing of the same target (simple vs. complex) yields similar embeddings, while different targets remain distinct. It down-weights negative samples from the same concept category to stabilize training.
Parent-Rank Concept Anchoring Loss ( $L_{anchor}$ ): Prevents instruction embeddings from drifting too far from their parent concept. It enforces that an instruction embedding is closer to its own concept than to concepts of other categories.
Mask Distribution Alignment Loss ( $L_{mask}$ ): Uses KL-divergence to align the mask predictions generated by the simple and complex branches, ensuring semantic consistency.
Uncertainty-Aware Hard-Region Supervision ( $L_{hard}$ ): Identifies pixels where the two branches disagree (high uncertainty) and applies stronger supervision to these "hard regions" (e.g., occlusions or complex relations) to improve robustness.

3. Key Contributions: HMPL-Instruct Dataset

To support instruction-following learning, the authors constructed HMPL-Instruct (Hierarchical Multi-grained PACO-LVIS-Instruct), a large-scale dataset derived from PACO-LVIS.

Hierarchical Semantics: Covers three levels: Concept (NPs), Simple (Referring expressions), and Complex (Reasoning/Functional).
Multi-Granularity Targets: Supports Object-level and Part-level segmentation.
Diverse Cardinalities: Explicitly covers One-to-One, One-to-Many (selecting a subset of similar objects), and One-to-All scenarios.
Construction Pipeline: Uses a human-in-the-loop process involving automatic generation by MLLMs, agentic quality inspection, and targeted human correction to ensure high-quality grounding.
Scale: Contains 44,109 images, 849,792 instructions, and over 347k masks.

4. Experimental Results

Experiments were conducted on HMPL-Instruct and standard benchmarks (RefCOCO, Ref-ZOM).

Performance vs. SAM3 Agent: SAM3-I significantly outperforms the SAM3 + MLLM Agent baseline.
- Simple Instructions: 59.7 gIoU (SAM3-I) vs. 28.4 gIoU (Agent).
- Complex Instructions: 49.0 gIoU (SAM3-I) vs. 26.4 gIoU (Agent).
- Efficiency: SAM3-I achieves this with a single-pass inference and fewer parameters (1.1B vs. 8.8B for the agent), eliminating the need for iterative filtering.
One-to-Many Capability: SAM3-I shows superior performance in selecting specific subsets of visually similar objects, a task where the agent-based approach often fails due to oversimplification.
Concept Preservation: SAM3-I maintains the original SAM3 concept-level performance (29.5 gIoU), proving that instruction learning does not degrade the model's foundational capabilities.
Ablation Studies: Confirmed that the cascaded adapter design and the specific alignment losses (especially $L_{inst}$ and $L_{anchor}$ ) are critical for performance.

5. Significance

Unified Framework: SAM3-I is the first model in the SAM family to unify concept-level grounding and instruction-level reasoning within a single, efficient architecture.
Real-World Applicability: By handling complex, implicit, and multi-instance instructions directly, it bridges the gap between academic benchmarks and real-world applications like home robotics, autonomous driving, and AR.
Efficiency: It demonstrates that complex instruction following does not require heavy external agents or multi-step loops, making it more deployable.
Resource Availability: The release of the HMPL-Instruct dataset and the SAM3-I code provides a new standard benchmark for future research in instruction-driven segmentation.

In conclusion, SAM3-I represents a significant evolution from "promptable concept segmentation" to "promptable instruction segmentation," enabling models to understand and execute complex, human-like visual commands without sacrificing the robustness of their underlying foundation models.