INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning

1. Problem Statement

While Large Multimodal Models (LMMs) have achieved significant success in holistic image and video understanding through instruction tuning, they struggle with instance-level understanding. Current models often fail to:

• Comprehend specific elements (instances) users are interested in.
• Perform fine-grained alignment between text instructions and specific visual objects.
• Track temporal dynamics of individual objects in videos (spatiotemporal understanding).

Existing benchmarks and datasets primarily focus on global or scene-level descriptions, lacking the fine-grained annotations necessary to train models to reason about individual objects, their attributes, and their interactions over time.

2. Methodology

The authors propose Inst-IT, a comprehensive framework consisting of a data annotation pipeline, a large-scale dataset, a benchmark, and a continuous training recipe.

A. Instance-Centric Annotation Pipeline

To address the lack of fine-grained data, the authors developed an automated pipeline leveraging GPT-4o and Set-of-Marks (SoM) visual prompting:

• Visual Prompting: Instead of processing raw images/videos, the pipeline overlays numerical IDs (SoM) on specific instances. This explicitly guides GPT-4o to focus on individual objects, reducing hallucinations and improving grounding.
• Multi-Level Annotation:
  • Frame-Level: Generates instance-level captions (describing specific objects), image-level captions (holistic scene), and temporal difference descriptions (changes between consecutive frames).
  • Video-Level: Aggregates frame-level data into a comprehensive video summary.
  • Open-Ended QA: Generates diverse question-answer pairs focusing on instance attributes, relationships, and temporal changes.

B. Inst-IT Dataset

Using the pipeline above, the authors constructed Inst-IT Dataset, the first instruction-tuning dataset grounded on individual instances for both images and videos.

• Scale: 51k images and 21k videos.
• Content: Includes 207k image-level captions, 135k temporal dynamics descriptions, 21k video-level captions, and 335k open-ended QA pairs.
• Sources: Derived from high-quality instance segmentation (e.g., SA-1B, OVIS) and tracking datasets (e.g., BenSMOT, VidOR).
• Quality Control: Rigorous manual verification and diversity filtering ensure high data quality (average human rating of 4.49/5).

C. Inst-IT Bench

A new evaluation benchmark designed to diagnose instance-level understanding capabilities.

• Composition: 338 images and 206 videos, each with manually verified open-ended and multiple-choice QA pairs.
• Features: Supports both open-ended generation and multiple-choice evaluation, covering both static images and dynamic video scenarios.

D. Continuous Instruction Tuning Recipe

The authors propose a training strategy to integrate Inst-IT data with general instruction-tuning data (LLaVA-NeXT-DATA):

• Visual Prompt Integration: All training inputs are augmented with SoM visual prompts.
• Continuous SFT: A continuous supervised fine-tuning stage is applied to existing models.
• Architecture Optimization: To prevent distribution shifts caused by visual prompts, the first 12 layers of the vision encoder are frozen during the continuous tuning stage, preserving low-level features while adapting high-level reasoning.

3. Key Contributions

1. Inst-IT Dataset: The first large-scale, instance-grounded instruction-tuning dataset covering both images and videos, featuring explicit visual prompts and multi-granularity annotations.
2. Inst-IT Bench: A human-verified benchmark specifically designed to evaluate the instance-level understanding of LMMs in both static and dynamic contexts.
3. Training Paradigm: A continuous instruction-tuning recipe that effectively combines instance-specific data with general data, enhancing instance understanding without sacrificing general capabilities.

4. Experimental Results

The authors evaluated their method (Inst-IT) on various benchmarks using LLaVA-NeXT as a baseline.

• Performance on Inst-IT Bench:
  • The Inst-IT enhanced models achieved a ~20% average improvement over baselines on the Inst-IT Bench.
  • Outperformed other state-of-the-art models (e.g., LLaVA-OV, Qwen2-VL) in instance-specific tasks.
• Generalization to Other Benchmarks:
  • Instance Understanding: Significant gains on ViP-Bench and RefCOCOg, proving the model learned generalizable instance reasoning rather than overfitting.
  • Generic Image Benchmarks: Improved performance on AI2D (+4.4%) and ChartQA (+13.5%), indicating better grounding and reasoning.
  • Generic Video Benchmarks: Substantial improvements on Egoschema (+7.8%) and NExT-QA (+11.8%), demonstrating enhanced spatiotemporal understanding.
• Ablation Studies:
  • Confirmed that combining image and video data yields the most balanced performance.
  • Showed that freezing the lower layers of the vision encoder is crucial for maintaining performance on general image tasks while learning instance-specific video dynamics.

5. Significance

• Bridging the Gap: Inst-IT addresses a critical gap in LMMs by shifting focus from holistic scene understanding to precise, instance-level reasoning, which is essential for real-world applications like robotics, autonomous driving, and interactive assistants.
• Scalability: The automated annotation pipeline using SoM and GPT-4o provides a scalable method for generating high-quality, fine-grained data for future research.
• Dual Benefit: The study demonstrates that improving instance-level understanding does not come at the cost of general capabilities; rather, it strengthens the model's overall comprehension of both images and videos.
• Community Resource: The release of Inst-IT Dataset and Bench provides the community with the necessary tools to further advance multimodal instance understanding.