💻 computer science

InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding

This paper introduces InstAP, an instance-aware vision-language pre-training framework supported by the large-scale InstVL dataset, which significantly enhances both instance-level reasoning and global spatial-temporal understanding by jointly optimizing holistic scene alignment with fine-grained, grounded instance-level contrastive objectives.

Original authors: Ashutosh Kumar, Rajat Saini, Jingjing Pan, Mustafa Erdogan, Mingfang Zhang, Betty Le Dem, Norimasa Kobori, Quan Kong

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ashutosh Kumar, Rajat Saini, Jingjing Pan, Mustafa Erdogan, Mingfang Zhang, Betty Le Dem, Norimasa Kobori, Quan Kong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a busy street scene in a video. A traditional AI model is like a tourist who only sees the "big picture." If you ask, "What's happening?" it might say, "It's a busy street with cars and people." That's true, but it's vague. If you then ask, "Where is the red ball the child is throwing?" the tourist might point at the whole street, or worse, point at a dog nearby because the dog is also moving. It sees the scene, but it can't pick out the specific actors.

This paper introduces InstAP, a new way of teaching AI to be not just a tourist, but a detective.

Here is the breakdown of how they did it, using simple analogies:

1. The Problem: The "Blurry" Camera

Current AI models are trained on millions of videos and captions. But the captions usually describe the whole video (e.g., "A dog chases a ball"). The AI learns to match the whole video to that sentence.

The Flaw: It doesn't learn which part of the video is the dog and which is the ball. It's like listening to a symphony and knowing the song title, but not being able to pick out the violin solo from the rest of the orchestra.

2. The Solution: The "Detective's Notebook" (InstVL Dataset)

To fix this, the researchers (from Toyota's Woven division) created a massive new training library called InstVL.

The Old Way: They gave the AI a video and one sentence: "A dog chases a ball."
The New Way (InstVL): They gave the AI the same video, but with two types of notes:
1. The Big Picture: "A dog chases a ball." (Global Caption)
2. The Detective's Notes: "The brown dog is running," and "The red ball is bouncing." Crucially, these notes are tied to specific moving boxes around the dog and the ball over time.

Think of it like giving a student a textbook where, instead of just reading a chapter summary, they are also forced to highlight every specific character and object in the text and draw a line connecting the word "dog" to the picture of the dog.

3. The Training: "Spot the Difference" (InstAP Framework)

The AI model, InstAP, is trained using a special game called Contrastive Learning.

The Game: The AI sees a video clip and a sentence. It has to guess: "Does this sentence describe the whole clip, or just this specific moving part?"
The Twist: If the sentence says "The red ball," the AI is punished if it looks at the whole street. It must zoom in and focus only on the red ball. If it looks at the dog instead, it gets a "wrong answer" signal.
The Result: The AI learns to build a mental map where words like "red ball" are tightly locked to the specific pixels of the ball, not just the general vibe of the video.

4. The Surprise: Better at Everything

The researchers expected that teaching the AI to be a "detective" (focusing on small details) might make it worse at being a "tourist" (understanding the whole scene).

The Reality: It made the AI better at both.
The Analogy: Imagine a chef who learns to perfectly chop a single carrot. You might think they'd forget how to cook the whole stew. Instead, because they understand the carrot so well, they become a better chef for the whole stew. By understanding the specific parts (the instances), the AI understands the whole story (the global scene) more deeply.

5. Why This Matters

This technology is a big deal for the future of AI in cars and robots.

Self-Driving Cars: Instead of just seeing "a car ahead," the car needs to know, "That is a child on a bicycle wearing a red helmet." If the car confuses the child with a trash can, that's a disaster. InstAP helps the car see the specific object, not just the general scene.
Video Search: Imagine searching your home videos for "the moment my dog caught the frisbee." Current AI might show you the whole day. InstAP would find that exact second and show you just the dog and the frisbee.

Summary

InstAP is like upgrading an AI from a tourist who takes a blurry photo of a crowd, to a detective who can point a finger at exactly who is doing what, while still remembering the whole story. They did this by creating a massive "detective training manual" (InstVL) and teaching the AI to link specific words to specific moving objects in time and space.

1. Problem Statement

Current Vision-Language Pre-training (VLP) paradigms, such as CLIP and its video extensions, excel at global scene understanding (matching an entire video to a caption) but struggle with instance-level reasoning.

The Gap: Existing models rely on global-only supervision, causing them to fail at identifying and distinguishing specific objects or entities mentioned in text (e.g., distinguishing "the red ball" from "the blue ball" in a complex scene).
Limitations of Current Approaches:
- Data: Large-scale video-text datasets typically provide only high-level, holistic descriptions, lacking grounded annotations for specific objects or trajectories.
- Architecture: Prevailing objectives reward holistic alignment, offering no incentive for the model to attend to fine-grained details.
- Post-hoc Fixes: Recent attempts to add instance awareness often rely on grafting external object detectors or auxiliary heads onto pre-trained models. These approaches inherit detector errors and fail to integrate instance-level understanding into the core representation learning.

2. Methodology

The authors propose InstAP, a framework that jointly optimizes global vision-text alignment and fine-grained instance-level contrastive alignment.

A. The InstVL Dataset

To enable this training, the authors introduce InstVL, a large-scale dataset containing 2 million images and 50,000 videos.

Dual-Granularity Annotations: Each sample includes:
1. Global Captions: Holistic scene descriptions.
2. Instance Captions: Dense, grounded descriptions linked to specific spatial regions (images) or spatial-temporal trajectories (videos).
Data Curation:
- Images sourced from LAION-400M; Videos from HDVILA.
- Pipeline: Uses AutoShot for segmentation, GroundingDINO for open-vocabulary detection, and SAM2 for instance tracking.
- Generation: A large vision-language model generates free-form sentences for both the whole scene and specific tracked instances.
- Test Splits: Includes held-out zero-shot splits (from COYO) to ensure performance isn't just memorization of training distributions.

B. The InstAP Framework

The model employs a Teacher-Student architecture with a two-stage training process:

Self-Supervised Masked Video Modeling (Pre-training):
- A student video transformer ( $f_\theta$ ) is trained to reconstruct features of a frozen CLIP-ViT teacher ( $g$ ) from masked tokens.
- Mechanism: Instead of pixel reconstruction, it performs semantic feature regression on unmasked tokens. An attention-guided masking strategy hides the least informative tokens, forcing the student to learn rich spatial-temporal representations from the visible context.
Instance-Aware Global-Local Alignment:
The model is fine-tuned with a composite loss function combining global and instance-level objectives:
- Global Objectives: Standard Video-Text Contrastive (VTC), Video-Text Matching (VTM), and Masked Language Modeling (MLM) losses align the whole video with the caption.
- Instance Objectives:
  - Feature Extraction: Instance crops (or trajectory crops) are encoded and cross-attended with the full-scene features to inject global context.
  - Instance VTC Loss: Enforces alignment between specific instance crops and their corresponding text mentions. Crucially, it uses a masking mechanism ( $\alpha_{n,m}$ ) to prevent treating other instances within the same video as negative samples (avoiding false negatives due to overlapping semantics).
  - Instance VTM & MLM: Similar matching and recovery tasks applied at the instance level.
- Total Loss: $L = L_{rec} + L_{global} + L_{inst}$ , where weights allow independent tuning of global vs. instance objectives.

3. Key Contributions

InstVL Dataset: The first large-scale, general-domain resource providing free-form sentence annotations for both static regions and full video trajectories, bridging the gap between holistic and instance-level data.
InstAP Framework: A novel pre-training paradigm that embeds instance awareness directly into the core encoder via joint optimization, rather than as an auxiliary task.
Proven Generalization: Demonstrates that fine-grained instance alignment enhances, rather than degrades, holistic scene understanding, leading to state-of-the-art zero-shot performance.

4. Experimental Results

The authors evaluated InstAP against state-of-the-art models (e.g., VideoPrism, CLIP4Clip, UMT-L) on multiple benchmarks.

Instance-Level Retrieval (InstVL Benchmark):
- InstAP significantly outperforms all baselines. On the InstVL-1K (video) instance retrieval task, it achieves 60.63 T2V R@1, vastly exceeding the previous best (VideoPrism at 40.86).
- Ablation: When compared to a strong UMT-L baseline trained on the exact same InstVL corpus (using only global captions or treating all captions as global), InstAP still outperforms it significantly (e.g., 44.05 vs. 34.83 T2V R@1 on InstVL-10K img). This proves the gains come from the alignment framework, not just data volume.
Zero-Shot Generalization:
- InstAP achieves new state-of-the-art results on standard zero-shot benchmarks: MSR-VTT (41.1 R@1) and DiDeMo (54.0 R@1).
- Notably, simply fine-tuning baselines on InstVL often degrades their global performance, whereas InstAP improves it, confirming that instance-aware pre-training creates more robust representations.
Visual Grounding:
- On the InstVL-1K grounding task, InstAP significantly improves IoU@90 (from 14.44 to 25.13 on video splits), proving its ability to precisely localize textual mentions to spatial-temporal regions.
Qualitative Analysis:
- Visualizations show InstAP focuses attention on specific caption-relevant regions (e.g., a specific license plate), whereas global-only models exhibit diffuse, scene-level attention.

5. Significance

Paradigm Shift: InstAP challenges the notion that instance-level understanding must be an auxiliary task. It demonstrates that instance-awareness is a core property that should be learned during pre-training to achieve true spatial-temporal understanding.
Robustness: The work shows that forcing models to ground specific entities does not sacrifice global context; rather, it creates a dual-granularity representation that is superior for both fine-grained retrieval and holistic scene comprehension.
Foundation for Future Tasks: By solving the "diffuse attention" problem, InstAP paves the way for advanced downstream applications requiring precise object tracking, spatial-temporal question answering, and complex video reasoning.