💻 computer science

Segmenting, Fast and Slow: Real-Time Open-Vocabulary Video Instance Segmentation with Dual-Path Processing

This paper introduces SegFS, a dual-stream fast-slow framework for real-time open-vocabulary video instance segmentation that achieves up to 14x lower latency than mobile-oriented models by decoupling multimodal semantic understanding on sparse keyframes from efficient feature-space conditioning for dense mask prediction in subsequent frames.

Original authors: Luca Barsellotti, Martin Sundermeyer, Mattia Segu, Nikita Araslanov, Muhammad Ferjad Naeem, Marcella Cornia, Yongqin Xian, Maxim Berman

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Luca Barsellotti, Martin Sundermeyer, Mattia Segu, Nikita Araslanov, Muhammad Ferjad Naeem, Marcella Cornia, Yongqin Xian, Maxim Berman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to film a busy street scene with a smartphone, and you want your phone to instantly draw a perfect outline around every moving car, dog, and person, even if you ask it to find things it has never seen before (like "a purple scooter"). This is called Open-Vocabulary Video Instance Segmentation.

The problem is that doing this with high accuracy usually requires a supercomputer. Trying to run such a heavy brain on a mobile phone makes the video lag, stutter, or freeze.

The paper introduces a new system called SegFS (Segmenting, Fast and Slow) that solves this by using a "Fast and Slow" team approach. Here is how it works, using simple analogies:

1. The Problem: The "Heavy Brain" Bottleneck

Current high-tech video systems work like a master chef who tastes every single ingredient in a soup before serving it.

The Chef (The Slow Path): This is the heavy, accurate part of the AI. It looks at the video, reads your text prompt (e.g., "find the dog"), and carefully figures out exactly what the dog looks like and where it is.
The Bottleneck: This chef is incredibly slow. If you ask them to taste every single frame of a video (30 times a second), the phone overheats and the video stops.

2. The Solution: The "Fast and Slow" Team

Instead of asking the Master Chef to taste every single frame, SegFS splits the work into two roles:

The Slow Path: The Master Chef (Keyframes)

What it does: This heavy-duty AI only looks at one frame every few seconds (like a "keyframe").
The Job: It does the hard work. It identifies the objects, reads the text prompts, and creates a detailed "ID card" (an embedding) for every object it finds. It says, "Okay, at this exact moment, here is a dog, and here is a cat."
Analogy: Think of this as taking a high-quality photo of the scene and writing a detailed description of everyone in it.

The Fast Path: The Quick Sketch Artist (Inter-frames)

What it does: This lightweight AI looks at every single frame in between the keyframes.
The Job: It doesn't try to "think" or "read" the text again. Instead, it takes the "ID cards" created by the Master Chef and uses them as a guide. It looks at the raw video pixels and asks, "Based on the Chef's notes, where is that dog right now?"
The Trick: The Fast Path is very clever. It knows that the basic "skeleton" of the image (the shapes and edges) is already there in the video data. It just needs to "paint" the outlines based on the Chef's instructions.
Analogy: Imagine the Master Chef draws a map of where the dog is. The Sketch Artist then just quickly traces the dog's movement on that map for the next few seconds. The Sketch Artist is so fast they can draw 14 times faster than the Master Chef could ever taste the soup.

3. How They Talk to Each Other

The paper describes a special "injection" process.

When the Master Chef (Slow Path) finds a dog, it doesn't just send a picture. It sends a digital fingerprint of the dog.
The Sketch Artist (Fast Path) takes this fingerprint and "injects" it directly into the raw video data.
This allows the Sketch Artist to instantly know, "Ah, this blurry shape moving left is the dog the Chef identified." It doesn't need to re-identify the dog from scratch; it just tracks the fingerprint.

4. The Results: Speed Without Losing Quality

The paper tested this on a Samsung Galaxy S25 Ultra (a very powerful phone).

Speed: The new system is 14 times faster than previous mobile-friendly models. It can process video at 30 frames per second, which is the standard for smooth, real-time video.
Accuracy: Despite being so fast, it is almost as accurate as the slow, heavy models. The "Sketch Artist" makes very few mistakes because it is guided by the "Master Chef's" detailed notes.
Efficiency: It saves a massive amount of battery and computing power because the heavy "tasting" (complex calculations) only happens rarely, while the "sketching" (light calculations) happens constantly.

Summary

Think of SegFS as a conductor and an orchestra.

The Conductor (Slow Path) stands on the podium once every few seconds, looks at the sheet music, and tells the orchestra exactly what to play.
The Orchestra (Fast Path) plays the music continuously, following the conductor's last instruction but adjusting instantly to the tempo of the moment.

By separating the "thinking" (slow) from the "doing" (fast), the system achieves real-time video understanding on a phone without needing a supercomputer.

Technical Summary: SegFS – Real-Time Open-Vocabulary Video Instance Segmentation with Dual-Path Processing

1. Problem Statement

Open-Vocabulary Video Instance Segmentation (OV-VIS) aims to identify, track, and segment object instances across video frames based on arbitrary text prompts. While recent object-centric models (e.g., GLEE, MOBIUS) have achieved high accuracy, they suffer from severe computational overhead, particularly in the feature enhancer and object decoder components. These components dominate inference costs, making real-time, on-device deployment on mobile hardware (e.g., smartphones) infeasible. Existing attempts to optimize these models for mobile devices still fail to meet high frame-rate requirements (e.g., 30 FPS) without significant accuracy degradation. The core challenge is decoupling the computationally expensive multimodal semantic understanding from the dense, frame-by-frame mask prediction required for temporal propagation.

2. Methodology: The SegFS Framework

The authors propose SegFS, a dual-stream "Fast and Slow" framework designed to amortize computational costs by separating processing into two distinct paths operating at different temporal frequencies.

The Dual-Path Architecture

The Slow Path (Keyframes): Operates only on sparsely sampled keyframes (e.g., every $T$ frames). It utilizes a standard, heavy object-centric OV-VIS model (such as MOBIUS, GLEE, or TROY-VIS) to perform full multimodal fusion and object decoding. This path extracts high-quality, text-aligned object embeddings (instance representations).
The Fast Path (Inter-frames): Operates on every intermediate frame between keyframes. It bypasses the expensive feature enhancer and object decoder entirely. Instead, it relies on a lightweight Fast Feature Aggregator that conditions the visual backbone's feature maps using the object embeddings propagated from the slow path.

Core Mechanisms

Projection to Fast Feature Space: Object embeddings decoded by the slow path are projected into the backbone's feature space. A feed-forward network (FFN) selects the top- $K$ embeddings based on text similarity and appends a learnable background token.
Object Guidance Module: On the lowest-resolution backbone feature map ( $P_5$ $P_{5}$ ), the selected object embeddings are spatially injected. This is achieved via:
- Object Injection: Computing cosine similarity between spatial cells of $P_5$ and object tokens to create an object-aware feature map.
- Gated Fusion: A gating mechanism blends the injected object semantics with the original backbone features, preserving precise visual cues while incorporating instance semantics.
Progressive Upsampling: The enriched $P_5$ map is progressively upsampled and fused with higher-resolution backbone features ( $P_4, P_3, P_2$ ) using lightweight Depthwise Separable Convolution blocks (DSConvGN).
Mask Prediction: The final high-resolution feature map is convolved with the object embeddings (acting as kernels) to generate mask logits.
Temporal Association: The framework employs the standard "tracking-by-matching" paradigm (MinVIS), associating object queries across frames via cosine similarity of their embeddings, avoiding explicit optical flow estimation.

Training Strategy

SegFS is trained on image-based instance segmentation datasets (COCO, LVIS, RefCOCO, etc.) and treated as video datasets by ignoring temporal continuity during training.

Hybrid Matching: During training, the slow network processes the image to generate accurate category logits and bounding boxes. These are used to establish the optimal bipartite matching (Hungarian algorithm) with ground truth.
Supervision: The fast network is supervised using the mask predictions derived from this matching, utilizing Mask and DICE losses. This allows the fast network to learn to reproduce the slow network's segmentation quality without running the slow network's heavy components.

3. Key Contributions

Dual-Path Paradigm Extension: The authors extend the keyframe-based approach to OV-VIS, demonstrating that backbone features inherently contain sufficient spatial detail for localization. Consequently, the heavy feature enhancer can be restricted to sparse keyframes without significant accuracy loss.
Novel Fast Feature Aggregator: A lightweight module that injects instance embeddings into the backbone's feature pyramid, enabling accurate mask generation with minimal latency.
Efficiency-Accuracy Trade-off: The framework successfully decouples semantic understanding from dense prediction, achieving real-time speeds while maintaining competitive segmentation performance.

4. Experimental Results

The authors evaluated SegFS on standard OV-VIS benchmarks: YouTubeVIS19, OVIS, BURST, and LV-VIS.

Performance vs. Baselines:
- SegFS achieves performance comparable to the "Reuse Objects" baseline (which runs the full slow network but skips the decoder) and significantly outperforms optical flow-based propagation methods (RAFT, LiteFlowNet2) and the dual-network competitor MPVSS.
- On the MOBIUS-Mini-M backbone, SegFS achieves an AP of 41.6 on YouTubeVIS19, compared to 42.1 for the full slow network, while drastically reducing latency.
Efficiency and Latency:
- Speedup: SegFS achieves up to a 14× reduction in latency compared to the mobile-oriented MOBIUS model.
- Real-Time Capability: On a Samsung Galaxy S25 Ultra, SegFS crosses the 30 FPS threshold (achieving ~38 FPS amortized FPS on a 6-frame span), whereas baseline models struggle to exceed 10–16 FPS.
- Hardware: The method was tested on various mobile chips (Snapdragon 8 Elite, Gen 5, X2, XR2) and GPUs (A100, T4), consistently showing superior edge latency.
Ablation Studies:
- Propagation Interval ( $T$ ): SegFS maintains robust performance as the interval between keyframes increases (up to $T=10$ ), outperforming mask-warping methods which degrade rapidly.
- Component Impact: The "Object Injection" and "Attention-based Refinement" modules were identified as the most critical components for bridging the performance gap between the fast path and the full slow network.

5. Significance and Claims

The paper claims that SegFS establishes a new standard for real-time, on-device OV-VIS. By shifting the computational burden of instance propagation from object decoding to feature-space conditioning, the framework effectively bypasses the most expensive components of current state-of-the-art models.

The authors emphasize that their approach:

Preserves Zero-Shot Capabilities: It retains the open-vocabulary nature of the underlying slow models.
Enables Mobile Deployment: It is the first approach to consistently achieve >30 FPS on modern mobile hardware for this task without sacrificing mask quality.
Decouples Semantics from Geometry: It proves that high-level semantic understanding need not be recomputed for every frame, as long as the instance embeddings are correctly projected into the feature space of the visual backbone.

The work concludes that for OV-VIS, the heavy feature enhancer is redundant for inter-frames, and a lightweight fusion of backbone features with projected embeddings is sufficient for high-quality, real-time segmentation.