💻 computer science

Weakly-Supervised Referring Video Object Segmentation through Text Supervision

This paper introduces WSRVOS, a novel weakly-supervised referring video object segmentation method that eliminates the need for expensive pixel-level or even box annotations by training solely on text expressions through contrastive expression augmentation, bi-directional feature interaction, and temporal ranking constraints to generate high-quality pseudo-masks.

Original authors: Miaojing Shi, Jun Huang, Zijie Yue, Hanli Wang

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Miaojing Shi, Jun Huang, Zijie Yue, Hanli Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a home video of a chaotic birthday party. You want to find the specific clip where "the little girl in the red dress is blowing out the candles."

In the world of computer vision, this task is called Referring Video Object Segmentation (RVOS). The computer needs to draw a perfect outline around that specific girl in every single frame of the video.

The Problem: The Expensive "Art Class"

Traditionally, teaching a computer to do this is like hiring an army of art students. You have to sit them down and have them manually trace the outline of the girl in every single frame of the video. This is incredibly expensive, slow, and boring.

Some newer methods tried to make it easier by just asking the students to draw a box around the girl or put a dot on her nose. But even that is still a lot of manual work.

The Solution: The "Text-Only" Detective

This paper introduces a new method called WSRVOS. It's like teaching the computer to be a detective who only needs a verbal description (a text sentence) to find the target. It doesn't need any drawings, boxes, or dots. It just needs the sentence: "The little girl in the red dress."

Here is how WSRVOS works, broken down into five simple steps using a creative analogy:

1. The Creative Writing Coach (Contrastive Augmentation)

The original text descriptions in video datasets are often too simple, like just saying "a girl." The computer gets confused.

What WSRVOS does: It uses a super-smart AI (a Multimodal Large Language Model) to act as a creative writing coach.
The Analogy: Imagine you give the coach the sentence "a girl." The coach writes better versions for you: "A girl in a red dress blowing candles" (Positive). But to make the computer smarter, the coach also writes tricky versions: "A boy in a blue shirt" or "A girl eating cake" (Negative).
Why? By showing the computer both the right description and the wrong-but-similar descriptions, it learns to spot the tiny details that matter.

2. The Filter & The Translator (Feature Selection & Interaction)

Videos are messy. A 10-second clip might have a girl, a cake, a dog, and a clown. The text only cares about the girl.

What WSRVOS does: It acts like a bouncer at a club and a translator at the same time.
The Analogy:
- The Bouncer: It looks at the video and says, "Hey, the clown and the dog aren't mentioned in the text. Get out!" It filters out the visual noise.
- The Translator: It looks at the text and says, "The word 'red' is important, but the word 'the' is useless." It filters out the text noise.
- The Interaction: Then, it forces the "filtered video" and "filtered text" to talk to each other, ensuring they are perfectly synchronized.

3. The Quiz Master (Instance-Aware Classification)

Now the computer has to prove it understands the difference between the "Right Girl" and the "Wrong Girl."

What WSRVOS does: It runs a quiz.
The Analogy: The computer is shown a video frame and a list of sentences (some right, some wrong). It has to point to the sentence that matches the video. If it picks the wrong one, it gets a "red pen" correction. This forces the model to learn the specific differences between similar-looking objects.

4. The Crowd-Sourced Map (Positive-Prediction Fusion)

Remember those "better" descriptions the coach wrote in Step 1?

What WSRVOS does: It takes all the different ways the computer found the girl using those different descriptions and blends them together.
The Analogy: Imagine three different detectives are looking for the girl. Detective A says, "She's near the cake." Detective B says, "She's wearing red." Detective C says, "She's blowing air."
- WSRVOS takes all three clues and draws a single, super-accurate map (a "pseudo-mask") of where the girl is. It uses this map as a "teacher" to train the computer further, even though no human drew that map originally.

5. The Smooth-Operator (Temporal Segment Ranking)

Videos move. If the girl moves smoothly, her position in frame 10 should be very close to her position in frame 11, but further away from frame 100.

What WSRVOS does: It adds a rule to keep the movement logical.
The Analogy: It's like a dance instructor. If the computer draws the girl's outline in a wobbly, jumping way (like a glitchy video), the instructor slaps its hand and says, "No! If she is here in frame 1, she can't teleport to the other side of the room in frame 2. Keep it smooth!" This ensures the video segmentation looks natural and steady.

The Result

By using only text and these clever tricks, WSRVOS can find and outline objects in videos almost as well as methods that require expensive human drawings. It's faster, cheaper, and proves that you don't need a human artist for every frame if you have a smart enough AI to understand the story.

In short: It turns a computer from a "dumb tracer" into a "smart reader" that can watch a video, listen to a description, and draw the picture for you.

1. Problem Statement

Referring Video Object Segmentation (RVOS) aims to segment a specific target object in a video based on a natural language expression (e.g., "the person holding a red umbrella").

Current Limitation: State-of-the-art RVOS methods rely heavily on pixel-level mask annotations, which are expensive, time-consuming, and labor-intensive to create for every frame.
Existing Weak Supervision: Previous weakly-supervised approaches use bounding boxes or points, which are still costly to annotate across multiple frames.
The Goal: The authors propose WSRVOS, a framework that trains an RVOS model using only text expressions (no masks, boxes, or points), leveraging the capabilities of Multimodal Large Language Models (MLLMs) to overcome supervision scarcity.

2. Methodology

The WSRVOS framework is an end-to-end system comprising five core components:

A. Contrastive Referring Expression Augmentation (CREA)

Since original dataset expressions are often simple, the authors use an MLLM (specifically Qwen3-VL) to generate richer training signals:

Positive Expressions: The MLLM enriches the original text by adding details about visual appearance (color, texture), actions, and inter-instance relations. These are filtered using a visual-language similarity score (via InternVideo2) to ensure accuracy.
Negative Expressions: Instead of random sampling, the MLLM generates "hard negatives" by altering the target's attributes (e.g., changing "red" to "blue") or actions. This forces the model to learn discriminative features.
Usage: This augmentation is performed offline; the MLLM is not used during inference.

B. Bi-directional Vision-Language Feature Selection & Interaction

To handle the temporal dynamics and semantic redundancy in videos:

Feature Selection: A module selects the top- $K$ visual tokens most relevant to linguistic tokens (and vice versa) at multiple encoder layers. Irrelevant tokens are zeroed out, and the selected ones are added back to the original features.
Feature Interaction: A bi-directional attention mechanism allows visual features to query linguistic features (to find relevant regions) and linguistic features to query visual features (to identify relevant text components), refining the representations before final alignment.

C. Instance-Aware Expression Classification (IEC)

Inspired by Multiple Instance Learning (MIL), this module distinguishes between positive and negative expressions:

Proposal Aggregation: The model computes similarity maps between video frames and all generated expressions. It aggregates visual features from high-similarity regions to create "proposal features."
Dual-Flow Classification:
1. Classification Flow: Predicts which expression corresponds to a specific proposal. Notably, the weights here are fixed based on the linguistic features of the expressions (non-trainable), ensuring semantic consistency.
2. Segmentation Flow: Learnable weights that determine which proposals contain informative frame fragments.
Loss: A binary cross-entropy loss supervises the distinction between positive and negative expressions.

D. Positive-Prediction Fusion (PPF)

To generate supervision signals without ground-truth masks:

The model re-weights the similarity maps of all positive expressions based on their generation confidence scores.
These weighted maps are fused to create a high-quality pseudo-mask.
This pseudo-mask serves as a target for a segmentation loss (Focal + DICE loss), effectively training the model to segment the object.

E. Temporal Segment Ranking (TSR)

To ensure temporal consistency:

The model enforces a constraint that the Intersection over Union (IoU) between mask predictions of temporally closer frames should be higher than that of distant frames.
This is implemented via a ranking loss that penalizes violations of this temporal order, smoothing the segmentation over time.

3. Key Contributions

Text-Only Supervision: The first RVOS framework to achieve competitive performance using only text expressions, eliminating the need for any spatial annotations (masks, boxes, points).
MLLM-Driven Augmentation: A novel scheme using MLLMs to generate diverse, high-quality positive and hard-negative expressions, solving the data scarcity and semantic simplicity issues in existing datasets.
Fine-Grained Alignment: The introduction of bi-directional feature selection and interaction modules to handle the heterogeneity between video and text features.
Pseudo-Mask Generation: A strategy to synthesize reliable pseudo-masks from multiple positive predictions, acting as a robust surrogate for ground-truth masks.
Temporal Consistency: A ranking constraint that explicitly models the temporal dynamics of object overlap, improving video coherence.

4. Experimental Results

The method was evaluated on four standard benchmarks: A2D-Sentences, J-HMDB-Sentences, Ref-YouTube-VOS, and Ref-DAVIS17.

Performance: WSRVOS significantly outperforms existing weakly-supervised methods (using boxes or points) and even surpasses some fully-supervised methods in specific metrics.
- On Ref-YouTube-VOS, it achieved a J&F score of 36.5%, outperforming the previous best text-supervised method (DViN) by +7.3%.
- It achieves comparable or better results than point-supervised methods (e.g., OCPG) while requiring zero manual point annotations.
Efficiency: WSRVOS is highly efficient, requiring only 31M parameters and running at 58 FPS, making it faster and lighter than many SOTA models (e.g., ReferDINO).
Ablation Studies: Removing any core component (CREA, Feature Selection, IEC, PPF, or TSR) resulted in significant performance drops, validating the necessity of each module.

5. Significance

Paradigm Shift: WSRVOS demonstrates that high-quality video object segmentation can be achieved without pixel-level annotations, drastically reducing the cost of data preparation.
Scalability: By relying solely on text, the method can be scaled to massive video datasets where text descriptions are abundant but masks are not.
MLLM Integration: It effectively bridges the gap between generative AI (MLLMs) and discriminative computer vision tasks, showing how LLMs can be used for data augmentation and supervision in weakly-supervised learning.
Practical Application: The real-time capability and lack of annotation requirements make this approach highly suitable for applications like automated video editing, surveillance, and human-computer interaction.