💻 computer science

PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition

The paper introduces PRIMED, a novel framework for Referring Audio-Visual Segmentation that leverages biased competition theory to adaptively suppress irrelevant modalities through a modality prior decoder and token distiller, achieving state-of-the-art performance by dynamically adjusting attention based on the specific needs of each referring expression.

Original authors: Yuchen He, Jing Zhang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuchen He, Jing Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Three-Headed Monster" Problem

Imagine you are trying to find a specific person in a crowded, noisy room based on a description like, "The person playing the flute."

Your Eyes (Vision): You see many people. Some are wearing red, some are dancing, some are holding instruments.
Your Ears (Audio): You hear a flute, a drum, and a guitar.
Your Brain (Text): You have the sentence "The person playing the flute."

The problem with current AI systems is that they treat all these clues as a giant, messy smoothie. They mix the sound of the flute, the sight of a dancer, and the words "playing the flute" together without deciding which clue matters most. If the drum is very loud, the AI might get distracted and point at the drummer, even though the text said "flute."

PRIMED is a new AI system designed to fix this. It acts like a smart detective that knows when to listen to its eyes, when to listen to its ears, and when to trust the written clue.

How PRIMED Works: The "Biased Competition" Theory

The paper is inspired by a concept from neuroscience called Biased Competition.

The Analogy: The Talent Show
Imagine a talent show where many acts are on stage at once (a singer, a magician, a juggler). The judges (the AI) have limited attention.

Bottom-up: The acts are all shouting and performing (this is the raw video and audio data).
Top-down: The host announces, "We are looking for the magician!" (this is the text description).

In the old way, the judges tried to watch everyone equally, getting confused by the loud singer.
In PRIMED, the judges use the host's announcement to bias the competition. They actively suppress the singer and the juggler so they can focus entirely on the magician.

PRIMED does this in three main steps:

1. The "Modality Prior Decoder" (The Smart Detective)

Before the AI even starts looking at the video, it reads the text description and asks: "Is this task mostly about what I hear, what I see, or a mix of both?"

If the text says "The loud drum," the AI knows to trust its ears more.
If the text says "The red car," it knows to trust its eyes more.
It creates a "Modality Prior"—a mental note that says, "Focus 80% on audio, 20% on video." This note acts as a filter to ignore irrelevant noise.

2. The "Token Distiller" (The Global GPS)

Sometimes, looking at a single frame is confusing. You might see a hand holding a flute, but is it the right hand?
PRIMED takes a "snapshot" of the whole video's most important visual information and compresses it into a few compact tokens (like a set of GPS coordinates).

These tokens are shared across all stages of the AI's processing.
The Analogy: Imagine you are navigating a city. Instead of just looking at the street corner you are standing on, you also have a map in your hand showing the whole city. This "global map" helps the AI stay consistent and not get lost in local details.

3. The "Competition" (The Final Showdown)

Now, the AI brings the text clues, the audio clues, and the visual clues together.

Because of the Modality Prior (Step 1), the AI knows which clues to weigh heavily.
Because of the Global Map (Step 2), it knows where the object is supposed to be in the big picture.
They "compete" for attention. The irrelevant clues (like a loud drum when you are looking for a flute) are suppressed (silenced), and the correct clues are enhanced (amplified).

The Extra Polish: "Spatial-Aware Semantic Alignment"

To make sure the AI doesn't accidentally grab the background (like the wall behind the flute player), the researchers added a special training rule.

The Analogy: It's like a teacher telling a student, "Make sure you are looking at the flute, not the wall behind it."
The AI is trained to push the "flute" signal away from the "wall" signal, making the final picture much sharper and more accurate.

What Did They Find?

The paper tested PRIMED on a benchmark dataset (a collection of videos with text descriptions).

The Result: PRIMED beat all previous methods. It was better at finding the right object, even when there were many distractions (like loud noises or confusing visuals).
Why it worked: By explicitly deciding which sense to trust (vision vs. audio) rather than mixing them blindly, the AI became much more robust. It could handle tricky situations where the text description was vague or the environment was noisy.

Summary

PRIMED is an AI that stops trying to be a "jack of all trades" and instead becomes a smart strategist. It reads the instructions, decides which senses to use, filters out the distractions, and uses a global map to find the exact target. It turns a chaotic mix of sound, sight, and words into a clear, accurate answer.

Technical Summary: PRIMED – Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition

Problem Statement
Referring Audio-Visual Segmentation (Ref-AVS) aims to localize and segment target objects in video frames based on visual, auditory, and textual referring cues. While recent multimodal learning has advanced bimodal tasks, extending these to Ref-AVS remains challenging due to the need for fine-grained alignment across three heterogeneous modalities. Existing approaches—categorized as fusion-based, prompt-based, or reasoning-based—typically treat multimodal inputs as homogeneous signals to be fused or reasoned over. A critical limitation of these methods is their failure to explicitly model the distinct roles and varying relevance of each modality. Consequently, when multiple candidate regions partially satisfy a referring expression, irrelevant or misleading modalities (e.g., a salient sound from a distractor object) can interfere with robust target segmentation.

Methodology
The authors propose PRIMED (PRior-Informed Modulation and Embedding Distillation), a framework built upon a frozen SAM2 backbone that explicitly models the "biased competition" theory from cognitive neuroscience. In this paradigm, bottom-up visual evidence competes for representation, while top-down signals (the referring expression) bias the competition toward task-relevant targets. The architecture consists of four key components:

Semantic Flow Generator & Modality Prior Decoder (MPD):
- Text and audio features are aligned into a unified semantic flow. To handle temporal audio dynamics, a cached memory mechanism amplifies current audio representations when they deviate significantly from historical context.
- The Modality Prior Decoder (MPD) is a lightweight MLP that predicts a modality prior ( $P_M$ ) characterizing whether the referring expression relies primarily on audio, vision, or their joint interaction. Since explicit modality annotations are unavailable, the authors use Qwen3-omni to generate soft labels for training, verified by an audit prompt to ensure reliability.
Token Distiller:
- To provide a compact high-level visual bias, the framework distills the highest-level features from the Hiera trunk into a set of learnable global visual tokens.
- These tokens are shared across all fusion stages via cross-attention, ensuring consistent global context. Orthogonality regularization is applied during training to encourage diversity among these tokens.
Cross-modal Biased Competition Fusion (CBCF):
- A cascade of CBCF modules integrates the unified semantic flow with multi-scale visual features.
- In lower stages, semantic interactions guide fine-grained alignment. In the highest stage (Stage 3), the Modality Prior is injected as an additive attention bias into the Multi-Head Cross-Attention (MHCA) blocks. This bias modulates the competition between visual and semantic cues, suppressing irrelevant modalities based on the predicted prior.
- The fused features are injected into the SAM2 decoder as both dense (visual) and sparse (semantic) prompts.
Spatial-Aware Semantic Alignment (SASA) Loss:
- To enhance foreground-background discrimination, a contrastive loss aligns the semantic anchor of each frame with the foreground prototype while pushing it away from the top- $k$ most similar background tokens.

Key Contributions

Reframing Ref-AVS: The paper reinterprets Ref-AVS through the lens of modality-aware biased competition, emphasizing the necessity of modeling the varying relevance of modalities to resolve ambiguity in target selection.
PRIMED Framework: A novel SAM2-based architecture that explicitly implements this theory via:
- A Modality Prior Decoder providing top-down guidance.
- CBCF modules that progressively integrate semantics into visual hierarchies.
- A Token Distiller propagating compact global context.
Enhanced Discrimination: The introduction of the SASA loss and orthogonality regularization to improve the separation of target regions from confusing background distractors.

Experimental Results
Extensive experiments were conducted on the Ref-AVS benchmark (containing seen, unseen, and null splits).

Performance: PRIMED achieves state-of-the-art overall performance, obtaining the best mixed Jaccard Index (J) and F-score (F). Specifically, it improves the seen split J by 2.8% and the unseen split J by 2.1% over the previous best method, AURORA.
Generalization: The method shows consistent improvements over other SAM-based approaches (e.g., SAM2-LOVE, TSAM), particularly in the unseen split, suggesting that explicit modality competition modeling enhances generalization to new object classes.
Ablation Studies: Removing the modality prior or the biased competition mechanism leads to performance degradation, particularly in scenarios with strong distractors (e.g., a flute sounding louder than the target piano). The ablation of the SASA loss confirms its role in foreground-background separation, while the Token Distiller is shown to be critical for stable competition across stages.

Significance and Claims
The paper claims that PRIMED offers a promising alternative to treating multimodal cues as homogeneous inputs. By explicitly modeling how different modalities participate in candidate competition, the framework achieves more reliable target selection under multimodal ambiguity. The authors note that while PRIMED excels at stabilizing target grounding, it still exhibits a gap in F-score compared to recent MLLM-based reasoning methods, indicating that fine boundary refinement and high-level semantic reasoning for complex scenarios remain areas for future improvement. The work suggests that leveraging modality-aware priors and hierarchical visual context is a crucial direction for future multimodal segmentation research.