Technical Summary: Multimodal Data Curation Through Ranked Retrieval

Problem Statement

Shared embedding spaces are the standard backbone for multimodal search and data curation. However, the authors identify two compounding issues that degrade the effectiveness of these systems, particularly when blending heterogeneous, human-labeled datasets:

1. Modality-Driven Clustering (The Modality Gap): Embeddings often cluster by input modality (e.g., all text together, all images together) rather than by semantic meaning. This geometric separation causes nearest-neighbor retrieval to prioritize modality identity over semantic similarity, undermining cross-modal search and downstream tasks like clustering or deduplication.
2. Noisy Paired Supervision: Training signals in paired datasets (e.g., image-caption, video-transcript) frequently suffer from misalignment. Annotations may omit critical objects, include irrelevant context, or refer to content not present in the raw data (e.g., off-screen audio). When models are trained on such noisy pairs, they learn shortcuts that correlate with the dataset or modality rather than the underlying event, reinforcing the modality gap.

Existing solutions typically address only one side of this problem: representation-level methods attempt to align embeddings but accept noisy supervision as given, while data-centric filtering improves pair quality but often assumes a fixed encoder architecture that cannot correct for systematic geometric biases.

Methodology

The paper proposes a coupled framework that intervenes simultaneously on the training data pairs and the embedding model architecture. The approach consists of three main components:

1. Symmetric Nucleus Subsampling (SNS)

SNS is a data-side preprocessing technique designed to increase the information density of paired samples by trimming irrelevant content from both the raw input and the annotation. It operates via two phases:

• Forward Extraction: Identifies the most relevant portions of the raw data (e.g., specific image regions, video moments, or text spans) that explain the annotation content. This is achieved using modality-specific segmentation models (e.g., Grounding-DINO for images, moment detection for video) guided by embedding similarity.
• Backward Extraction: Identifies the most grounded portions of the annotation that are supported by the raw data. This involves generating a description of the raw data (using models like Nemotron-Nano or Cosmos-Reason2) and filtering the original annotation to retain only sentences with high similarity to this description.
• Symmetric Gate: A pair is retained or modified only if the mutual information ratio between the extracted "nucleus" and the original pair exceeds a threshold ($\rho$). This ensures that trimming improves alignment without discarding valid data.

2. Expert Embedding Engine (EEE)

To address modality bias, the authors propose a Mixture-of-Experts (MoE) architecture that leverages three distinct embedding strategies, each with different strengths and inherent biases:

• End-to-End Expert: Uses a unified multimodal model (Omni-Embed-Nemotron-3B) to process all modalities directly.
• Fusion Expert: Uses an ImageBind-style architecture that fuses embeddings from separate encoders, using images as an anchor.
• Text Expert: Converts all modalities to text descriptions using modality-to-text models and then uses a dedicated text embedding model (Llama-Nemotron-Embed-1B).

3. Projection Network with Bias-Aware Objective

Since the three experts produce embeddings in different spaces, a lightweight neural network is trained to project the concatenated embeddings from all experts into a unified space. This network is trained with a composite loss function:

• Task Loss ($L_{task}$): An InfoNCE-style contrastive loss to maintain semantic similarity between paired data and annotations.
• Cluster Bias Loss ($L_{cluster}$): Penalizes the distance between the centroid of each modality and the global centroid, forcing modalities to overlap in the shared space.
• Scale Bias Loss ($L_{scale}$): Penalizes differences in the spread (variance) of embeddings across modalities to ensure uniform cluster geometry.

Crucially, the projection network treats the text annotations as "grounded anchors" (leaving their embeddings unchanged) and projects the raw data embeddings toward them, ensuring the geometry is guided by the text modality.

Key Contributions

1. Analysis of Joint Failure Modes: The paper analyzes how modality bias and raw-annotation misalignment jointly degrade embedding quality in mixed, human-labeled corpora.
2. Symmetric Nucleus Subsampling (SNS): A novel, symmetric subsampling method that improves pair consistency by trimming irrelevant content from both sides of a multimodal pair based on mutual information density.
3. Expert Embedding Engine (EEE): A mixture-of-experts architecture combined with a learned projection layer and a bias-aware objective to reduce modality-driven separation while preserving semantic structure.
4. Robust Evaluation: A comprehensive study using human-selected task sets to compare downstream model performance across various data curation strategies, including the proposed method and industry-standard baselines.

Results

The framework was evaluated on a mixture of five human-annotated datasets (AudioCaps, TextCaps, ALFRED, CoNaLa, TriviaQA) spanning audio, image, video, and text. The curated data blends were used to fine-tune a multimodal language model (Qwen2.5-Omni-3B), with validation perplexity serving as the primary metric.

• Downstream Performance: The full configuration (SNS Bidirectional + EEE + Projection) achieved the lowest validation perplexity (7.732), outperforming stratified random sampling, uniform random sampling, and a "Traditional" data curation pipeline (which relied on heuristic filtering and single-encoder ranking).
• Modality Gap Reduction: The projection network successfully collapsed the modality gap. Table 2 shows that the $\ell_2$ distance between per-modality centroids was reduced by over 90% on average compared to the base expert embeddings.
• Ablation Insights:
  • SNS variants (Forward, Backward, Bidirectional) consistently outperformed the EEE-only baseline, confirming the value of data-side denoising.
  • The "Traditional" baseline performed significantly worse, highlighting the limitations of heuristic filtering and single-encoder approaches in multimodal settings.
  • The projection network was essential; without it, the distinct embedding spaces of the experts could not be effectively ranked or combined.

Significance and Claims

The paper claims that standard retrieval-based data selection can actively degrade training mixtures if the embedding space exhibits uncorrected modality bias. The proposed framework demonstrates that addressing both the data (via SNS) and the model geometry (via EEE and projection) is necessary to create robust, unified embedding spaces.

The authors assert that their approach enables reliable cross-modal retrieval by ensuring that distance in the embedding space reflects semantic similarity rather than input format. While the evaluation is limited in scale (5,000 samples, specific model architectures), the results suggest that "small geometric quirks and label noise" can compound into systematic errors in large-scale training, and that a coupled intervention strategy is a viable solution. The paper concludes by noting limitations, such as the reliance on text-based supervision for the traditional baseline and the scale of the evaluation, suggesting future work could focus on uncertainty-aware trimming and decoupling proposal from verification.