Technical Summary: Mixture of Probes (MoP) for Privileged Modalities in MLLMs

1. Problem Formulation: The Privileged Modality Setting

Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training (e.g., video, depth, audio) will also be accessible during inference. However, real-world deployment often violates this assumption. A model may be trained with rich auxiliary streams (privileged modalities) but must operate using only a single, cheaper, or more reliable modality at test time. This scenario is defined as the privileged modality setting.

Existing MLLMs struggle in this setting because they primarily treat modalities as interchangeable inputs to be aligned into a shared representation space. They fail to effectively leverage auxiliary modalities available only during training to improve performance on the target modality at inference. The authors identify two key limitations in current approaches:

1. Sub-optimal modality representations: New modalities are often incorporated via shared encoders with limited adaptation at final layers, neglecting intermediate features that capture low-level, modality-specific signals.
2. Unstructured cross-modal representations: Existing methods lack explicit mechanisms to disentangle modality-general information (transferable across modalities) from modality-specific information, hindering effective knowledge transfer.

2. Methodology: Mixture of Probes (MoP)

To address these limitations, the authors propose Mixture of Probes (MoP), a framework designed to disentangle modality-specific and modality-general signals within an MLLM. The core intuition, inspired by neuroscience and the Modality Focusing Hypothesis, is that effective cross-modal transfer requires preserving unique modality structures while isolating shared abstractions.

2.1 Architecture

MoP augments a standard unified MLLM architecture (consisting of modality-specific tokenizers, a universal modality encoder, a connector, and an LLM) with a structured probing mechanism:

• Universal Modality Encoder: A shared Transformer encoder processes tokenized inputs from all modalities into a common latent space.
• Learnable Probes: Instead of relying solely on the final encoder layer, MoP introduces two families of learnable tokens (probes) that interact with the encoder's intermediate representations:
  • Modality-Specific Probes ($P^m_s$): A set of probes conditioned only on a specific modality $m$. These capture signals unique to that modality.
  • Modality-General Probes ($P_g$): A shared set of probes conditioned on all modalities. These capture structural information common across modalities.
• Layer-wise Interaction: Both probe sets are updated at every layer $l$ of the encoder via cross-attention with the encoder's hidden states ($H^m_l$). The probes act as queries, extracting salient cues from intermediate representations that might be lost in the final layer.
• Integration: After the final encoder layer, the probe tokens are projected into the LLM embedding space and concatenated with the standard modality tokens before being fed to the LLM.

2.2 MoP Cross-modal Training (MoP-X)

To ensure the probes learn distinct representations and facilitate cross-modal transfer, the authors introduce a specific training strategy:

1. Probe Disentanglement Loss ($\mathcal{L}_{dis}$): A loss function that explicitly penalizes similarity between modality-specific and modality-general probes, as well as between different modality-specific probes. This prevents "probe collapse" (where probes converge to identical representations) and encourages the model to learn orthogonal, specialized features.
2. Modality-Interleaved Batching: Unlike methods that train modalities sequentially or in isolation, MoP-X constructs mini-batches by sampling examples from multiple modalities simultaneously. This ensures that shared components (like modality-general probes) are exposed to heterogeneous signals within a single optimization step, facilitating indirect cross-modal interaction.

The total training objective combines the standard autoregressive language modeling loss ($\mathcal{L}_{AR}$) with the disentanglement loss:
$$\mathcal{L} = \mathcal{L}_{AR} + \lambda \mathcal{L}_{dis}$$

3. Experimental Evaluation

The authors evaluate MoP across two domains, eight tasks, and four modalities under a rigorous protocol where each modality is treated as the sole input at inference time.

3.1 Domains and Benchmarks

• Activities of Daily Living (ADL): Evaluated on the Ego-in-Exo Perception benchmark. Training utilizes egocentric video, exocentric video, and depth. Inference is tested separately on egocentric, exocentric, and depth modalities.
• Music Understanding: Evaluated on Music-AVQA. Training utilizes paired audio-visual data. Inference is tested separately on audio and video modalities.

3.2 Baselines

MoP is compared against:

• Unimodal MLLM: Trained only on the target inference modality.
• Naive MLLM: Trained jointly on all modalities without structured cross-modal mechanisms.
• State-of-the-Art MLLMs: Including X-InstructBLIP and OneLLM, which focus on multimodal alignment.

3.3 Key Results

• Superior Performance: MoP consistently outperforms all baselines across all inference modalities. In the ADL domain, it achieves up to 65% relative improvement over strong baselines in specific settings (e.g., depth inference).
• Effective Privileged Learning: Unlike the Naive MLLM, which often fails to improve over unimodal baselines when trained on multiple modalities, MoP successfully leverages auxiliary modalities to boost single-modality performance.
• Ablation Studies:
  • Probe Types: Removing either modality-specific or modality-general probes degrades performance, confirming the necessity of both.
  • Disentanglement Loss: The loss is critical; without it, probe variance collapses, and performance drops significantly.
  • Probing Depth: Probing at every encoder layer yields the best results, suggesting the benefit of extracting signals throughout the model's depth.
  • Data Scaling: MoP demonstrates consistent performance gains as training data scales, whereas Naive MLLMs show diminishing returns.

4. Contributions and Significance

The paper claims the following contributions:

1. Formalization of the Privileged Modality Setting: The authors formalize the scenario where auxiliary modalities enrich training but are absent at inference, demonstrating that standard multimodal training fails to exploit this asymmetry.
2. Mixture of Probes (MoP): A novel architecture that learns disentangled representations through structured probing across the depth of a universal encoder, separating modality-specific signals from modality-general abstractions.
3. MoP Cross-modal Training (MoP-X): A training strategy featuring a probe disentanglement loss and interleaved batching to enforce representation separation and promote cross-modal interaction.
4. Empirical Validation: Comprehensive evaluation showing that auxiliary modalities, when properly leveraged via MoP, can substantially improve single-modality inference performance, outperforming both unimodal models and existing multimodal alignment methods.

Significance: The work challenges the prevailing paradigm of treating modalities merely as interchangeable inputs for alignment. Instead, it posits that the key to leveraging privileged information lies in explicitly structuring the model to separate what is unique to a modality from what is shared. This allows MLLMs to "learn from" rich, unavailable-at-inference data to become more robust and capable in constrained, single-modality deployment scenarios.

Limitations: The authors note that MoP currently assumes a universal modality encoder and does not directly extend to heterogeneous modality-specific encoders. Furthermore, the study focuses strictly on the single-modality inference setting, leaving multi-modal inference scenarios for future work.