Technical Summary: Multimodal LLMs under Pairwise Modalities

Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in cross-modal understanding and generation. However, their training typically relies on jointly curated multimodal datasets where all modalities (e.g., text, image, audio, 3D) are perfectly aligned for every instance. Constructing such multi-way aligned data is expensive, labor-intensive, and often infeasible in emerging domains like robotics, tactile sensing, and medical imaging.

This heavy dependence on fully aligned corpora limits the scalability of MLLMs to new modalities. In contrast, pairwise aligned data (e.g., text-image, image-tactile, text-3D) is far more abundant and naturally available from diverse sources. The central question addressed by this work is: How can we train a multimodal model using only paired modalities, without ever observing the full joint distribution?

Methodology: The MPM Framework

The authors propose Multimodal Pairwise Model (MPM), a two-stage framework designed to learn aligned latent representations and enable cross-modal generation using only pairwise supervision.

1. Theoretical Foundation: Identifiability

Before proposing the architecture, the paper provides a theoretical analysis of subspace identifiability under pairwise modalities.

• Generative Model: The authors define a generative process where observations arise from shared latent factors ($z_c$) and modality-specific factors ($z_s$).
• Identifiability Conditions: They establish that if the modality graph formed by pairwise datasets is sufficiently connected, the shared latent subspace can be recovered from overlapping pairwise observations.
• Key Insight: Under mild assumptions (smoothness, invertibility, and collective linear independence of neighbors), the shared latent block $z_c$ is identifiable up to an invertible mapping, even without full joint alignment. This provides a principled guarantee that shared semantics can be recovered from a chain of pairwise connections.

2. Stage I: Latent Representation Alignment

The first stage learns a unified shared latent space using only pairwise data.

• Architecture: An autoencoder framework is employed for each modality, consisting of an encoder ($Enc_m$) and a decoder ($Dec_m$). The encoder maps inputs into a shared latent block ($\hat{z}_c$) and a specific block ($\hat{z}_s$).
• Objectives:
  • Self-modal Reconstruction: Ensures the latent representation preserves all information about the original modality, preventing information collapse.
  • Cross-Modal Contrastive Alignment: Uses a CLIP-style objective to align shared representations ($\hat{z}_c$) of paired modalities.
  • Partial and Asymmetric Alignment: Recognizing that not all latent factors in a new modality correspond to the target modality (e.g., tactile roughness may not be explicitly described in text), the authors introduce a binary mask mechanism. This allows for partial alignment, selecting only the relevant shared components from the new modality to align with the target, avoiding spurious constraints.

3. Stage II: Cross-Modal Recomposition

The second stage integrates the newly learned modalities into a pre-trained MLLM without modifying the backbone.

• Mechanism: The encoder of a new modality (e.g., 3D point clouds) is aligned with the latent space of a target modality (e.g., text) learned in Stage I.
• Recomposition: The aligned shared representation is passed through an asymmetric mask and fed into the decoder of the target modality. This reconstructed representation serves as a surrogate input for the frozen pre-trained MLLM.
• Advantage: This design allows the new modality to leverage the generative and reasoning capabilities of the pre-trained LLM without requiring full joint retraining, thereby avoiding catastrophic forgetting.

Key Contributions

1. Theoretical Analysis: The paper provides a formal proof that shared latent representations are identifiable from overlapping pairwise modality data under specific connectivity and rank conditions, offering a theoretical basis for pairwise-only training.
2. Unified Framework: It proposes a two-stage framework (Latent Alignment + Cross-Modal Recomposition) that enables scalable modality extension using only pairwise supervision, eliminating the need for fully aligned multi-way datasets.
3. Empirical Validation: The method is validated by integrating 3D point clouds and tactile sensing into a pre-trained MLLM (Qwen3-Omni), demonstrating strong cross-modal transfer performance.

Experimental Results

The authors evaluated MPM by adding 3D point clouds and tactile modalities to Qwen3-Omni-30B-A3B, a pre-trained MLLM.

• 3D Benchmarks: On ModelNet40 (object classification) and 3D MM-VET (3D QA), MPM significantly outperformed baselines, including full fine-tuning approaches (e.g., PointLLM, ShapeLLM) and adapter-based methods. Notably, MPM achieved 70.5% accuracy on ModelNet40 compared to 53.1% for the best full-fine-tuned baseline.
• Tactile Benchmarks: On the TVL dataset, MPM achieved the best performance among tactile-capable models (e.g., 5.22/10 on the TVL score), outperforming both the frozen backbone and adapter-based baselines.
• Ablation Studies:
  • Removing Stage I (recomposition-only) caused a significant performance drop, confirming the necessity of learning a structured shared latent space.
  • Removing Stage II (Stage I only) also degraded performance, highlighting the role of the frozen MLLM decoder in generating fluent language.
  • Both reconstruction and contrastive alignment losses in Stage I were found to be critical, with reconstruction being essential for preventing latent collapse.

Significance and Claims

The paper claims that pairwise modality supervision is a scalable and effective alternative to conventional fully jointly aligned multimodal training. By leveraging the theoretical insight that shared latent factors can be recovered from connected pairwise graphs, the authors demonstrate that:

• It is possible to extend pre-trained MLLMs to new modalities (like 3D and tactile) without the prohibitive cost of collecting fully aligned multi-modal datasets.
• The proposed framework preserves the original capabilities of the pre-trained model (avoiding catastrophic forgetting) while enabling new cross-modal generation tasks.
• The approach offers a principled mechanism for modality extension that does not rely on a universal anchor modality or extensive instruction tuning.

The work concludes that while the framework shows promising results, future work is needed to scale it to broader sets of modalities and validate its effectiveness on more diverse and complex downstream tasks.