🤖 machine learning

MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

MultiLoReFT is an efficient low-rank fine-tuning framework that decouples shared and modality-specific information in pretrained unimodal models to enable scalable multimodal learning without requiring large-scale paired datasets.

Original authors: Sana Tonekaboni, Viktoria Schuster, Caroline Uhler

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sana Tonekaboni, Viktoria Schuster, Caroline Uhler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world. Right now, most robots are like specialists who only speak one language: one robot sees pictures but doesn't understand words, and another reads text but can't see images. To make them work together, we usually try to force them to learn a brand new, giant language from scratch. But that's like trying to teach a toddler a whole new language just by showing them a dictionary with no pictures—it takes forever, and you need millions of examples to get it right.

The big idea in this corner of computer science is "multimodal learning," which is just a fancy way of saying "teaching computers to use different senses at once, like sight and sound." The problem is that real-world data is messy. We often have a million photos and a million text descriptions, but they aren't perfectly paired up like a matching game. Plus, when we force these different senses to mix, they often get tangled up. It's like pouring milk and orange juice into the same glass; you get a drink, but you can't tell which part is which, and if you spill a little, you lose the whole flavor. Scientists want to know: Can we teach a robot to keep the "milk" (the unique visual details) and the "orange juice" (the unique sound details) separate, while still letting them mix to understand the big picture, without needing a million perfect examples?

Enter MultiLoReFT, a clever new method that acts like a masterful bartender for these digital drinks. Instead of trying to rebuild the whole robot from scratch, MultiLoReFT takes two already-smart, pre-trained specialists (one for images, one for text) and gives them a tiny, efficient upgrade. Think of it as adding a special set of "mixing straws" to their brains. These straws are designed to do two things at once: first, they pull out the parts of the information that both senses agree on (the shared story), and second, they keep the parts that are unique to just one sense (the specific visual texture or the unique tone of voice) in separate, neat compartments.

The paper shows that this approach is surprisingly effective. By using a technique called "low-rank representation fine-tuning," the method only tweaks a tiny fraction of the robot's brain, making it fast and cheap to train. The researchers tested this on both made-up data (where they knew the exact answer) and real-world datasets, like videos of people speaking or images with captions in different languages. They found that MultiLoReFT successfully untangled the information: the "shared" part of the brain learned the general meaning, while the "unique" parts kept the specific details safe.

What's really cool is that this method doesn't just separate the information; it makes the robot more robust. If you take away one sense—say, you mute the audio—the robot can still guess what's happening because the "shared" part of its brain has learned to carry the weight of the missing sense. It's like if you lost your hearing, but your brain had already learned to read lips so well that it could fill in the gaps. The paper suggests that this works better than older methods that try to mash everything together or force a perfect separation that doesn't quite stick. While the results are very promising in simulations and on specific real-world datasets, the authors note that this is a step forward in making AI more efficient and understandable, especially in fields like healthcare where data is hard to get and understanding why a decision was made is just as important as the decision itself.

Technical Summary: MultiLoReFT

1. Problem Statement

Multimodal learning faces two primary obstacles in real-world applications, particularly in data-scarce domains like healthcare:

Data Scarcity: Collecting large-scale, well-aligned paired multimodal datasets is often impractical or prohibitively expensive, making end-to-end training of multimodal models difficult.
Entanglement: Existing multimodal representations frequently conflate information shared across modalities with modality-specific information. This entanglement hinders interpretability and control, making it difficult to understand how specific modalities contribute to a decision or to handle missing modalities robustly.

While parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) and Low-Rank Representation Fine-Tuning (LoReFT) offer efficient alternatives to full fine-tuning for unimodal models, they have not been explicitly extended to disentangle shared and unique information in multimodal settings.

2. Methodology: MultiLoReFT

The authors propose MultiLoReFT, a low-rank representation fine-tuning framework designed to adapt pretrained unimodal encoders for multimodal learning. The method operates on top of frozen pretrained encoders, requiring only a small number of additional parameters.

Core Architecture

Given two pretrained unimodal encoders ( $E_1, E_2$ ) producing representations $h_1, h_2 \in \mathbb{R}^d$ , MultiLoReFT learns fine-tuned representations $\Phi_1, \Phi_2$ that decouple information into:

Shared Subspace ( $R_s$ ): Captures information common across modalities.
Modality-Specific Subspaces ( $R_{m1}, R_{m2}$ ): Capture unique signals specific to each modality.

The framework applies structured low-rank interventions directly to the pretrained representations. The fine-tuned representation $\Phi_i$ for modality $i$ is computed as:
$\Phi_i = h_i + R_s^\top(f_{si}(h_i) - R_s h_i) + R_{mi}^\top(f_{mi}(h_i) - R_{mi} h_i)$
where $R_s, R_{mi} \in \mathbb{R}^{r \times d}$ are low-rank projection matrices, and $f_{si}, f_{mi}$ are lightweight nonlinear transforms learned in the low-rank subspaces. The terms $(f(h) - Rh)$ represent the edits applied in the low-rank space before projecting back to the original dimension.

Learning Objective

Unlike supervised fine-tuning, MultiLoReFT is trained using a self-supervised composite objective that enforces structural constraints on the representation space:
$\mathcal{L} = \lambda_1 \mathcal{L}_{indep} + \lambda_2 \mathcal{L}_{orth} + \lambda_3 \mathcal{L}_{MI}$

Independence Loss ( $\mathcal{L}_{indep}$ ): Minimizes statistical dependence between shared and modality-specific components (and between the two modality-specific components) using the Hilbert–Schmidt Independence Criterion (HSIC). This prevents information leakage.
Orthogonality Loss ( $\mathcal{L}_{orth}$ ): Enforces geometric disjointness between subspaces by minimizing the Frobenius norm of the pairwise inner products of the projection matrices ( $R_s, R_{m1}, R_{m2}$ ).
Cross-Modal Mutual Information Loss ( $\mathcal{L}_{MI}$ ): Uses an InfoNCE-style contrastive loss to ensure the disentangled projections retain sufficient information from the original unimodal embeddings and enforce consistency of shared components across modalities.

Adaptive Rank Pruning

To address the challenge of determining the optimal dimensionality of subspaces, MultiLoReFT employs an adaptive pruning mechanism. During training, the singular value decomposition (SVD) of projection matrices is computed. Directions with singular values below a threshold $\epsilon$ are pruned, and the matrices are updated with a compressed basis. This mechanism dynamically adjusts the rank based on information content, reducing redundancy and eliminating the need for manual rank tuning.

3. Key Contributions

Framework Extension: Extends Low-Rank Representation Fine-Tuning (LoReFT) to the multimodal setting, enabling efficient fusion of pretrained unimodal models.
Disentanglement: Introduces a method to explicitly learn orthogonal shared and modality-specific subspaces, enhancing interpretability by revealing the relative contributions of each modality.
Adaptive Efficiency: Proposes a dynamic rank pruning mechanism that automatically selects effective subspace dimensions, yielding compact parameterization without performance degradation.
Robustness: Demonstrates that the learned shared subspace improves robustness to missing modalities at inference time.

4. Experimental Results

The authors evaluated MultiLoReFT on synthetic datasets (Simulation I & II) and real-world benchmarks (Flickr30K-Multi, Crema-D, UR-FUNNY).

Information Decoupling: On synthetic data with known ground-truth generative factors, MultiLoReFT achieved the highest accuracy in predicting shared and modality-specific labels using only their respective components. It showed significantly larger performance gaps ( $\Delta$ ) between the correct component and others compared to baselines (DRIM-U, APOLLO), indicating superior separation and less information leakage. On real-world datasets, it effectively concentrated language information in the text-specific subspace and visual details in the image-specific subspace.
Downstream Performance: When fine-tuned representations ( $\Phi$ ) were concatenated and fed into a lightweight logistic regressor, MultiLoReFT consistently outperformed or matched state-of-the-art baselines, including late fusion, cross-attention, and other disentanglement methods.
Missing Modality Robustness: In scenarios where one modality was missing, the fine-tuned representations ( $\Phi$ ) provided stronger prediction signals than the original pretrained embeddings ( $h$ ), demonstrating that the shared subspace successfully captured cross-modal predictive information.
Parameter Efficiency: The adaptive pruning mechanism reduced the effective rank of projections, confirming the method's ability to learn compact representations.

5. Significance and Claims

The paper claims that MultiLoReFT offers a lightweight, interpretable, and scalable approach to multimodal learning in data-scarce settings. Its significance lies in:

Leveraging Unimodal Strengths: It allows the community to utilize powerful, pretrained unimodal models for multimodal tasks without the need for massive paired datasets.
Interpretability: By decoupling shared and specific information, the framework provides insights into how different modalities contribute to a model's decision, which is critical for high-stakes domains like healthcare.
Practicality: The method is model-agnostic and parameter-efficient, making it suitable for deployment where computational resources and data availability are limited.

The authors note that while the current study focuses on bimodal data, the framework can be extended to more modalities, though this introduces challenges in interpreting partially shared factors. They also acknowledge that the method's performance is ultimately bounded by the quality of the underlying pretrained unimodal encoders.