🤖 machine learning

SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning

This paper proposes the Submodular Modality Aligner (SMA), a set-based multimodal learning framework that leverages Submodular Mutual Information to overcome data scarcity by capturing richer cross-modal geometric structures, thereby achieving strong zero-shot generalization with orders of magnitude fewer training samples than standard pairwise approaches.

Original authors: Truong Pham, Anay Majee, Rishabh Iyer

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Truong Pham, Anay Majee, Rishabh Iyer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Learning with a Tiny Dictionary

Imagine you are trying to teach a robot to understand the world by showing it pictures and reading it descriptions. Usually, to do this well, you need a massive library of millions of picture-and-description pairs. This is like teaching a child to speak by reading them an entire encyclopedia.

However, in many real-world situations (like rare medical scans or specific satellite photos), you don't have millions of examples. You might only have a few hundred. When you try to teach the robot with so little data using standard methods, it gets confused. It learns to memorize the specific examples rather than understanding the general concept, leading to a "modality gap"—a disconnect where the robot sees a picture of a cat but doesn't quite "get" the word "cat" in the same way.

The Old Way: The "One-on-One" Date

Current popular methods (like CLIP) treat learning like a series of one-on-one dates.

The Setup: You show the robot one picture of a dog and one sentence saying "a dog."
The Flaw: The robot learns that this specific picture matches this specific sentence. If you show it a different angle of the same dog or a slightly different sentence, the robot might get confused because it was trained to only look at single pairs in isolation. It misses the bigger picture of what a "dog" really is because it never saw the whole family of dog descriptions together.

The New Solution: SMA (The "Group Hug" Approach)

The authors propose a new method called SMA (Submodular Modality Aligner). Instead of one-on-one dates, SMA treats learning like a group hug or a team meeting.

1. The "Set" Concept
Imagine you have one object (a specific car). Instead of showing the robot just one photo and one caption, you give it a set:

5 different photos of that car (from different angles, lighting, etc.).
5 different descriptions of that car (e.g., "red sports car," "fast vehicle," "car with shiny wheels").

SMA tells the robot: "Don't just match Photo A to Sentence A. Look at the whole group of photos and the whole group of sentences. Figure out how they all fit together."

2. The "Submodular" Magic (The Diminishing Returns Rule)
The paper uses a mathematical concept called Submodularity. Think of this like building a playlist.

If you add a song to an empty playlist, it adds a lot of value.
If you add the exact same song again, it adds zero value.
If you add a similar song, it adds some value, but less than a completely new genre.

SMA uses this logic to tell the robot: "You don't need to memorize every single tiny detail of every photo. You just need to find the most important, unique details that represent the whole group." This prevents the robot from getting overwhelmed and helps it learn faster with less data.

3. Closing the Gap
The goal is to close the "Modality Gap." Imagine the robot has two rooms: one for pictures and one for words. In old methods, these rooms are far apart, and the robot has to run a long distance to connect them.
SMA builds a bridge between the rooms. By looking at the whole group of pictures and the whole group of words at once, it realizes, "Oh, these two groups are actually describing the same thing!" It pulls the picture-room and the word-room closer together, making the connection stronger and more accurate.

What Did They Find?

The researchers tested this new "Group Hug" method on 14 different tasks (like identifying flowers, cars, or finding specific images in a database).

The Result: They used tens of thousands of examples (a tiny amount compared to the millions usually needed).
The Outcome: SMA performed significantly better than the old methods. In some cases, it was 25% more accurate.
The Efficiency: It achieved these results with far fewer samples and less computing power.

The Bottom Line

The paper argues that we have been trying to teach AI by showing it isolated pairs of data, which is inefficient when data is scarce. By switching to a set-based approach—treating multiple views and descriptions of the same thing as a single, cohesive group—we can teach AI to understand the world much faster and with much less data. It's the difference between memorizing a single flashcard and understanding the whole story.

Technical Summary: Submodular Modality Aligner (SMA) for Data-Efficient Multimodal Learning

Problem Statement

Despite the success of Multimodal Foundation Models (FMs) like CLIP and SigLIP, their reliance on massive paired datasets (millions to billions of samples) limits their applicability in low-data regimes and rare scenarios (e.g., medical imaging, remote sensing) where aligned data is scarce and expensive. Current state-of-the-art methods rely on instance-level formulations (pairwise contrastive learning), which maximize correlation between individual image-text pairs. The authors argue this approach is structurally insufficient because:

It neglects the underlying geometric structure across modalities, leading to a persistent modality gap (where image and text embeddings remain linearly separable even in a shared space).
It treats multiple views or descriptions of the same entity as independent pairs, resulting in inefficient data usage and conflicting optimization signals (e.g., negative sampling pushing apart different views of the same object).
Standard objectives like InfoNCE often overfit in low-data settings and fail to jointly optimize for both alignment (correlation) and modality gap reduction (divergence).

Methodology: Submodular Modality Aligner (SMA)

The paper proposes a combinatorial, set-based paradigm for multimodal alignment. Instead of treating data as isolated pairs, SMA treats multiple augmentations and descriptions of a single entity as a set. The framework operates on frozen pre-trained unimodal encoders (e.g., DINOv2 for images, GTE-1.5 for text) and trains only a lightweight alignment layer.

Core Objective:
The method introduces a loss function based on Submodular Mutual Information (SMI). The objective jointly:

Maximizes inter-modality mutual information between sets of aligned image and text embeddings.
Minimizes cross-modal divergence between mismatched sets.

Loss Formulations:
The authors instantiate SMA using two specific SMI functions, adapted for continuous optimization via cosine similarity kernels:

FLVMIA (Facility Location Variant Mutual Information Alignment): Uses a min-max structure to enforce that the set of image embeddings is representative of the set of text embeddings, encouraging balanced coverage across modalities.
FLQMIA (Facility Location Mutual Information Alignment): Uses a max-based formulation to focus on the strongest cross-modal associations within the sets.

Theoretical Connections:

Generalization: The authors demonstrate that standard pairwise objectives (CLIP's NT-Xent and SigLIP's sigmoid-based loss) are special cases of their formulation when the sets collapse to singletons (i.e., $|A^+_i| = 1$ ).
Modality Gap: The paper posits that minimizing the modality gap is equivalent to maximizing the Submodular Mutual Information function over the entire set. While SMI alone does not solve the gap entirely (as it must be balanced with uniformity constraints), the formulation explicitly addresses both alignment and gap reduction simultaneously.

Key Contributions

Set-Based Formulation: Introduction of a novel combinatorial perspective for multimodal learning that models multiple views and descriptions of an entity jointly, rather than independently.
Principled Objective: The development of SMA, an objective based on Submodular Mutual Information that naturally enables the joint modeling of alignment (correlation) and the modality gap (divergence).
Data Efficiency: Demonstration that strong multimodal generalization can be achieved using only tens of thousands of samples (orders of magnitude fewer than standard approaches) by leveraging set structures.

Experimental Results

The authors evaluated SMA on 14 zero-shot classification tasks and 1 zero-shot retrieval task (Flickr30k) from the CLIP benchmark, using MS-COCO (80k images with 5 captions each) as the training data.

Performance Gains:
- In the multiple positive pair setting, SMA (specifically FLQMIA and FLVMIA) achieved consistent gains over baselines (SAIL w/ CLIP, SAIL w/ SigLIP, and CSA).
- Improvements ranged from 15% to 25% relative improvement on harder datasets, with an average proportional gain of 7.7% across zero-shot classification tasks and ~5% for retrieval tasks.
- On datasets like CIFAR-10 and STL-10, accuracies approached 90%, with SMA providing marginal but consistent improvements.
Single Pair Setting: When trained on single positive pairs (collapsing the set structure), SMA performed comparably to or slightly worse than SAIL on very difficult datasets (e.g., Cars, FGVC) but remained competitive on others. This highlights that the method's primary advantage lies in leveraging multiple positive associations.
Modality Gap Reduction: Analysis of the average $\ell_2$ distance between modalities showed that SMA (particularly FLVMIA) reduced the modality gap more effectively than SAIL and InfoNCE-based methods. For instance, FLVMIA achieved an average distance of 12.82 in the multiple pair setting compared to 15.40 for SAIL w/ SigLIP.
Efficiency: The model achieved these results using batch sizes of ~1k–16k (due to memory constraints of the combinatorial loss) and converged in fewer epochs (11 epochs for FLVMIA) compared to baselines that required up to 200 epochs.

Significance and Claims

The paper claims that multimodal alignment should fundamentally shift from isolated pair formulations to set-based formulations. The significance of this work lies in:

Data Efficiency: Proving that high-quality multimodal alignment does not require billions of samples but can be achieved with tens of thousands by exploiting the rich structure of multiple descriptions per entity.
Theoretical Insight: Highlighting that the "modality gap" is a consequence of pairwise formulations and that submodular objectives provide a principled way to address both alignment and gap reduction simultaneously.
Practical Impact: Offering a viable path for applying multimodal foundation models to domains with scarce data (e.g., medical, scientific, and security applications) where collecting massive paired datasets is infeasible.

The authors conclude that while SMA currently shows the strongest results in low-data, multi-view settings, future work could extend this framework to additional modalities (audio, video) and integrate it into larger-scale foundation models.