MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learninga

Technical Summary: MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning

1. Problem Statement

While memorization in machine learning models is known to aid performance on rare in-distribution samples by capturing atypical patterns, it also leads to the harmful retention of noise and outliers, degrading generalization. Extensive research has characterized memorization in supervised learning (SL) and self-supervised learning (SSL) within the vision domain. However, memorization in multi-modal contrastive learning remains unexplored.

Existing definitions of memorization are often modality-specific (e.g., based on label prediction in SL or augmentation matching in SSL) or limited to bi-modal settings (e.g., image-text pairs in CLIP). These definitions fail to capture the complex, global interactions across diverse modalities (such as audio, video, image, and text) inherent in multi-modal models. Furthermore, prior methods like d´ej`a vu or CLIPMem are not directly applicable to models involving more than two modalities. There is a critical need for a framework that can quantify memorization across all modalities simultaneously and understand how cross-modal inconsistencies drive this phenomenon.

2. Methodology

The MultiMem Metric
The authors introduce MultiMem, the first metric designed to quantify memorization in multi-modal contrastive learning for any number and type of modalities.

• Framework: It builds on the leave-one-out framework. Two models are trained: model $f$ on the full dataset $S$, and model $g$ on the dataset $S \setminus i$ (excluding a specific multi-modal sample $x_i$).
• Proxy for Memorization: Instead of relying on label prediction or pairwise alignment alone, MultiMem uses Cross-Modal Consistency (CMC) as a proxy. The CMC measures the alignment of representations across all modalities for a specific sample versus their alignment with unrelated samples from a held-out set $H$.
• Calculation:
  1. Representations for a sample $x_i$ across $n$ modalities are normalized and aggregated into a representation space $\Phi$.
  2. CMC is calculated as the difference between the expected similarity of modalities within the sample (intra-sample consistency) and the expected similarity between the sample and held-out samples (inter-sample separation).
  3. MultiMem is defined as the difference in CMC scores between model $f$ and model $g$:
     $$\text{MultiMem}(i, H, f) = \text{CMC}_{f \sim S}(i, H) - \text{CMC}_{g \sim S \setminus i}(i, H)$$
     A high MultiMem score indicates that the model's ability to align modalities for sample $x_i$ is heavily dependent on the presence of $x_i$ in the training set, signifying memorization.

Experimental Setup
The authors evaluated MultiMem on several models:

• AudioCLIP: Audio, Image, Text.
• AVT-CLIP: Audio, Video, Text (custom-built).
• AVIT-CLIP: Audio, Video, Image, Text (custom-built).
• Baselines: OpenCLIP, VideoCLIP.
  Datasets included UrbanSound8K, MSR-VTT, and COCO.

Mitigation Strategies
Based on the metric, two strategies were proposed to mitigate memorization:

1. In-Training Mitigation: During training, the top 5% of most memorized samples (identified via periodic MultiMem evaluation) are regrouped into new batches. Only these samples receive targeted noise-based augmentations (Gaussian noise) on their representations, while other batches remain unaltered.
2. Post-Training Mitigation: After initial training, the top memorized samples are identified and removed from the training set. The model is then fine-tuned on the remaining data.

3. Key Results and Findings

Characterization of Memorization

• Global vs. Partial: Memorization measured across all modalities (Global MultiMem) behaves differently than memorization measured on subsets of modalities (e.g., just Image-Text). Global assessment is necessary for accurate evaluation.
• Drivers of Memorization: The most memorized samples are not merely mis-captioned (as often found in bi-modal models) but exhibit semantic misalignment across all modalities. The information provided by different modalities often contradicts or is unrelated to one another.
• Modality Influence: Cross-modal semantic misalignment has the strongest influence on memorization. Text is the dominant modality driving memorization, followed by video, image, and audio.
• Behavioral Shift: Multi-modal models exhibit memorization behaviors closer to Self-Supervised Learning (SSL) (memorizing patterns) rather than Supervised Learning (memorizing labels). The introduction of audio and video shifts the vision encoder's behavior away from being purely caption-driven.
• Generalization Correlation: There is a strong negative correlation between global memorization and generalization performance (measured via retrieval, linear probing, and zero-shot tasks). Higher memorization leads to lower downstream performance.

Impact of Mitigation

• Augmentation: Applying augmentations to all modalities yields the lowest memorization and highest performance. Applying augmentations to only a single modality can increase memorization in non-augmented modality pairs.
• Mitigation Effectiveness: Both in-training and post-training mitigation strategies significantly reduced memorization (by up to 20%) and improved model performance:
  • Retrieval: Up to 8% improvement.
  • Zero-shot Classification: Up to 10% improvement.
  • Downstream Classification: Up to 4% improvement.
• Mechanism: Mitigation works by increasing the average modality distance for highly memorized samples (reducing their "over-alignment" due to noise) while preserving the overall modality gap of the dataset. In-training mitigation was found to be slightly more effective than post-training.

4. Significance and Claims

The paper claims to establish the first framework for measuring and mitigating memorization specifically for multi-modal contrastive learning. Its significance lies in:

1. Defining a New Metric: Providing MultiMem, which generalizes beyond bi-modal limitations to capture global interactions across arbitrary modality configurations.
2. Revealing New Patterns: Demonstrating that multi-modal memorization is driven by cross-modal semantic inconsistency rather than just label noise or single-modality outliers, and that these models behave more like SSL models than SL models.
3. Practical Mitigation: Showing that targeted interventions based on this metric (specifically noise augmentation on memorized samples) effectively prevent harmful data retention and lead to more robust, higher-performing models without requiring the removal of large portions of data.

The authors emphasize that their work prevents harmful data retention and contributes to the development of better generalizing multi-modal models, addressing a gap left by previous uni-modal or bi-modal studies.