🤖 machine learning

Regularizing modality contribution drift in multimodal continual learning

This paper identifies and quantifies "Modality Contribution Drift" (MCD) as a critical issue in multimodal continual learning where the relative importance of modalities shifts across tasks, and proposes a novel regularization framework (CMCDR) with both replay-based and replay-free variants to preserve modality contribution structures and mitigate forgetting.

Original authors: Zhen Zhang, Jielei Chu, Bin Liu, Tianrui Li

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhen Zhang, Jielei Chu, Bin Liu, Tianrui Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to understand the world. You don't just show it pictures; you give it a full sensory experience: a video of a dog barking, a recording of a siren, or a photo of a sunset with a description. This is called multimodal learning. The robot learns to combine these different "senses" (like sight and sound) to make smart decisions. But here's the tricky part: the world is constantly changing. New types of dogs, new sounds, new questions appear every day. This is continual learning. The robot has to keep learning new things without forgetting the old ones.

Think of it like a student taking a series of exams. First, they learn to identify animals. Then, they learn to identify vehicles. A smart student shouldn't forget how to spot a cat just because they are now studying trucks. In the past, scientists tried to solve this by making sure the robot's "brain" (its internal representation) stayed consistent. They wanted the robot to see a cat and a truck in a way that didn't get messy. But this paper suggests that keeping the brain's internal map stable isn't enough. The real problem is how the robot uses that map to make a final decision. Does it rely mostly on what it sees? Or does it listen more? If the robot learns a new task that requires it to listen more, it might accidentally start ignoring the visual clues it used to rely on for old tasks. This paper calls that shift Modality Contribution Drift. It's like a student who, after learning a new subject, suddenly starts guessing the answers to old math problems based on a hunch they never used before, leading to mistakes.

The researchers, Zhen Zhang and their team from Southwest Jiaotong University, noticed that current methods were missing this specific type of forgetting. They found that even when a robot's internal understanding of images and sounds stayed the same, the way it weighted those inputs to make a choice was drifting. To fix this, they invented a new method called Continual Modality Contribution Drift Regularization (CMCDR).

Think of CMCDR as a strict "decision coach" for the robot. When the robot learns a new task, this coach checks: "Hey, remember how you used to solve that old problem? You relied on a specific mix of senses to get it right. Don't change that recipe just because you learned something new." The method works in two ways. If the robot has a memory bank of old examples (replay-based), the coach tests the robot on those old examples to make sure it's still using the right mix of senses. If the robot doesn't have old examples (replay-free), the coach uses the new examples as a test bed, asking the robot to pretend it's solving the old problems and ensuring it doesn't drift from its original strategy.

The team tested this idea on several challenging datasets, including audio-visual tasks like recognizing sounds in videos and answering questions about images. They found that simply adding their "coach" to existing methods made a huge difference. For example, on a dataset called AVE, adding CMCDR to a standard method called iCaRL boosted the robot's average accuracy from 64.20% to 71.90% and cut the forgetting rate in half (from 25.60% down to 17.40%). Even more interestingly, they showed that CMCDR works alongside other methods that try to keep the robot's internal brain stable. It's like having a coach who fixes the decision-making process while another coach keeps the brain's memory organized; together, they work better than either one alone.

The paper suggests that this "drift" is a real and distinct problem that previous methods didn't fully address. They proved through experiments that just keeping the robot's internal features stable (making sure it sees a cat the same way) doesn't guarantee it will make the same decision (relying on sight vs. sound). By directly regularizing the contribution of each sense, they showed that robots can learn new things without losing their old decision-making habits. This isn't just a small tweak; it's a new principle for how we teach machines to learn continuously, ensuring they don't just remember facts, but remember how they figured things out in the first place.

Technical Summary: Regularizing Modality Contribution Drift in Multimodal Continual Learning

1. Problem Definition: Modality Contribution Drift (MCD)

Multimodal Continual Learning (MMCL) aims to learn emerging knowledge from sequential multimodal data while preserving previously acquired knowledge. Existing methods primarily focus on mitigating catastrophic forgetting by aligning cross-modal representations or preserving semantic similarity across incremental stages. However, the authors identify a critical, underexplored failure mode: Modality Contribution Drift (MCD).

While representation alignment ensures that feature spaces remain stable, it does not guarantee that the relative contributions of individual modalities (e.g., audio, visual) and their interactions to decision-making remain stable. Different tasks often rely on distinct modality dependencies (e.g., a road-crossing task relies heavily on visual cues, while a question-answering task requires joint audio-visual reasoning). When a model learns a new task with different dependency patterns, it can alter the decision-level contribution structure of previously learned tasks, leading to forgetting even if the underlying representations appear stable.

The authors formalize MCD as a shift in the contribution-strength (magnitude of benefit) and relative-reliance (distribution of evidence across modalities) of previously learned tasks. They demonstrate theoretically and empirically that conventional continual learning mechanisms (like replay and distillation) and representation-stability-based methods (like cross-modal alignment) do not necessarily prevent this drift.

2. Methodology: Continual Modality Contribution Drift Regularization (CMCDR)

To address MCD, the authors propose Continual Modality Contribution Drift Regularization (CMCDR), a framework designed to preserve the modality contribution structure of previously learned tasks while allowing adaptation to new dependencies.

Core Mechanism: Intervention-Based Estimation
The method relies on an intervention-based diagnostic estimator. By systematically removing subsets of modalities (replacing them with absence symbols $\perp$ ) and measuring the resulting change in predictive benefit (log-odds margin), the model computes a contribution profile. This profile decomposes the full-input benefit into specific terms for individual modalities and their interactions using a Möbius transform.

Two Variants of CMCDR
The framework adapts to different MMCL settings regarding the availability of old exemplars:

Replay-Based CMCDR:
- Context: Old samples are available.
- Mechanism: The current model is compared against a frozen previous model on stored old samples.
- Constraint: The method minimizes the distance between the current model's contribution profile and the frozen model's profile for the same ground-truth labels. This constrains changes in both modality-specific and interaction contributions.
Replay-Free CMCDR:
- Context: No old samples are available.
- Mechanism: Current-task samples are used as "probes" to query the old-task decision space.
- Constraint: The current model is regularized to match the frozen previous model's contribution responses for old-class logits. To avoid noise from weak teacher responses, the contribution profiles are weighted by the frozen model's full-modality probability over the old-class output space.

Loss Function
The regularization loss ( $L_{cmcdr}$ ) combines two terms:

Absolute Matching: Minimizes the difference in raw contribution vectors.
Relative Matching: Minimizes the difference in normalized (scale-adjusted) contribution vectors to preserve the relative reliance patterns across modalities.

3. Key Contributions

Problem Identification: The paper formally defines and quantifies Modality Contribution Drift (MCD), identifying it as a distinct cause of forgetting in MMCL that persists even when cross-modal representations are aligned.
Theoretical Insight: The authors provide theoretical proofs (Propositions 1 and 2) demonstrating that:
- Sufficiently large negative contribution shifts can exhaust the retention margin, causing forgetting.
- Representation stability (constraining feature maps) is insufficient for contribution stability because the decision map can still shift independently of the feature map.
Methodological Proposal: The introduction of CMCDR, a plug-and-play regularizer available in both replay-based and replay-free versions, which explicitly constrains the evolution of modality contribution structures.
Complementarity: The work demonstrates that CMCDR addresses a failure mode distinct from representation alignment, offering complementary benefits when combined with existing methods.

4. Experimental Results

The authors evaluated CMCDR on multimodal class-incremental learning (AVE, Kinetics-Sounds, UESTC-MMEA-CL) and continual visual question answering (VQAv2, Split-AVQA, Split-MUSIC-AVQA).

Performance Gains: CMCDR consistently improved average accuracy (gains of 2.13% to 7.70%) and reduced average forgetting (reductions of 0.85% to 11.55%) across various baselines (e.g., LwF, EWC, iCaRL, AV-CIL).
Drift Reduction: The method significantly reduced the MCD score, indicating better preservation of the original modality reliance structures.
Complementarity: When combined with representation alignment methods (RepAlign, CrossSDC), CMCDR provided further improvements in accuracy and forgetting reduction, confirming that representation stability and contribution stability are complementary objectives.
Class-wise Analysis: The results showed that classes with higher baseline MCD (higher initial drift) benefited most from CMCDR, exhibiting stronger drift reduction and accuracy preservation.

5. Significance and Claims

The paper claims that modality contribution stability is a necessary, complementary principle to representation stability in Multimodal Continual Learning. The authors argue that preserving cross-modal geometry alone is insufficient because it does not constrain how modalities and their interactions support individual decisions.

By introducing a diagnostic metric (MCD) and a regularization framework (CMCDR), the work provides a mechanism to explicitly manage the evolution of modality dependencies. The authors position CMCDR as a generalizable solution that improves performance across diverse MMCL benchmarks and settings, motivating future research into scalable coalition approximations for systems with many modalities. The work does not claim to solve all MMCL problems but specifically targets the gap between representation stability and decision-level contribution stability.