Technical Summary: Hidden Forgetting in Continual Multimodal Learning

Problem Statement

Multimodal Large Language Models (MLLMs) must continually adapt to evolving tasks and domains, such as new visual formats, instructions, and reasoning skills. While standard continual learning (CL) metrics focus on whether old answers remain correct, they largely overlook the stability of multimodal grounding.

The paper identifies a specific failure mode termed Hidden Evidence-Use Forgetting. In this scenario, an MLLM retains high accuracy on previous tasks but silently shifts its reliance from grounded evidence channels (e.g., visual regions, OCR tokens, chart elements, document text) to less grounded shortcuts (e.g., language priors or dataset biases). Standard answer-level evaluation fails to detect this drift, creating a false sense of stability while the model's reasoning process becomes less faithful.

Methodology: Reliance-Constrained Continual Learning (RCL)

To address this, the authors propose RCL, a replay-free continual learning framework that preserves not only what the model answers but how it uses evidence.

Core Mechanism

RCL operates by freezing the previous checkpoint as a behavioral reference ("teacher") and constraining the current model ("student") to maintain the teacher's evidence-reliance profile during adaptation.

1. Counterfactual Channel Interventions:

   • For each input, RCL constructs counterfactual examples by systematically suppressing specific evidence channels (e.g., masking visual regions, removing OCR tokens, or hiding chart elements).
   • The model's sensitivity to these suppressions is measured by the increase in loss or shift in output distribution.
   • These sensitivities are normalized into a reliance vector $r_\theta(x, y)$, representing the probability distribution over which evidence channels the model depends on for a specific sample.

2. Reliance Preservation Objective:

   • The framework jointly optimizes three objectives:
     • Task Learning ($L_{task}$): Standard cross-entropy loss for the current stage.
     • Prediction Preservation ($L_{pred}$): Token-level distillation to ensure the current model produces the same answers as the previous one.
     • Reliance Preservation ($L_{rel}$): A loss term minimizing the Jensen–Shannon Divergence (JSD) between the teacher's reliance vector and the student's reliance vector.
   • A confidence gate weights the reliance loss, down-weighting samples where the teacher's predictions are uncertain or where the teacher shows no measurable dependence on specific channels, preventing the propagation of noisy targets.

3. Implementation Details:

   • RCL is replay-free: It does not store previous multimodal examples.
   • It uses Parameter-Efficient Fine-Tuning (PEFT) (e.g., LoRA), freezing the backbone and updating only adapter parameters.
   • No Inference Overhead: Counterfactual interventions and reliance estimation occur only during training and evaluation. The deployed model has the same architecture and latency as a standard PEFT-adapted MLLM.

Key Contributions

1. Formalization of Hidden Evidence-Use Forgetting: The authors define and quantify a failure mode where answer accuracy is preserved while the underlying evidence path drifts. They introduce metrics such as Modality Reliance Drift (MRD), Dominant Evidence Flip Rate (DEF), and Hidden Forgetting Rate (HFR) to diagnose this phenomenon.
2. RCL Framework: A novel, replay-free framework that constrains evidence-use drift through counterfactual channel interventions without increasing inference costs.
3. Empirical Validation: The paper demonstrates that preserving reliance profiles improves both answer retention and grounding stability, offering a more robust notion of continual multimodal learning than answer accuracy alone.

Experimental Results

The authors evaluated RCL on four benchmarks: CoIN, COAST, MCITlib, and a custom evidence-sensitive multimodal stream (covering VQA, OCR, charts, and documents) using backbones like LLaVA-1.5-7B/13B and InternVL-Chat-7B.

• Performance vs. Forgetting: RCL consistently outperformed replay-free baselines (e.g., Seq-LoRA, EWC, LwF) and recent multimodal CL methods (e.g., HiDe-LLaVA, BranchLoRA, SEFE) in terms of final average accuracy while significantly reducing average forgetting.
• Grounding Stability:
  • RCL reduced Modality Reliance Drift (MRD) by approximately 50% compared to strong baselines (e.g., from ~0.23 to ~0.10).
  • It drastically lowered the Dominant Evidence Flip Rate (DEF), particularly for structured evidence like OCR, charts, and documents.
  • Hidden Forgetting Rate (HFR) was reduced from ~65% (in baselines like Answer-KD) to ~11% with RCL.
• Cost-Efficiency: Despite the training-time cost of counterfactual interventions, RCL achieved near-Pareto optimal performance, offering substantial gains in grounding stability with only a moderate increase in training compute (approx. 1.56x the cost of sequential LoRA).

Significance and Claims

The paper argues that robust continual multimodal learning requires preserving the evidence path behind correct answers, not merely the answers themselves.

• Beyond Accuracy: The authors claim that standard answer-centric evaluation is insufficient for safety-sensitive and long-lived applications, as it can mask a model's drift toward shortcut-based reasoning.
• Replay-Free Efficiency: RCL demonstrates that high-quality reliance preservation is achievable without storing past data or incurring inference-time penalties, making it suitable for real-world deployment.
• Generalizability: The method is shown to be effective across different model scales and architectures, suggesting that evidence-use stability is a fundamental requirement for reliable MLLM adaptation.

The authors conclude that future continual learning systems should be evaluated not just by what they answer, but by how they use multimodal evidence to arrive at those answers.