🤖 machine learning

SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

The paper proposes SAME, a stabilized Mixture-of-Experts framework for Multimodal Continual Instruction Tuning that mitigates router and expert drift through orthogonal subspace routing, curvature-aware scaling, and adaptive expert activation, achieving state-of-the-art performance on a new long-sequence benchmark.

Original authors: Zhen-Hao Xie, Jun-Tao Tang, Yu-Cheng Shi, Han-Jia Ye, De-Chuan Zhan, Da-Wei Zhou

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhen-Hao Xie, Jun-Tao Tang, Yu-Cheng Shi, Han-Jia Ye, De-Chuan Zhan, Da-Wei Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a student who is brilliant at solving math problems but, when asked to read a poem, suddenly forgets how to count. This is the core challenge facing modern artificial intelligence systems known as multimodal large language models. These are powerful computers that can see images and read text, learning to answer questions about the world by studying vast amounts of data. They are trained to follow instructions, like "describe the cat in the photo" or "solve this physics problem." However, in the real world, these systems do not learn everything at once. Instead, they encounter new types of tasks one after another, like a student moving from a math class to a history class, and then to an art class. The problem is that as these models learn new skills, they often forget the old ones, a phenomenon scientists call catastrophic forgetting. To keep these models useful, researchers need a way to teach them new things without erasing what they already know.

For some time, experts have tried to solve this by giving the model a team of specialized helpers, a structure known as a mixture of experts. Think of the model as a large office where different workers, or "experts," handle different types of jobs. When a question comes in, a manager, called a router, decides which worker is best suited to answer it. If the question is about a chart, the router sends it to the data analyst; if it is about a poem, it goes to the writer. This system works well when the tasks are stable, but researchers discovered a flaw when the model had to learn a long sequence of new tasks. As the model learned new skills, the manager began to make mistakes, sending old types of questions to the wrong workers. Simultaneously, the workers themselves started to change, losing the specific skills they had developed for earlier tasks. This double failure meant the model was not only sending questions to the wrong people but was also forgetting how to do the jobs those people were supposed to do.

A team of researchers has now proposed a new method called SAME to fix these issues. They found that the manager's confusion and the workers' memory loss were two separate problems that needed different solutions. To stop the manager from getting confused, the researchers designed a system that carefully tracks how the model sees the world. Instead of letting the manager change its mind completely with every new lesson, they restrict how much it can shift its focus. They allow the manager to learn new ways of sorting questions but protect the old ways of sorting that were essential for previous tasks. This ensures that a question about a medical image, for instance, continues to go to the medical expert, even after the model has learned to read maps or solve chemistry problems.

To prevent the workers from forgetting their jobs, the researchers introduced a way to measure how much a worker's new learning might hurt their old skills. They looked at the types of data the workers had seen in the past and used that history to slow down changes that would damage those old skills. It is like telling a worker, "You can learn this new skill, but do it in a way that doesn't erase how you used to fix the old machines." By doing this, the model preserves the specific abilities it gained from earlier tasks while still being able to adapt to new ones. Additionally, the researchers found that not every worker needs to be active for every lesson. They created a rule to temporarily pause the workers who are not needed for the current task, which saves energy and prevents them from being accidentally changed by irrelevant information.

The team tested this new approach on a series of challenging benchmarks, including a new set of tasks they created that mimics the messy, varied reality of the real world. This new test included ten different types of tasks, ranging from reading medical documents and analyzing scientific charts to understanding complex road scenes and chemical formulas. The results showed that their method, SAME, was significantly better at remembering old tasks than previous methods. In one specific test involving a long sequence of eight tasks, the new method maintained a much higher level of accuracy on the very first task compared to other approaches. It also proved more efficient, requiring less time and computer memory to train because it only updated the necessary workers.

Perhaps most importantly, the researchers observed that their method prevented a subtle but common type of failure where the model would get the right answer but format it incorrectly. For example, if a task required an answer in capital letters, the model might start giving the answer in lowercase letters after learning a new task that used lowercase. The new system kept the model consistent, ensuring it followed the specific formatting rules of each task without being confused by the style of the next one. By stabilizing both the decision of who answers the question and the memory of the person answering it, this work offers a more reliable way for artificial intelligence to learn continuously, expanding its capabilities without losing the foundation of what it already knows.

Technical Summary: SAME – Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

1. Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated strong generalization through instruction tuning. However, real-world deployment requires these models to continually expand their capabilities by learning sequential streams of multimodal tasks (Multimodal Continual Instruction Tuning, or MCIT). A primary challenge in this setting is catastrophic forgetting, where learning new tasks degrades performance on previously learned ones.

Recent approaches utilize Mixture-of-Experts (MoE) architectures combined with LoRA (Low-Rank Adaptation) to promote task specialization via sparse routing. Despite their promise, the authors identify that existing MoE-based MCIT methods suffer from two distinct but coupled sources of forgetting:

Router Drift: As the data distribution evolves with new tasks, the router's decision-making process becomes unstable. Inputs from earlier tasks that previously activated specific "localization" or "grounding" experts are increasingly reassigned to irrelevant experts. This leads to inconsistent access to the correct functional modules.
Expert Drift: Even if the router correctly identifies the relevant expert, the expert's internal parameters are overwritten by updates from new tasks. This causes the expert to lose its original functionality (e.g., formatting conventions or specific reasoning capabilities) acquired during earlier training stages.

The authors demonstrate through diagnostic experiments that forgetting persists even when the router is re-trained to match the original task distribution, confirming that the degradation of the experts themselves is a critical failure mode.

2. Methodology: SAME

The authors propose StAbilized Mixture-of-Experts (SAME), a framework designed to mitigate both router and expert drift without requiring data rehearsal (storing past data). SAME consists of three core components:

2.1 Spectral-Aware Routing (Addressing Router Drift)

To stabilize expert selection, SAME decomposes the routing dynamics into orthogonal subspaces based on the input covariance history.

Mechanism: The method maintains an uncentered covariance matrix of the router's hidden inputs. Through Singular Value Decomposition (SVD), it separates the update space into a signal subspace (high-energy directions relevant to new tasks) and an approximate null space (low-energy directions associated with historical tasks).
Update Strategy: Router gradients are projected onto these subspaces. Updates in the signal subspace are scaled using a sliding-window average of singular values to prevent aggressive changes in directions critical for old tasks. Updates in the null space are projected to ensure they have negligible impact on historical inputs ( $V_\perp^T x_{old} \approx 0$ ).
Outcome: This ensures that routing decisions for old tasks remain consistent while allowing adaptation for new tasks.

2.2 Curvature-Aware Scaling (Addressing Expert Drift)

To prevent experts from being overwritten, SAME regulates expert updates using the geometry of historical inputs.

Mechanism: Instead of minimizing a hard constraint on past data (which is unavailable in rehearsal-free settings), the method approximates the historical input geometry using the accumulated covariance matrix ( $C_{t-1}$ ).
Optimization: The authors formulate a constrained optimization problem where the functional degradation (expected squared output change on historical inputs) is bounded. This leads to a Riemannian gradient update where the Euclidean gradient is preconditioned by the inverse of the historical covariance matrix ( $C_{t-1}^{-1}$ ).
Implementation: To handle computational costs, the inverse is approximated using a damped pseudo-inverse of the low-rank factors retained from the spectral decomposition. This downweights updates along directions that were heavily relied upon by previous tasks, preserving their functionality.

2.3 Adaptive Expert Activation

To further reduce interference and computational cost, SAME introduces a task-level freezing mechanism.

Mechanism: During the training of a specific task, experts are evaluated based on two signals:
1. Current Utilization ( $U$ ): How frequently the expert is activated for the current task.
2. Historical Importance ( $F_{pre}$ ): A proxy for historical importance derived from the trace of the Average Gradient Outer Product (AGOP), estimated via routing-weighted input energy.
Freezing Rule: Experts with low current utility but high historical importance are temporarily frozen (both forward and backward propagation are skipped) during the current task's training. They are reactivated for subsequent tasks.
Outcome: This reduces redundant computation and prevents unnecessary parameter drift in experts that are critical for past tasks but not needed for the current one.

3. Key Contributions

Diagnosis of Drift: The paper empirically isolates and defines router drift and expert drift as the primary causes of forgetting in MoE-based MCIT, demonstrating that both must be addressed simultaneously.
Stabilized Framework (SAME): The introduction of a rehearsal-free framework that combines spectral-aware routing, curvature-aware Riemannian scaling, and adaptive expert activation to stabilize the MoE adaptation process.
New Benchmark (TriGap): The authors introduce TriGap, a new benchmark designed to stress-test MCIT models under realistic conditions. Unlike existing benchmarks (CoIN, UCIT), TriGap features:
- Extended Sequence: 10 sequential tasks (vs. 6–8 in others).
- Domain Span: Diverse domains including medical, scientific, remote sensing, and document understanding.
- Scale Gap: Highly imbalanced dataset sizes per task (10k–40k samples) to simulate real-world data streams.
Efficiency: The method reduces training time and GPU memory usage by freezing low-utility experts, achieving significant speedups without sacrificing accuracy.

4. Experimental Results

The authors evaluated SAME on three benchmarks: CoIN, UCIT, and the new TriGap.

Performance: SAME consistently outperforms state-of-the-art baselines (including MoELoRA, HiDe-LLaVA, CL-MoE, and ProgLoRA).
- On TriGap, SAME achieved an average accuracy of 46.53%, outperforming the best baseline (MoELoRA) by +2.08%.
- On CoIN, SAME reached 66.82% average accuracy, surpassing the previous best by a significant margin.
- On UCIT, SAME achieved 67.12% average accuracy.
Forgetting Analysis: SAME demonstrated the lowest average forgetting ( $\bar{F} = -4.23\%$ ) compared to MoELoRA ( $-19.04\%$ ) and HiDe-LLaVA ( $-6.38\%$ ).
Formatting Stability: A specific analysis on the ScienceQA task revealed that baseline methods often suffer from "format-induced forgetting" (e.g., switching from uppercase to lowercase answers). SAME successfully preserved task-specific output formatting across the entire sequence.
Efficiency: Adaptive expert activation reduced training time by an average of 32.1 minutes per task and GPU memory usage by 2.3 K MiB/GPU.

5. Significance and Claims

The paper claims that SAME provides a robust solution for Multimodal Continual Instruction Tuning by addressing the root causes of forgetting in MoE architectures. Its significance lies in:

Rehearsal-Free Operation: It achieves strong retention without storing or replaying past data, making it suitable for privacy-sensitive or storage-constrained environments.
Scalability: The use of low-rank covariance approximations and adaptive freezing allows the method to scale to long task sequences and large models without prohibitive computational overhead.
Generalization: The stabilized expert selection and preservation of historical knowledge enable the model to retain generalizable capabilities, as evidenced by improved Out-of-Distribution (OOD) performance on the MMMU benchmark.

The authors conclude that by stabilizing both the routing mechanism and the expert parameters, SAME enables MLLMs to continually learn new vision-language capabilities while reliably preserving previously acquired knowledge.