💻 computer science

Continual Video-MLLM Adaptation over Evolving Domains

This paper proposes Distribution-Aware Expert Routing (DAER), a parameter-efficient framework that employs domain-isolated lightweight experts with intra- and inter-domain routing mechanisms to enable Video-MLLMs to adapt to continuously evolving domains while preventing catastrophic forgetting and cross-domain interference.

Original authors: Rui Cheng, Meixing Shi, Yuxiang Cai, Jingcai Guo, Jianwei Yin, Zhi Chen

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rui Cheng, Meixing Shi, Yuxiang Cai, Jingcai Guo, Jianwei Yin, Zhi Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to understand the world through videos. This robot, a "Video Multimodal Large Language Model" (Video-MLLM), is like a brilliant student who has already read every book and watched every movie ever made. It knows how to answer questions about a cat chasing a laser pointer or a car driving through a storm. But here's the catch: in the real world, new types of videos don't just appear all at once; they arrive one by one, day after day. Maybe today it's traffic cameras, tomorrow it's cartoons, and the next day it's physics experiments.

The problem is that when you teach this robot something new, it often forgets what it learned yesterday. This is called "catastrophic forgetting." It's like a student who studies for a math test, passes it, but then immediately forgets how to read because they are so focused on the new math. Previous attempts to fix this involved trying to make the robot learn everything in one big, shared brain, but that caused too much confusion, or they tried to save old videos to show the robot later, which takes up too much memory. Scientists wanted a way for the robot to learn new things continuously without losing its old skills or needing a massive library of old videos.

Enter a new team of researchers who came up with a clever solution called DAER (Distribution-Aware Expert Routing). Think of the robot's brain as a giant, frozen library that never changes. Instead of rewriting the books in the library, DAER builds a set of small, specialized "study groups" or "expert teams" that sit outside the library. Each team is dedicated to a specific type of video, like a "Traffic Team," a "Cartoon Team," or a "Science Team."

When a new video arrives, the system doesn't just guess which team to use. Instead, it uses a special "detective" tool called MMD (Maximum Mean Discrepancy). Imagine this tool as a scent-sniffer that can smell the "vibe" of the video. If the video smells like a busy highway, the sniffer directs it to the Traffic Team. If it smells like a funny cartoon, it goes to the Cartoon Team. This ensures that the Traffic Team only learns about cars and never gets confused by cartoons, and vice versa. This keeps the learning clean and prevents the robot from mixing up its memories.

But what if the robot doesn't know the name of the video category when it's watching it? No problem. The system has a second detective that looks at the video in a special "discriminative space" (a fancy way of saying it looks at the video's unique fingerprint) to figure out which expert team should take over. Even better, if two types of videos are very similar (like two different kinds of traffic videos), the system can gently merge their study groups to save space, so the robot doesn't get bloated with too many teams.

The researchers tested this idea on a tough challenge: teaching the robot ten very different types of video questions, ranging from "What is the traffic like?" to "Why is this magic trick funny?" They used two powerful robot brains to test their method. The results were impressive. On one of the strongest robot brains (InternVideo2.5-8B), their method boosted the average accuracy from 64.38% to 67.59%. Even more importantly, it stopped the robot from forgetting old lessons. While other methods made the robot forget about 1.81% of what it knew, DAER actually reduced forgetting to -0.14%, meaning the robot remembered everything perfectly and even got slightly better at old tasks.

The team also showed that their method is much more efficient. While a previous top method needed 302.30 million trainable parameters and took 1.46 seconds to process a video, DAER only needed 139.54 million parameters and 1.33 seconds. This means the robot learns faster, uses less energy, and doesn't need a massive computer to run.

In short, the paper suggests that by keeping the main brain frozen and using smart, specialized teams that are routed by the "scent" of the data, we can teach AI to learn continuously in a changing world without losing its mind. The researchers found that this approach works better than trying to share a single brain or just replaying old videos, offering a stable and efficient way for video robots to grow smarter every day.

Technical Summary: Distribution-Aware Expert Routing (DAER) for Continual Video-MLLM Adaptation

1. Problem Statement

Video Multimodal Large Language Models (Video-MLLMs) have demonstrated strong capabilities in video understanding, yet their adaptation to sequentially evolving domains remains underexplored. In real-world deployments, video data arrives continuously from heterogeneous domains (e.g., traffic scenes, educational cartoons, abstract reasoning) with significant shifts in visual style, scene composition, and reasoning demands.

The core challenge is domain-incremental continual learning: the model must acquire new domain-specific knowledge without overwriting previously learned capabilities (catastrophic forgetting) or suffering from severe cross-domain interference. Existing approaches face specific limitations:

Regularization-based methods struggle under large domain shifts.
Replay-based methods incur high memory costs and raise privacy concerns.
Prompt-based and Adapter-based methods often rely on shared adaptation spaces or soft parameter partitioning, which are insufficient to prevent interference among highly heterogeneous domains.
Task Identity: A practical difficulty is the absence of explicit task identifiers at inference time, making it hard to select the correct adaptation module.

2. Methodology: Distribution-Aware Expert Routing (DAER)

The authors propose DAER, a parameter-efficient framework that combines strict parameter isolation with distribution-aware routing. The framework maintains a frozen pretrained Video-MLLM backbone to preserve general multimodal knowledge, while confining all incremental learning to domain-specialized, lightweight LoRA-style expert pools.

Key Components:

Domain-Isolated Expert Pools:
Instead of a shared adapter, DAER allocates a dedicated set of lightweight experts for each task. During the learning of task $t$ , parameters from previous tasks remain frozen (unless merged), ensuring gradient updates for the current task do not shift the parameter space of older tasks.
Intra-Domain Distribution-Aware Routing (IntraDAR):
To enable fine-grained specialization within a domain, DAER uses a non-parametric routing mechanism based on Maximum Mean Discrepancy (MMD).
- It maps input hidden states to a domain representation space.
- It maintains an expert-level prototype reservoir for each expert.
- Routing decisions are made by matching the input representation to these reservoirs using kernel-based similarity (approximating MMD minimization). This activates the expert whose modeled distribution best matches the current input, rather than relying on a parametric gating network.
Inter-Domain Distribution-Aware Routing (InterDAR):
To address the lack of task identifiers at inference, DAER performs task prediction via prototype matching.
- It maintains a domain-level prototype reservoir for each task, aggregating video and text semantic representations.
- Matching occurs in a discriminative subspace (learned via two-stage projection to remove redundant low-variance directions and enlarge inter-domain margins) rather than the raw feature space.
- The model predicts the task ID by finding the domain with the highest prototype affinity, then activates the corresponding expert pool.
Adaptive Domain Merging (ADM):
To prevent the parameter count from growing linearly with the number of tasks, DAER introduces a merging strategy. It quantifies distribution discrepancy between the current task and historical tasks using MMD between domain-level prototype reservoirs. If the discrepancy falls below a threshold $\tau$ , the current task's expert parameters are averaged with those of the most similar historical task.
Two-Stage Optimization Strategy:
To stabilize learning and avoid cold-start instability:
- Warm-up Stage: An auxiliary gating network initializes the prototype reservoirs and establishes a stable routing space without relying on the MMD routing mechanism.
- Main Training: The full DAER routing mechanism is activated, and expert parameters are optimized end-to-end using standard cross-entropy loss.

3. Experimental Setup

Benchmark: A domain-incremental benchmark constructed from 10 VidQA datasets covering diverse environments (MSVD, MSRVTT, Traffic, Pororo, SVQA, Physics, Magic, Humor, Creative, Abstraction).
Backbones: Evaluated on two strong Video-MLLMs: InternVideo2.5-8B and VideoLLaMA3.
Metrics: Average Accuracy ( $A_t$ ) and Backward Forgetting ( $BWF_t$ ).
Baselines: Compared against Zero-Shot, Sequential Fine-tuning, Regularization-based (LwF, EWC), Prompt-based (L2P, CODA-Prompt, S-Prompts), and Adapter-based (MoE-Adapters, DAM) methods.

4. Key Results

Performance: On InternVideo2.5-8B, DAER achieved an average accuracy of 67.59%, outperforming the strongest prior baseline (DAM) by 3.21% (64.38%).
Forgetting: DAER reduced backward forgetting to -0.14 (indicating slight improvement on old tasks), compared to DAM's 1.81 and other baselines ranging from 2.48 to 4.24.
Generalization: On VideoLLaMA3, DAER improved average accuracy from 61.97% (DAM) to 63.79% while reducing forgetting from 1.93 to 0.02.
Robustness: In a 20-task setting (splitting domains into sub-tasks), DAER maintained an average accuracy of 65.89% with a negative forgetting score (-0.47), significantly outperforming DAM.
Efficiency: DAER achieved higher accuracy with 139.54M trainable parameters, compared to DAM's 302.30M. This represents a 53.84% reduction in trainable parameters and a 92.67% reduction in GFLOPs, while maintaining comparable latency.

5. Significance and Claims

The paper claims that DAER provides an effective and scalable solution for continual Video-MLLM adaptation under evolving domains. Its significance lies in:

Decoupling Adaptation: Successfully separating domain-specific adaptation from general multimodal knowledge by freezing the backbone and using isolated experts.
Distribution-Aware Routing: Demonstrating that non-parametric, MMD-based routing (both intra- and inter-domain) is superior to parametric gating for handling heterogeneous domain shifts without explicit task IDs.
Scalability: Proving that adaptive domain merging allows the model to handle long task streams without unbounded parameter growth.
State-of-the-Art: Establishing new performance benchmarks for domain-incremental continual VidQA, showing that combining domain-isolated expert adaptation with distribution-aware routing yields strong stability and accuracy.

The authors emphasize that their approach addresses the open problem of robust domain-specific adaptation without explicit task identity while largely preserving the zero-shot capabilities of the pretrained model.