🤖 AI

Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning

This paper introduces FLEX, a challenging 34-task benchmark that exposes the limitations of current expert routing in multimodal continual instruction tuning by removing textual fingerprints and extending task horizons, while proposing a principled class-incremental learning framework that significantly improves routing accuracy and overall performance.

Original authors: Huiyu Yi, Yongqi Xu, Bogang Zhang, Dunwei Tu, Xu Zhiming, Zhen-Hao Xie, Baile Xu, Furao Shen

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Huiyu Yi, Yongqi Xu, Bogang Zhang, Dunwei Tu, Xu Zhiming, Zhen-Hao Xie, Baile Xu, Furao Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to be a master of many different skills, one after another. First, you teach it to recognize cats and dogs. Then, you teach it to solve math problems. Next, you teach it to read medical X-rays. This is called Continual Learning. The tricky part is that as the robot learns new skills, it often forgets the old ones, or it gets confused about which skill to use when you ask it a question.

To fix this, scientists give the robot a "toolbox" of specialized helpers, called LoRA experts. Think of these experts like different chefs in a massive kitchen. One chef is amazing at baking, another at grilling, and another at making sushi. When you order a meal, you need a Router—a smart waiter—to look at your order and send it to the right chef. If the waiter sends a sushi order to the grilling chef, the meal will be a disaster. The big question in this field is: Can our waiter get better at knowing exactly which chef to pick, especially when the restaurant grows to have dozens of chefs and the orders start to look very similar?

The Problem: The Waiter is Relying on Shortcuts (But Only Because the Menu is Too Easy)

A team of researchers from Nanjing University decided to investigate how well these "waiters" (routers) are actually doing their jobs. They looked at the standard tests used to train these robots and found something suspicious: the waiters were getting perfect scores, but they were relying on shortcuts.

In the existing tests, the "menus" (the instructions given to the robot) had obvious clues. For example, a math problem might always start with the phrase "Calculate the sum," while a picture description might always start with "Describe the image." The waiter didn't need to look at the picture or understand the math; it just needed to read the first few words of the instruction to know which chef to pick. The researchers call these clues "textual fingerprints."

Because the tests only had a few tasks (like 6 to 10 chefs) and the menus were so obvious, the routing problem was "saturated." It was like a waiter in a tiny restaurant with only three tables; they could memorize the table numbers without ever actually learning the customers' names. The researchers realized that if we want to build a robot that can learn hundreds of skills over a long time, we need a much harder test where the menus look the same, forcing the waiter to actually understand the content.

The Solution: A New, Tougher Restaurant (FLEX)

To fix this, the team created a new benchmark called FLEX (Fingerprint-reduced Long-horizon Expert eXamination).

Imagine FLEX as a massive, chaotic restaurant with 34 different chefs (tasks) instead of just a few. In this restaurant, every chef uses the exact same menu template. Whether you are ordering a math problem, a medical diagnosis, or a description of a sunset, the instruction looks identical. The only way to know which chef to pick is to actually look at the picture and understand the context.

The researchers also removed the "shortcut" clues. They made sure that the instructions for a chemistry problem didn't look different from an art history problem. This forced the routing system to stop relying on shortcuts and start doing the hard work of figuring out what the task actually is.

The Big Idea: The Waiter is Just a Classifier

Here is the clever twist the paper introduces. The researchers realized that the job of the waiter (the router) is actually the same as a classic machine learning problem called Class-Incremental Learning (CIL).

In CIL, you teach a computer to recognize new categories (like "cats," then "dogs," then "birds") without forgetting the old ones. The researchers showed that picking the right LoRA expert is exactly the same as picking the right category.

Task 1 (Math) = Class 1
Task 2 (Art) = Class 2
Task 34 (Medicine) = Class 34

By viewing the problem this way, they could borrow powerful "waiter training" techniques that had already been invented for CIL. They took four different methods (HC, HC-SOINN, RanPAC, and DDAS) and plugged them into the routing systems of existing robots.

The Results: A Much Smarter Waiter

When they tested these new "plug-in" waiters on the tough FLEX restaurant, the results were impressive:

Better Accuracy: The new routers correctly identified the right chef up to 16.3 percentage points more often than the old ones.
Better Performance: The robots' overall performance (called MacroScore) improved by up to 4.6 points.
Closing the Gap: The new routers managed to recover between 28% and 50% of the gap between the current performance and the "perfect" performance (where the robot always knows the right chef).

However, the paper also found that not all robots benefit equally. Some existing systems (like HiDe-LLaVA) were so designed that even a perfect waiter couldn't improve the final meal much, because the problem was actually in how the chefs cooked, not how the waiter ordered. But for the systems that relied heavily on the waiter (like DISCO and SAME), the improvement was huge.

Why This Matters

This paper suggests that for a robot to truly learn continuously over a long time, we can't just give it easy tests with obvious clues. We need to build environments where the robot has to actually understand the task, not just read the label. By treating the "who do I call?" problem as a "what category is this?" problem, we can use existing, powerful tools to make our AI much more reliable.

The researchers didn't just suggest this; they built the new test (FLEX), proved the old tests were too easy, and showed that swapping in these new routing methods works. They didn't claim to have solved everything—the gap to perfection is still there—but they showed a clear, principled path forward for making multimodal AI that can actually keep learning without getting confused.

Technical Summary: Beyond Routing Saturation in Multimodal Continual Instruction Tuning

Problem Statement

Multimodal Continual Instruction Tuning (MCIT) aims to enable Multimodal Large Language Models (MLLMs) to learn new tasks sequentially while retaining prior capabilities. Recent approaches often employ a "routing–composition–inference" paradigm, where task-specific Low-Rank Adaptation (LoRA) experts are maintained, and a router selects or weights these experts at inference time.

The authors identify a critical, under-explored limitation in current MCIT evaluation: Expert Routing Saturation. On widely used benchmarks (e.g., CoIN, UCIT, TriGap), routing mechanisms achieve near-perfect accuracy (often >96% matching) not because they have learned robust task representations, but due to two confounding factors:

Textual Fingerprints: Superficial instruction patterns (e.g., unique templates, specific fields like "Reference OCR token," or distinct option layouts) allow routers to identify the dataset identity using text alone, bypassing the need for visual understanding or deep semantic analysis.
Short Horizons: Existing benchmarks typically contain only 6–10 tasks. This limits the number of competing LoRA experts, making the task-identification problem trivial even for simple text-dominant routers.

Consequently, the scalability of expert routing in long-horizon, real-world scenarios remains untested, and the difficulty of distinguishing between semantically similar tasks with diverse visual domains is obscured.

Methodology

1. FLEX: A New Benchmark

To expose the true difficulty of expert routing, the authors introduce FLEX (Fingerprint-reduced Long-horizon Expert eXamination).

Scale: FLEX consists of 34 tasks, significantly expanding the expert pool compared to previous benchmarks.
Design: It groups datasets with similar instruction/answer formats but diverse visual and knowledge domains (e.g., 5 ImageNet-200 splits, 2 captioning, 7 multiple-choice, 20 short-answer VQA).
Fingerprint Reduction: The benchmark normalizes outer answer-format templates and removes unique interface cues that act as "shortcuts." This forces routers to rely on genuine multimodal understanding rather than superficial text patterns.
Composition: The tasks span classification, captioning, multiple-choice, and short-answer VQA, ensuring a mix of visual and textual complexity.

2. Theoretical Reformulation: Soft Task-as-Class MCIL

The authors propose a new theoretical perspective: Progressive-LoRA routing can be formulated as Soft Task-as-Class Multimodal Class-Incremental Learning (MCIL).

Mapping: In a progressive LoRA pool, each incoming task $t$ corresponds to a unique LoRA expert $t$ . The routing problem becomes identifying the correct class (task) among an expanding set of $t$ classes.
Soft vs. Hard: Unlike standard MCIL which typically outputs a hard argmax prediction, MCIT usually requires a soft distribution of scores to weight multiple LoRA experts simultaneously.
- Hard Routing: Equivalent to standard incremental classification ( $\alpha_{k} = \mathbb{I}[k = \arg\max]$ ).
- Soft Routing: A continuous relaxation where the complete score distribution serves as mixture weights ( $\alpha_{k} \propto \exp(\tau s_k)$ ).
Implication: This reformulation allows established Class-Incremental Learning (CIL) methods to be directly adapted as "plug-in" routers for MCIT frameworks without retraining the LoRA experts or altering the generation pipeline.

3. PureLoRA Baseline and Router Adaptation

To isolate the effect of routing, the authors introduce PureLoRA, a controlled baseline that:

Trains independent task-specific LoRA experts.
Uses a frozen multimodal prototype router (CLIP-based).
Performs soft LoRA composition.
Contains no shared experts, gating networks, or replay buffers.

The authors then adapt four existing CIL/MCIL methods to serve as routers for PureLoRA, HiDe-LLaVA, DISCO, and SAME:

HC (Hierarchical Clustering): Uses hierarchical cluster prototypes and cosine similarity.
HC-SOINN: Dynamically grows prototype nodes via a self-organizing network.
RanPAC: Uses random projections and a closed-form ridge-regression classifier.
DDAS: Uses task-specific autoencoders and routing scores based on reconstruction error.

Key Results

Routing Saturation on FLEX

Evaluation on FLEX reveals that routing is far from saturated:

Ground-Truth Weight (G): FLEX significantly lowers the macro ground-truth LoRA weight compared to existing benchmarks (e.g., dropping from ~99% on UCIT to ~53–80% on FLEX depending on the method).
Strict Matching (R): Strict task-LoRA matching drops from near-perfect levels to ~64–84%, indicating a genuine task-identification challenge.
Textual Shortcuts: Ablation studies confirm that restoring "strong" textual fingerprints (unique suffixes) on FLEX immediately restores high matching rates, proving that the difficulty in FLEX stems from the removal of these shortcuts.

Performance Improvements

Adapting CIL methods as plug-in routers yields consistent improvements:

Routing Quality: All 12 CIL transfers on routing-sensitive frameworks (DISCO, SAME, PureLoRA) improved both MacroScore (M) and Ground-Truth Weight (G).
MacroScore Gains:
- DISCO: Improved by up to +4.61 points (using DDAS).
- SAME: Improved by up to +4.05 points (using RanPAC).
- PureLoRA: Improved by up to +1.54 points (using RanPAC).
Oracle Gap Recovery: The best routers recovered between 28.3% and 50.2% of the performance gap between the baseline router and the "Oracle" (perfect routing) upper bound.
HiDe-LLaVA Insensitivity: HiDe-LLaVA showed minimal downstream performance gains despite routing improvements. Analysis reveals this is due to its architectural bottleneck: a shared "remain" layer that averages all historical experts regardless of the top-layer routing decision, limiting the impact of improved task identification.

Calibration and Soft Routing

The paper demonstrates that calibrated soft mixtures outperform hard top-1 selection. Even when the top-1 expert is correctly identified, a soft mixture that reuses compatible LoRAs from related tasks yields higher performance (up to +2.01 points over hard routing in PureLoRA).

Significance and Claims

The paper claims three primary contributions:

Diagnosis of Saturation: It reveals that current MCIT benchmarks suffer from "routing saturation" caused by textual fingerprints and short task sequences, masking the true scalability challenges of expert routing.
FLEX Benchmark: It introduces FLEX, a 34-task benchmark with weakened textual fingerprints, which successfully exposes an expanding task-identification problem and provides a rigorous testbed for routing scalability.
Theoretical Bridge: It establishes a direct connection between expert routing and Multimodal Class-Incremental Learning (MCIL). By formulating routing as "soft task-as-class MCIL," the authors demonstrate that established CIL classifiers can be effectively transferred as plug-in routers, substantially improving both routing quality and downstream task performance without modifying the underlying LoRA experts.

The authors conclude that future MCIT systems must address both long-horizon task identification (via methods like those adapted from CIL) and compatibility-aware expert composition (via calibrated soft mixtures) to overcome the limitations of current routing paradigms.