🤖 machine learning

Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution

MADE-IT is an adaptive continual model merging method that resolves the saturation-redundancy dilemma by using manifold-aware geometry to autonomously evolve experts and a data-free implicit routing mechanism to activate them without additional training.

Original authors: Haiyun Qiu, Xingyu Wu, Kay Chen Tan

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haiyun Qiu, Xingyu Wu, Kay Chen Tan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a "Master Chef" robot.

Initially, you teach it how to make Italian food. Then, you want it to learn Japanese food, then Mexican, then Indian. You have two ways to do this, but both have big problems:

The "One Brain" Problem (Saturation): You try to cram all these new recipes into the same single brain. Eventually, the brain gets too full. To learn how to make Sushi, the robot starts forgetting how to make Pasta. This is called "catastrophic forgetting."
The "Too Many Brains" Problem (Redundancy): You decide to give the robot a new, separate brain for every single recipe. This works, but soon the robot is a giant, clunky mess of thousands of tiny brains. Most of them are useless—for example, the "Tomato Sauce Brain" and the "Pizza Sauce Brain" are basically the same thing, but the robot is treating them as two different organs. Plus, the robot needs a complex "manager" to decide which brain to use for every single bite, which is slow and requires constant retraining.

MADE-IT is a new way to solve this. It’s like giving the robot a "Smart, Growing Skill-Set" instead of just more brains.

Here is how it works using three simple concepts:

1. The "DNA" of a Skill (Manifold-Aware Evolution)

Instead of looking at every single tiny neuron (which is like trying to understand a person by looking at every single atom), MADE-IT looks at the "Shape" of the knowledge.

Think of a skill like a musical melody. You don't need to know the exact vibration of every air molecule to recognize "Twinkle Twinkle Little Star"; you just need to recognize the pattern of the notes. MADE-IT uses math (called SVD and Grassmann manifolds) to find the "melody" of a new task.

When a new task arrives, MADE-IT asks: "Is this melody similar to something we already know?"

If yes (Consolidation): It merges the new melody into an existing one. It realizes "Pizza Sauce" is just a variation of "Tomato Sauce," so it keeps the brain count low.
If no (Creation): It creates a brand-new "skill module." This keeps the robot from getting "full" while preventing it from becoming a bloated mess.

2. The "Intuitive Reflex" (Implicit Routing)

In most systems, you need a "Manager" (a gating network) to look at an ingredient and say, "Okay, use the Japanese Brain now!" But this manager is hard to train and needs constant supervision.

MADE-IT replaces the manager with "Intuition." It uses Feature Projection Alignment.
Imagine you see a piece of raw fish. You don't need a manager to tell you to use your "Sushi Skill"; the very sight of the fish "aligns" with that skill in your mind. MADE-IT does this mathematically: it looks at the incoming data and sees which "skill shape" it fits into most naturally. It’s fast, it’s automatic, and it doesn't need a manager to be retrained.

3. The "Teamwork" Rule (Path Consistency)

To make sure the robot doesn't act crazy, MADE-IT ensures the skills work together in a logical sequence.

If the robot decides to use a "Sushi" skill for the first step of a recipe, it won't suddenly switch to a "Beef Stew" skill for the second step. It follows a "Dependency Graph"—a mental map that says, "If you started with Japanese flavors, stay on the Japanese path until the dish is done." This keeps the robot's "thoughts" consistent from start to finish.

The Result?

In the researchers' tests, MADE-IT was like a chef who gets smarter with every lesson. It:

Learned more tasks without forgetting the old ones.
Stayed slim and efficient by not creating redundant "brains."
Worked faster because it didn't need a slow, heavy "manager" to make decisions.

In short: MADE-IT allows AI to grow its expertise like a human does—by recognizing patterns, building on what it knows, and developing an intuition for new information.

Technical Summary: MADE-IT (Manifold-Aware Dynamic Expert Evolution and Implicit rouTing)

1. Problem Statement

The paper addresses the challenges of Continual Model Merging (CMM)—the process of sequentially integrating task-specific models into a single unified architecture without intensive retraining. Existing CMM methods suffer from a "saturation-redundancy dilemma":

Backbone-centric approaches (e.g., orthogonal projection) attempt to fit new knowledge into a fixed-capacity model, leading to parameter saturation and catastrophic forgetting.
Mixture-of-Experts (MoE) variants avoid interference by adding experts for each task, but they lead to "architectural bloat" (uncontrolled expansion) and rely on explicit routing bottlenecks (parameterized gating networks that require data and training to function).

Furthermore, the authors identify that standard parameter-space metrics (like cosine similarity) are ineffective for measuring expert similarity due to the permutation invariance and basis rotation sensitivity of deep neural networks.

2. Methodology

To resolve these issues, the authors propose MADE-IT, which shifts the focus from raw parameters to the underlying subspace structures using manifold geometry. The method consists of two core components:

A. Manifold-Aware Dynamic Expert Evolution
Instead of treating weights as vectors, MADE-IT treats experts as points on a Grassmann manifold by extracting their principal subspaces via Truncated Singular Value Decomposition (SVD).

Subspace Extraction: For each module, the weight update is approximated by a low-rank expert $E = \tilde{U}\tilde{\Sigma}\tilde{V}^\top$ .
Projection-Based Subspace Affinity: To decide whether to merge or create a new expert, the method uses a metric based on the Hilbert-Schmidt inner product of projection operators. This metric is invariant to basis rotation and quantifies the overlap between subspaces.
Adaptive Evolution: A distribution-aware threshold mechanism ( $\gamma = \mu + \beta\sigma$ $γ = μ + β σ$ ) guides the process:
- Consolidation: If a new expert is highly similar to an existing one, they are merged via subspace merging to prevent redundancy.
- Creation: If the expert is unique, a new expert is added to the architecture to preserve specialization.

B. Implicit Routing Mechanism
To bypass the need for trainable gating networks, MADE-IT uses a data-free and training-free activation strategy:

Feature Projection Alignment (FPA): During inference, the model evaluates how well an input feature aligns with an expert's input subspace by measuring the cosine similarity between the feature and its projection onto that subspace.
Expert Dependency & Path Consistency: To ensure semantic coherence across layers, the method constructs an Expert Dependency Graph. It selects an "anchor" module and propagates task-identity constraints through the graph, ensuring that the activated experts across different layers belong to a logically consistent task path.

3. Key Contributions

Geometric Foundation: Proposes using manifold geometry (Grassmann manifolds) rather than parameter-space metrics to characterize expert representations.
Adaptive Management: Introduces a dynamic evolution strategy that balances architectural parsimony (pruning redundancy) with plasticity (adding specialized experts).
Training-Free Routing: Designs an implicit routing mechanism that uses feature-subspace alignment, eliminating the need for auxiliary gating networks and data-driven optimization.
Hierarchical Discovery: Demonstrates that the model naturally evolves a "generic-to-specific" hierarchy (shared experts in shallow layers, specialized experts in deep layers).

4. Results

The authors conducted extensive experiments using CLIP-ViT backbones across varying task scales (8, 14, and 20 tasks).

Superior Accuracy: MADE-IT consistently outperformed strong baselines (like MINGLE and OPCM) in Average Accuracy (ACC) across all architectures. In the 20-task ViT-B/32 experiment, it outperformed MINGLE by 4.3%.
Robustness: It achieved high Backward Transfer (BWT) scores, indicating minimal catastrophic forgetting compared to other merging methods.
Efficiency: The method significantly reduced expert redundancy, particularly in early layers and generic modules (e.g., achieving up to 95% reduction in some modules), while maintaining high performance.
Metric Validation: Experiments confirmed that the proposed subspace affinity metric is far more discriminative than standard cosine similarity, which tends to collapse in high-dimensional spaces.

5. Significance

MADE-IT represents a significant step toward scalable and autonomous foundation model adaptation. By providing a way to merge models sequentially without needing the original training data or expensive retraining of gating networks, it offers a highly efficient paradigm for deploying large-scale models in streaming, real-world environments where tasks arrive continuously. Its ability to autonomously manage its own architecture (growing only when necessary and merging when redundant) makes it a highly "intelligent" approach to model merging.