🤖 machine learning

Unlocking the Potential of Continual Model Merging: An ODE Perspective

This paper proposes ODE-M, a novel continual model merging framework that leverages an Ordinary Differential Equation perspective to trace low-loss paths in parameter space, thereby effectively mitigating catastrophic forgetting and outperforming existing methods on heterogeneous task benchmarks.

Original authors: Lihong Lin, Haidong Kang

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lihong Lin, Haidong Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a master chef who has learned to cook a perfect Italian pasta dish. Then, you hire a second chef who is a master of Japanese sushi. Finally, you hire a third chef who specializes in Mexican tacos.

The Problem:
In the world of AI, we often have "foundation models" (like our master chefs) that are good at many things. When a new task comes along (like learning to cook sushi), we usually have to retrain the whole chef from scratch or try to force the new knowledge on top of the old.

The paper talks about Continual Model Merging. This is like trying to combine the skills of the Italian chef, the sushi chef, and the taco chef into one "Super Chef" who can do all three perfectly, without ever having to retrain them from scratch.

The Old Way (The "Jump" Method):
Previous methods tried to merge these chefs by taking a snapshot of the Italian chef's brain and the sushi chef's brain, and then simply averaging them together.

The Flaw: Imagine the Italian chef's brain is in a valley (low loss/good performance) and the sushi chef's brain is in another valley. If you just draw a straight line between them, you might have to walk up a steep, rocky mountain in between.
The Result: When the AI tries to walk that straight line, it hits a "loss barrier" (the mountain). It forgets how to make pasta because it got stuck on the mountain trying to learn sushi. This is called catastrophic forgetting. The more chefs you add, the more the Super Chef forgets the original skills.

The New Idea (The "ODE" Method):
The authors propose a new way called ODE-M (Ordinary Differential Equation-driven Merging). Instead of jumping from one chef to another, they imagine a continuous path or a smooth road connecting the two brains.

Here is how they do it, using simple metaphors:

The Smooth Road (Mode Connectivity):
The paper assumes that even though the chefs look different, there is a hidden, smooth, low-lying road connecting their brains where they can both still cook well. The goal is to find this road, not the straight line over the mountain.
The GPS with a Speed Limit (The Velocity Field):
The authors create a "velocity field," which is like a GPS guiding the Super Chef from the Italian style to the Sushi style.
- The Problem: Sometimes, the GPS tries to drive the car up a steep hill (which increases the "loss" or makes the cooking worse).
- The Solution: The system checks the terrain in real-time. If the road ahead starts going up a hill (increasing loss), the system applies a "brake" or a "dampener" to that specific direction. It slows down the part of the movement that causes forgetting, while still letting the car move forward toward the new skill.
The "Stop Sign" (Operating Point):
You don't always have to drive all the way to the destination (the new chef's brain). Sometimes, you only want to learn some sushi skills without losing your pasta skills.
- The system allows you to stop the car at any point along the smooth road.
- If the new task is very important, you drive further (closer to the new chef).
- If the old tasks are more important, you stop earlier (keeping more of the old knowledge).
- This is controlled by a "time schedule," which acts like a dial deciding how much of the new skill to absorb versus how much of the old skill to keep.

Why is this better?

No Mountain Climbing: By following the smooth, low-loss path, the AI doesn't have to climb the "mountain" that causes it to forget old skills.
Control: It gives the user a dial to decide exactly how much new knowledge to add without breaking the old knowledge.
Results: In their tests (using AI models that recognize images like cars, flowers, and traffic signs), this method created a "Super Chef" that was better at remembering all the tasks than any previous method. It kept the pasta skills while learning the sushi, even when adding many new recipes one by one.

In Summary:
Instead of forcing two different AI brains together with a blunt instrument (which breaks things), this paper uses a smooth, guided path to merge them. It acts like a careful navigator that steers the AI around "danger zones" (where it would forget things) and lets you decide exactly how far to travel down that path to balance old and new skills.

Problem Formulation

The paper addresses Continual Model Merging (CMM), a setting where task-adapted models arrive sequentially, and a single deployed model must be updated iteratively to incorporate new capabilities without access to original training data. While standard model merging combines multiple models simultaneously, CMM faces the challenge of catastrophic forgetting and inconsistent performance allocation.

Existing CMM methods typically treat each merge as a fixed algebraic operation (e.g., weight averaging or task vector arithmetic) mapping a current model $\Psi_k$ and an incoming model $\psi_{k+1}$ directly to a new endpoint $\Psi_{k+1}$ . The authors argue this "endpoint-only" view suffers from two critical deficiencies:

Lack of Path Control: It ignores the trajectory taken in parameter space. If the update passes through high-loss regions (loss barriers), the resulting model may suffer functional degradation or interference with previously learned behaviors.
Inflexible Stability-Plasticity Trade-off: It provides limited control over how much historical knowledge is preserved versus how much new task expertise is adopted. This is particularly problematic in heterogeneous utility scenarios, where tasks have different frequencies or importance weights; fixed rules often fail to allocate capacity appropriately, leading to suboptimal system-level performance.

Methodology: ODE-Driven Merging (ODE-M)

Motivated by mode connectivity—the observation that independently trained models can often be connected by low-loss paths—the authors propose viewing continual merging not as a discrete jump, but as a controlled continuous transition in parameter space. They introduce ODE-M, a framework that generates a connecting path via a time-dependent velocity field.

1. Trajectory Perspective and Loss Barriers

Instead of linear interpolation, ODE-M constructs a continuous path $\theta(t)$ from the current model $\theta_0$ to the incoming model $\theta_1$ . The goal is to minimize the loss barrier along this path, defined as the maximum excess loss encountered relative to the linear interpolation of endpoint losses.

2. Rectified Velocity Field

The core mechanism is the construction of a barrier-aware velocity field.

Base Field: A base vector field $u_t(\theta) = \alpha(t)(\theta_1 - \theta)$ is defined to drive the model toward the target.
Gradient Decomposition: At each step, the velocity is decomposed into a component aligned with the loss gradient ( $u_\parallel$ ) and a component orthogonal to it ( $u_\perp$ ).
Adaptive Rectification: An adaptive factor $\gamma(\theta, t) \in [0, 1]$ $γ (θ, t) \in [0, 1]$ modulates the gradient-aligned component:
- If the movement toward the target decreases or maintains the loss ( $\langle g_t, u_t \rangle \le 0$ ), the full velocity is applied ( $\gamma=1$ ).
- If the movement increases the loss ( $\langle g_t, u_t \rangle > 0$ ), the component is damped to ensure the loss increase stays within a prescribed budget ( $\Delta L = L(\theta_1) - L(\theta_0)$ ). Specifically, $\gamma = \text{clip}(\frac{\Delta L}{\langle g_t, u_t \rangle}, 0, 1)$ .
Integration: The trajectory is generated by integrating the rectified velocity field $v_t(\theta) = u_\perp + \gamma u_\parallel$ using Euler integration on a small calibration set.

3. Utility-Aware Time Scheduling

To control the stability-plasticity trade-off, the method selects an operating time $t_k$ along the trajectory rather than integrating to the full endpoint ( $t=1$ ). The time $t_k$ is determined by the relative utility of the incoming task:
$t_k = \frac{w_k}{\sum_{i=1}^k w_i}$
where $w_k$ is the utility weight of the $k$ -th task. In a uniform setting, this reduces to $t_k = 1/k$ , naturally diminishing the contribution of new tasks as the stream grows, thereby preserving more historical knowledge.

Key Contributions

Trajectory Formulation: The paper reframes CMM as a path-searching problem, extending the concept of loss barriers from linear interpolation to general connecting paths.
Barrier-Aware Dynamics: It proposes a novel ODE-driven merging strategy using a rectified velocity field that regulates loss-increasing motion via lightweight first-order feedback, avoiding high-loss regions without requiring full retraining.
Explicit Control: By selecting operating points along the trajectory based on utility weights, the method provides an explicit mechanism to balance retained historical knowledge and incoming task expertise.

Experimental Results

The authors evaluated ODE-M on CLIP ViT backbones (ViT-B/32, ViT-B/16, ViT-L/14) across task streams of length 8, 14, and 20, using datasets from the FusionBench suite.

Performance: ODE-M achieved state-of-the-art performance compared to strong baselines (including Continual Task Arithmetic, Continual Ties-Merging, and OPCM) in terms of average accuracy (ACC) and backward transfer (BWT). For instance, on ViT-B/32 with 8 tasks, ODE-M improved ACC from 75.5 (OPCM) to 79.6.
Heterogeneous Utility: Under non-uniform task importance, ODE-M consistently achieved the highest utility-weighted accuracy ( $ACC_w$ ), demonstrating its ability to optimize for specific task priorities better than methods optimized for uniform macro-averages.
Robustness: Ablation studies showed the method is robust to the size of the calibration set (performing well with as few as 512 samples) and the numerical integration step size.
Efficiency: The computational overhead per merge is practical (e.g., ~124s for ViT-B/32), dominated by the numerical integration and gradient computations on the calibration set, which is negligible compared to full retraining.

Significance and Claims

The paper claims that ODE-M unlocks the potential of continual model merging by shifting the paradigm from fixed algebraic updates to controlled continuous transitions. By explicitly constructing paths that respect the loss landscape and regulating the speed of integration based on task utility, ODE-M effectively mitigates forgetting and handles heterogeneous task importance.

The authors position this work as a step toward more scalable and controllable model customization, reducing the engineering and computational overhead of maintaining multi-task systems. They acknowledge limitations, such as the requirement for a calibration set to compute gradients and the current focus on vision classification tasks, noting that future work will explore larger language models and more efficient integration strategies.