🤖 machine learning

Curvature-Guided Mixing for MLLM Adaptation

This paper proposes Curvature-Guided Mixing (CGM), a theoretically grounded framework that utilizes second-order Hessian approximations to analytically derive optimal parameter mixing ratios, effectively balancing task specialization and general knowledge retention in Multimodal Large Language Models while mitigating catastrophic forgetting.

Original authors: Jinglong Yang, Jiaxuan He, Wenjian Huang, Zhan Zhuang, Jianguo Zhang

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinglong Yang, Jiaxuan He, Wenjian Huang, Zhan Zhuang, Jianguo Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Amnesia" of Smart Models

Imagine you have a brilliant, well-read librarian (the Pre-trained Model) who knows everything about history, science, art, and literature. They are a general expert.

Now, you want to train this librarian to become a world-class expert in cooking (the Fine-tuning task). You take them to culinary school. They learn recipes, knife skills, and flavor profiles incredibly well.

However, there's a catch: When they graduate, they have forgotten how to talk about history or science. They are now a great chef, but a terrible librarian. In the world of AI, this is called Catastrophic Forgetting. The model learns the new skill but loses its old, general knowledge.

The Old Solutions: Guessing and Randomness

Previously, scientists tried to fix this by:

Blending: Taking a random mix of the "Chef" model and the "Librarian" model. (Like mixing two soups and hoping the flavor is right).
Heuristics: Using simple rules like "keep the weights that changed the least." (Like saying, "Don't touch anything that didn't move much").

The paper argues these methods are like guessing. They don't have a solid mathematical reason for why they are mixing things the way they are, and they often fail to keep the balance right.

The New Solution: Curvature-Guided Mixing (CGM)

The authors propose a new method called Curvature-Guided Mixing (CGM). To understand this, imagine the "knowledge" of the model as a landscape of hills and valleys.

The Valley: The bottom of a valley is where the model makes the fewest mistakes (lowest loss).
The Shape: Some valleys are wide and flat (easy to walk around without falling). Others are narrow and sharp (like a knife edge; if you move even a tiny bit, you fall off a cliff).

The Analogy of the "Curvature":
Imagine the "Cooking" knowledge is a narrow, sharp valley. If you move away from the center, you lose your cooking skills immediately.
Imagine the "General Knowledge" is a wide, flat valley. You can move around a bit, and you still know your history.

How CGM Works (The "Soft Mix"):
Instead of blindly mixing the two models, CGM looks at the shape of these valleys for every single piece of the model's brain (every parameter).

If a specific piece of the brain is in a sharp valley for Cooking, CGM says: "Keep the Cooking version of this piece! It's too sensitive to change."
If a piece is in a sharp valley for History but flat for Cooking, CGM says: "Keep the History version! It's critical for the old skills."
It calculates a perfect "mixing ratio" for every single piece of the brain based on how "sharp" or "flat" the terrain is for that specific piece.

This creates a Soft Mix: A new model that is a perfect blend, keeping the sharp skills for the new task and the flat skills for the old tasks.

The "Hard Mix" (CGM†): The Surgeon's Approach

The authors realized that sometimes, mixing everything (even the parts that don't need changing) is risky. It's like trying to fix a watch by sanding down every single gear.

So, they created CGM† (the "Hard Mix").

The Strategy: Instead of blending, this acts like a surgeon. It looks at the "sharpness" scores and decides: "We will keep the Cooking version for these specific parts, and revert (go back to) the History version for these other parts."
The Result: It only changes a tiny, critical percentage of the model (e.g., 10%). It leaves the vast majority of the model exactly as it was in the "Librarian" state, only swapping out the specific parts needed for the "Chef" skills.

What Did They Find?

The paper tested this on two famous AI models (LLaVA and Qwen) with different tasks (like answering questions about images or writing captions).

Better Balance: Their method (CGM and CGM†) consistently did a better job than previous methods. It kept the model good at the new task without making it forget its old general knowledge.
Efficiency: The "Hard Mix" (CGM†) was very efficient. They found that they only needed to change about 10% of the model's parameters to get the best results. The other 90% stayed exactly as they were in the original, smart model.
Structure: When they looked at which parts the model changed, it wasn't random. It changed specific, structured groups of data (like specific columns in a spreadsheet), proving that the "curvature" math actually found the right structural components to keep or change.

Summary

Think of CGM as a master chef who knows exactly how much salt to add to a soup based on the temperature of the kitchen (the curvature).
Think of CGM† as a master surgeon who knows exactly which single nerve to touch to fix a problem without cutting the rest of the body.

Both methods use the "shape" of the learning process to ensure that when an AI learns something new, it doesn't forget who it used to be.

Technical Summary: Curvature-Guided Mixing for MLLM Adaptation

Problem Statement

Multimodal Large Language Models (MLLMs) possess broad, general-purpose knowledge acquired through web-scale pre-training. However, adapting these models to specialized downstream tasks via fine-tuning often results in catastrophic forgetting, where the model loses its foundational general capabilities in favor of new task-specific skills. Existing solutions, such as heuristic model merging (e.g., Spider) or parameter-efficient fine-tuning, often fail to effectively balance task adaptation with knowledge retention. Many rely on hand-crafted scoring functions without a solid theoretical foundation or optimize solely for downstream performance, neglecting the preservation of pre-trained knowledge.

Methodology

The authors propose Curvature-Guided Mixing (CGM), a theoretically grounded framework that merges pre-trained ( $w_{pt}$ ) and fine-tuned ( $w_{ft}$ ) models by analyzing the geometry of their respective loss landscapes.

1. Loss Landscape Approximation

The method models the local geometry of the loss functions for both pre-training ( $\ell_{pt}$ ) and fine-tuning ( $\ell_{ft}$ ) tasks using a second-order Taylor expansion around their optimal weights.

Assumptions: The gradients at the converged optima are zero ( $\nabla \ell = 0$ ), and the merged solution lies within the joint low-loss region, allowing higher-order terms to be neglected.
Approximation: The loss is approximated as a quadratic function involving the Hessian matrix ( $H$ ), which captures the local curvature. To ensure computational tractability, the method utilizes the diagonal entries of the Hessian matrices ( $h_{pt}$ and $h_{ft}$ ), reducing complexity from $O(d^2)$ to $O(d)$ .

2. CGM: Soft Mixing (Closed-Form Solution)

The framework formulates a joint optimization objective to minimize the weighted sum of losses for both tasks. By parameterizing the merged weights as a linear interpolation ( $w_i = w_{ft,i} + \lambda_i(w_{pt,i} - w_{ft,i})$ ), the authors derive a closed-form optimal mixing ratio ( $\lambda_i^*$ ) for each parameter:

$\lambda_i^* = \frac{\alpha h_{pt,i}}{h_{ft,i} + \alpha h_{pt,i}}$

Where:

$h_{pt,i}$ and $h_{ft,i}$ are the diagonal Hessian entries (curvatures) for the pre-training and fine-tuning tasks, respectively.
$\alpha$ is a hyperparameter balancing the two objectives.
Interpretation: Parameters with high pre-training curvature (sharp loss landscape for general tasks) are retained closer to the pre-trained values, while those with high fine-tuning curvature are retained closer to the fine-tuned values. This "soft mixing" is a dense update that blends all parameters.

3. CGM†: Hard Mixing (Sparse Selection)

Recognizing that dense updates can be disruptive, the authors introduce CGM†, a "hard mixing" variant that reframes the problem as a sparse parameter selection task.

Mechanism: Instead of blending weights, the method makes a binary decision for each parameter: either keep the fine-tuned weight ( $w_{ft}$ ) or revert to the pre-trained weight ( $w_{pt}$ ).
Scoring: A curvature-aware score $c_i$ is derived to rank parameters:
$c_i = \frac{1}{2}(h_{ft,i} - \alpha h_{pt,i})\Delta_i^2$
where $\Delta_i = w_{pt,i} - w_{ft,i}$ .
Selection: The method reverts the top $K\%$ of parameters with the lowest $c_i$ scores (indicating they are critical for general knowledge but less critical for the new task) back to their pre-trained values. This imposes an implicit $L_0$ constraint, ensuring minimal, targeted modification.

Key Contributions

Theoretically Grounded Framework: The paper introduces CGM, which derives an optimal mixing rule directly from a joint optimization objective using second-order loss landscape information, moving beyond heuristic scoring.
Sparse Hard Mixing (CGM†): A robust variant that treats merging as a sparse selection problem, using a novel curvature-aware score to identify and revert only the most critical parameters for general knowledge retention.
Empirical Validation: Extensive experiments on LLaVA-1.5-7B and Qwen-2.5VL-3B across multiple downstream tasks (OKVQA, Flickr30k, LaTeX-OCR) demonstrate that CGM and CGM† consistently outperform existing baselines (Tailor, DARE, Grafting, Wanda) in balancing task specialization and general knowledge retention.

Experimental Results

Performance Metrics: The authors evaluate using Hscore (harmonic mean of target task performance and pre-trained average) and Avg (arithmetic mean).
Key Findings:
- On LLaVA-1.5 fine-tuned on OKVQA, CGM† achieved the highest Hscore (62.8) and maintained a high Pre-Avg (65.7), significantly outperforming the Fine-tuned Model (Hscore 53.9) and other merging baselines.
- On Qwen-2.5VL, the proposed methods consistently achieved superior trade-offs, often recovering near-original pre-trained performance on general benchmarks while maintaining strong task-specific scores.
- Ablation Studies: Removing either the fine-tuning curvature or pre-training curvature from the scoring function led to suboptimal results (either overfitting or excessive forgetting), confirming the necessity of the joint curvature formulation.
- Hyperparameter Sensitivity: The sparsity ratio $K$ was found to be robust; even with only 10% of parameters reverted, general knowledge was largely preserved. The balancing coefficient $\alpha$ primarily influenced target task performance, with optimal balance found at small values ( $\approx 0.1$ ).
- Selection Structure: Visual analysis of selection masks revealed that CGM† selects coherent, structured parameter groups (vertical bands in attention layers), whereas baselines like Magnitude and Tailor produce diffuse, unstructured noise.

Significance and Claims

The paper claims that Curvature-Guided Mixing provides a principled solution to catastrophic forgetting in MLLMs by leveraging the geometry of loss landscapes. By analytically deriving mixing ratios based on relative curvatures, the method avoids the pitfalls of heuristic approaches. The authors assert that their work establishes a new state-of-the-art in model merging, demonstrating that second-order information can be effectively utilized to create models that are both highly specialized for new tasks and robust in their foundational general intelligence. The work suggests that the "soft mixing" derived from joint optimization and the "hard mixing" sparse selection are complementary strategies for effective MLLM adaptation.