🤖 machine learning

Generalizing the Geometry of Model Merging Through Frechet Averages

This paper proposes a general framework for model merging based on Fréchet averaging over appropriate manifolds to ensure symmetry invariance, demonstrating its ability to encompass existing methods like Fisher merging and providing a practical algorithm for merging low-rank adapters (LoRA) that addresses the limitations of current approaches.

Original authors: Marvin F. da Silva, Mohammed Adnan, Felix Dangel, Sageev Oore

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marvin F. da Silva, Mohammed Adnan, Felix Dangel, Sageev Oore

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of expert chefs. Each chef has mastered a specific dish: one makes the perfect lasagna, another the ideal sushi, and a third the finest soufflé. Now, you want to create a "Super Chef" who can cook all three dishes perfectly, but you don't have time to train a new chef from scratch. You just want to combine the existing chefs' knowledge.

In the world of AI, this is called Model Merging. You take the "weights" (the internal settings) of different AI models and try to blend them into one.

The Problem: The "Wrong Map"

The paper argues that the old way of doing this is like trying to mix these chefs' recipes by simply averaging their numbers on a piece of paper. This works fine if everyone is writing in the same language and using the same units.

But AI models have a hidden quirk: Symmetry.
Think of it like this: Chef A might write "2 cups of flour," while Chef B writes "1 cup of flour + 1 cup of flour." They mean the exact same thing, but the numbers look different. Or, Chef C might have swapped the order of their ingredients in the notebook, but the taste of the dish is identical.

In AI, there are many ways to write down the exact same "recipe" (model) using different numbers. If you just take the average of the numbers (Naive Averaging), you might end up with a recipe that says "0 cups of flour" because the positive and negative numbers canceled each other out. You've created a broken model that can't cook anything.

The authors call this stepping off the orbit. Imagine the "true" recipe exists on a circular track (an orbit). If you just draw a straight line between two points on the track, you cut across the middle of the circle and end up in the mud (a broken model), rather than staying on the track where the good recipes live.

The Solution: The "Smart Compass" (GeoMerge)

The paper proposes a new method called GeoMerge. Instead of just averaging numbers, they treat the space of AI models like a curved landscape (a Riemannian manifold).

The Map: They realize that because of the symmetry (the different ways to write the same recipe), the "true" map isn't a flat sheet of paper. It's a folded, curved surface where different points actually represent the same thing.
The Compass: They use a mathematical tool called a Fréchet Average. Think of this not as a simple average, but as finding the "center of gravity" on a curved surface.
Alignment: Before they mix the models, they first "align" them. It's like making sure Chef A and Chef B are both using the same measuring cups and writing in the same order. They find the specific version of each chef's recipe that is closest to the others, ignoring the cosmetic differences in how the numbers are written.
The Path: Instead of cutting straight through the mud, they walk along the curved path (a geodesic) that stays on the track of "good recipes."

Why This Matters for "LoRA" (The Specialized Chefs)

The paper focuses heavily on a popular technique called LoRA (Low-Rank Adaptation). Imagine these are not full chefs, but specialized sous-chefs who only tweak a few specific ingredients to make a dish better.

The authors found that when you try to merge these specialized tweaks using old methods, the "symmetry" problem is even worse. The sous-chefs' notes are so flexible that they can be written in thousands of different ways that all mean the same thing.

Old Method (KnOTS): Tries to force the notes into alignment using a rigid, step-by-step math trick (SVD). It works okay, but it's like trying to force square pegs into round holes.
GeoMerge: Recognizes that the notes naturally live on a curved surface. It uses the "Smart Compass" to find the true center of the group without forcing them into a rigid shape.

The Results

The researchers tested this on a large AI model (Llama 3) that had been specialized for different types of language puzzles (Natural Language Inference).

The Outcome: GeoMerge created a "Super Chef" that performed better than the previous best method (KnOTS).
The Efficiency: Even better, GeoMerge did this while keeping the "size" of the recipe small. The old method ended up with a massive, bloated recipe (high rank), while GeoMerge kept it lean and efficient, proving it understood the geometry of the problem better.

In a Nutshell

The paper says: Don't just average the numbers. AI models are like recipes that can be written in many different languages. If you want to merge them, you first need to translate them into a common language (alignment) and then find the center of the group using a map that respects the shape of the world (geometry), rather than a flat ruler. This prevents the "Super Chef" from becoming a broken mess.

1. Problem Statement

Model merging aims to combine multiple trained neural networks into a single model without additional end-to-end training. While simple parameter-space averaging (e.g., Model Soups) works well for full-rank fine-tuned models, it fails significantly when applied to Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA (Low-Rank Adaptation).

The core issue is architectural symmetry:

Neural network parameters are not canonical coordinates; distinct parameter settings can represent the exact same input-output function due to symmetries (e.g., neuron permutations, rescaling, or gauge symmetries in low-rank factorizations).
Naive averaging (Euclidean) ignores these symmetries. It can "step off" the correct geometric orbit, leading to destructive interference where the merged model collapses or performs poorly.
Existing alignment methods (like KnOTS) attempt to fix this by aligning task updates via SVD, but they often rely on specific gauge choices or coordinate-dependent heuristics that do not fully respect the underlying geometry.

2. Methodology: GeoMerge

The authors propose GeoMerge, a framework that reframes model merging as computing a Fréchet mean on a Riemannian manifold.

Core Conceptual Shift

Instead of averaging parameters in Euclidean space, GeoMerge treats the parameter space as a quotient manifold ( $\mathcal{M} = \bar{\mathcal{M}} / G$ ), where $G$ is the symmetry group (e.g., $GL(r)$ for LoRA).

Intrinsic Averaging: The goal is to find a point on the quotient manifold that minimizes the sum of geodesic distances to the input models.
Symmetry Invariance: By operating on the quotient, the merge becomes invariant to the specific parameterization (gauge) of the input models.

Mathematical Formulation

Fréchet Mean: Given models $\theta_1, \dots, \theta_T$ , the merged model $\mu^*$ is the minimizer of the Fréchet functional:
$\mu^* \in \arg\min_{\mu \in \mathcal{M}} \sum_{i=1}^T w_i d_{\mathcal{M}}(\mu, \theta_i)^2$
where $d_{\mathcal{M}}$ is the geodesic distance on the manifold.
Quotient Geometry for LoRA:
- LoRA updates are represented as $\Delta W = GH^\top$ . This factorization has a $GL(r)$ symmetry: $(G, H) \sim (GA^{-1}, HA^\top)$ .
- The authors propose using a Polar Factorization ( $\Delta W = UBV^\top$ ) to map the problem to a product of Stiefel manifolds ($St$) and Symmetric Positive Definite (SPD) manifolds.
- They define specific Riemannian metrics:
  - Affine-invariant metric for the SPD factor ( $B$ ).
  - Canonical Stiefel metric for the orthogonal factors ( $U, V$ ).
Algorithm (Quotient GeoMerge):
The algorithm alternates between two steps to compute the mean:
- Alignment (Orbit Minimization): For a current estimate $\mu_t$ , find the symmetry transformation $G_i$ for each input $\theta_i$ that minimizes the distance to $\mu_t$ . This aligns the inputs to the same "gauge" (horizontal space).
- Intrinsic Averaging: Compute the logarithm map (tangent vectors) from $\mu_t$ to the aligned inputs. These vectors are guaranteed to be horizontal (orthogonal to the symmetry orbits). The mean is updated by moving along the geodesic defined by the weighted sum of these horizontal vectors.

3. Key Contributions

Geometric Unification: The paper establishes that model merging is fundamentally a problem of computing Fréchet means on Riemannian manifolds. It unifies various merging strategies under this geometric lens.
Symmetry-Aware Merging: It provides a principled solution to the "misalignment" problem in PEFT by explicitly modeling the parameter space as a quotient manifold, ensuring the merge is invariant to gauge choices.
Connection to Fisher Merging: The authors demonstrate that Fisher-weighted merging is a tractable approximation of the Fréchet mean under the Fisher-Rao geometry (information geometry), specifically when using a quadratic surrogate and ignoring covariance changes. However, they note that standard Fisher merging still suffers from symmetry issues because the averaging mechanism itself is not symmetry-invariant.
Practical Algorithm for LoRA: They derive a concrete algorithm for LoRA adapters using the polar factorization and specific Riemannian metrics, allowing for efficient computation of geodesics and alignment without data.
Toy Model Analysis: They provide a closed-form analytical solution for a simple scaling-symmetric model, proving that naive Euclidean averaging fails (collapsing to zero) while GeoMerge correctly recovers the intrinsic predictor.

4. Experimental Results

The authors evaluated GeoMerge on Llama-3 8B with LoRA adapters fine-tuned on six Natural Language Inference (NLI) datasets (SNLI, MNLI, SICK, QNLI, RTE, SciTail).

Baseline: They compared against KnOTS, a state-of-the-art method that aligns LoRA updates via joint SVD before merging.
Performance: GeoMerge consistently outperformed KnOTS across all datasets in terms of normalized accuracy (the ratio of merged model accuracy to the best specialist).
- Average Normalized Accuracy: GeoMerge achieved 91.88% vs. KnOTS's 90.38%.
Efficiency: A critical finding is that GeoMerge achieves superior performance with a much lower effective rank.
- KnOTS typically increases the rank of the merged model to $T \times r$ (where $T$ is the number of tasks).
- GeoMerge maintains the original rank $r$ , suggesting it utilizes the weight capacity more efficiently by respecting the manifold geometry rather than expanding the parameter space.

5. Significance and Impact

Theoretical Advancement: The paper shifts the paradigm of model merging from "heuristic parameter averaging" to "geometric optimization on manifolds." It highlights that the choice of geometry (metric and manifold) is the most critical design choice in merging.
Robustness: By addressing symmetries intrinsically, GeoMerge offers a more robust solution for merging specialized models, particularly in the PEFT era where low-rank updates are standard.
Future Directions: The framework opens the door for integrating other geometric concepts (e.g., Riemannian medians, Huber means) and extending to other PEFT methods or full-rank models where symmetries are present. It also suggests that "alignment" is not just a preprocessing step but an integral part of the averaging process on the quotient space.

In summary, GeoMerge solves the fragility of model merging under architectural symmetries by treating the problem as a geometric averaging task on a quotient manifold, resulting in state-of-the-art performance for LoRA merging with improved parameter efficiency.