🤖 machine learning

Connecting Independently Trained Modes via Layer-Wise Connectivity

This paper proposes a new empirical algorithm that reliably connects independently trained neural network modes across a broader range of modern architectures and training hyperparameters, overcoming the limitations of existing methods that are restricted to traditional models.

Original authors: Yongding Tian, Zaid Al-Ars, Maksim Kitsak, Peter Hofstee

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yongding Tian, Zaid Al-Ars, Maksim Kitsak, Peter Hofstee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have built two incredibly smart robots (neural networks) to solve a puzzle. You trained them separately, starting from scratch with different random seeds, like two different chefs following the same recipe but using different ingredients from the start.

Surprisingly, when both robots are finished, they are both excellent at the task. But here is the mystery: Are they actually the same robot underneath? Or are they just two different solutions that happen to work well?

In the world of AI, the "brain" of a robot is a massive list of numbers (parameters). If you plot all possible versions of this brain, the "good" ones (the ones that solve the puzzle well) sit in a deep, low valley called the low-loss region. The "bad" ones sit on high, rocky mountains.

The Problem: The "Lost in the Valley" Mystery

For a long time, scientists found that if you take two good robots and try to walk straight from one to the other in this number-space, you often hit a high mountain peak in the middle. The path breaks, and the robot stops working.

Previous methods tried to fix this by:

Wiggling the path: Trying to bend the line to go around the mountain.
Only connecting twins: Only trying to connect robots that were already very similar (like twins separated at birth).

The problem was that these old methods were unreliable. They worked on simple, old-school robot designs (like basic CNNs) but failed miserably on modern, complex ones (like MobileNet or EfficientNet). Sometimes, they would claim a path existed, but it would actually be a dead end.

The Solution: The "Layer-by-Layer" Elevator

The authors of this paper propose a new method called LLPF (Low-Loss Path Finding). They realized that trying to walk from one robot to another in one giant step is too hard. Instead, they decided to move layer by layer.

Think of a neural network like a multi-story building:

Layer 1 is the lobby (seeing edges and shapes).
Layer 2 is the first floor (recognizing eyes and noses).
Layer 3 is the second floor (recognizing whole faces).

The old methods tried to move the entire building at once. The new method says: "Let's just move the lobby first, make sure it's still stable, then move the first floor, then the second."

The Magic Trick: The Variance Sphere
The paper introduces a clever concept called a Variance Sphere. Imagine that every time you train a robot, its brain settles into a specific "size" or "spread" of numbers.

If you train two robots with the same settings, they land on spheres of roughly the same size.
The new algorithm ensures that as it moves from Robot A to Robot B, it keeps the "size" of the brain constant at every step. It prevents the robot from shrinking or exploding in size, which keeps it from falling off the cliff.

What They Discovered

Using this new "elevator" approach, the authors found that:

It works on almost everything: They successfully connected robots built with modern, complex architectures (like EfficientNet, RegNet, and even Transformer-based models) that previous methods couldn't touch.
It's reliable: If you run the experiment 100 times with different random seeds, you get the exact same smooth path every time. Old methods were like a coin flip; this new method is like a train on a track.
It bridges different training styles: They even managed to connect a robot trained with "heavy" settings (high weight decay) to one trained with "light" settings. This is like connecting a robot built for a heavy truck to one built for a bicycle, proving they are actually part of the same continuous landscape.

The Big Picture

The paper doesn't claim this will immediately make your phone faster or cure diseases. Instead, it solves a fundamental geometry puzzle.

It suggests that the "low-loss region" (where all good AI models live) isn't a bunch of isolated islands separated by oceans. Instead, it's likely one giant, connected continent. Even if two models look totally different on the surface, there is a smooth, safe road connecting them, provided you know how to drive it layer by layer.

This gives scientists a new map to navigate the complex world of AI, proving that different training runs aren't just lucky accidents—they are all part of the same continuous journey.

Technical Summary: Connecting Independently Trained Modes via Layer-Wise Connectivity

1. Problem Statement

The paper addresses the challenge of constructing continuous, low-loss paths between independently trained neural network modes (i.e., models trained to convergence with different random seeds). While empirical studies have established that such paths exist (a phenomenon known as mode connectivity), existing methods suffer from significant limitations:

Reliability: Prior algorithms, such as Fast Geometric Ensembling (FGE) and AutoNEB, often fail to connect truly independent modes. FGE is restricted to modes that are already close in weight space due to implementation constraints, while AutoNEB exhibits high variability in path quality, frequently failing to maintain low loss across different random seeds.
Architectural Scope: Existing methods have been evaluated primarily on simple or dated architectures (e.g., LeNet, basic CNNs, VGG, ResNet). Their effectiveness on modern, complex architectures (e.g., MobileNet, EfficientNet, Transformers) remains unclear.
Hyperparameter Sensitivity: Current approaches generally assume modes are trained with identical hyperparameters. The ability to connect modes trained under different hyperparameters (which place them on different "variance spheres" in parameter space) has not been robustly addressed.

2. Methodology: Low-Loss Path Finding (LLPF)

The authors propose Low-Loss Path Finding (LLPF), an empirical algorithm designed to construct low-loss paths between independent modes. The method is grounded in the hypothesis of layer-wise mode connectivity, positing that while two modes may not be linearly connected in the full parameter space, they can be connected layer-by-layer.

LLPF consists of two complementary algorithms:

A. Algorithm 1: Model-to-Model (LLPF M2M)

This algorithm connects two modes ( $P_0$ and $D$ ) located on similar variance spheres (i.e., trained with identical hyperparameters).

Layer-Wise Processing: Unlike global interpolation, LLPF processes layers sequentially. The authors propose a Follow Data Flow (FDF) strategy, processing layers from shallow to deep (or handling parallel modules individually) to navigate the high-dimensional manifold effectively.
Iterative Steps:
1. Move: The current point $P_i$ is moved slightly toward the destination $D$ using a weighted average.
2. Variance Correction: The moved point is projected back onto the original variance sphere ( $S_{var=v}$ ). This step counteracts the "vanishing variance problem," where averaging uncorrelated networks reduces parameter variance, hindering subsequent training.
3. Refinement (Train): The corrected point undergoes a limited number of training steps ( $r$ ) to reduce the loss below a threshold.
4. Projection: The refined point is projected back to the variance sphere to serve as the starting point for the next iteration.
Constraints: The optimizer excludes regularization (e.g., weight decay) and momentum, as the variance sphere constraint inherently preserves weight variance, and momentum is less suitable for small, localized updates.

B. Algorithm 2: Model-to-Origin (LLPF M2O)

This algorithm connects a low-loss model to the origin of the parameter space ( $O$ ), enabling traversal between different variance spheres.

Mechanism: Similar to Algorithm 1 but without the Variance Correction step.
Angle Conformal: As the model moves toward the origin, the variance of the weights decreases. To prevent the model from deviating from the low-loss path, the learning rate is adaptively scaled based on the current variance (Angle Conformal step).
Exclusions: Normalization layers are excluded from this process, as moving them toward the origin would impair their functionality (rescaling to zero mean/unit variance).

C. Connecting Different Variance Spheres

To connect modes trained with different hyperparameters (e.g., different weight decay), the authors propose a two-step hybrid approach:

Use Algorithm 2 to move the source mode to an intermediate point on the target variance sphere.
Use Algorithm 1 to connect this intermediate point to the destination mode.

3. Key Contributions

Generalization to Modern Architectures: LLPF successfully connects modes across a broad range of architectures, including MobileNet, ShuffleNet, EfficientNet, RegNet, Deep Layer Aggregation (DLA), and Compact Convolutional Transformers (CCT), which were not reliably supported by prior methods.
High Reproducibility: The method demonstrates strong consistency. Across multiple runs with different random seeds, LLPF consistently discovers paths with nearly identical training loss and test accuracy trajectories, a property absent in previous work like AutoNEB.
Cross-Hyperparameter Connectivity: The framework supports connecting modes trained with different hyperparameters (different variance spheres), a setting previously unevaluated in the context of mode connectivity.
Improved Path Quality: LLPF achieves lower worst-case training loss along the path compared to existing methods (e.g., reducing peak loss from >0.5 in AutoNEB to <0.05 in LLPF for ResNet18).

4. Experimental Results

The authors evaluated LLPF on CIFAR-10, CIFAR-100, and ImageNet-10 datasets using diverse architectures.

Consistency: Experiments repeated 10+ times showed that the constructed paths maintain low training loss (<0.1) and high accuracy throughout the transition.
Layer-Wise Dynamics: The results confirm that the layer-wise approach is necessary; moving all layers simultaneously fails for complex models like ResNet18 and DLA.
Continuity: Linear interpolation between discrete points generated by LLPF maintains low loss, confirming the continuity of the constructed path.
Sharp Minima: The algorithm successfully connects modes located in "sharp" minima (trained with zero weight decay and constant learning rate), demonstrating robustness beyond flat minima.
Failure Cases of Competitors: In specific cases where AutoNEB failed to connect independent modes (resulting in high loss), LLPF succeeded with low loss.

5. Significance and Claims

The paper claims that LLPF provides a robust, practical tool for navigating the low-loss region of the parameter space.

Geometric Insight: The empirical findings suggest that the loss landscape contains "low-loss tunnels" connecting independent minima, challenging the view of isolated basins. The authors hypothesize that all independently trained SGD modes may lie within a single, path-connected low-loss manifold.
Theoretical Limitations: The authors modestly note that while their empirical results support the existence of these paths, no existing theoretical framework formally proves this conjecture for complex architectures.
Practical Utility: By enabling the systematic exploration of the low-loss region, LLPF allows researchers to gain insights into the geometric properties of neural network solutions, potentially aiding in the development of better architectures and training efficiencies.

The source code for the proposed method is made available to the community.