Technical Summary: Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

Problem Statement

Deep learning models, particularly overparameterized neural networks, exhibit a phenomenon where independently trained solutions often converge to models with similar predictions and generalization performance, despite residing in highly nonconvex loss landscapes. A key property explaining this is Linear Mode Connectivity (LMC), where two trained solutions can be linearly interpolated in weight space while maintaining low loss.

However, LMC is often obstructed by parameter symmetries, specifically permutation symmetries of hidden units. In standard architectures, permuting hidden units and compensating in adjacent layers leaves the realized function unchanged. Consequently, independent training runs explore different representatives of the same function orbit, making direct linear interpolation fail (high loss barriers) unless a post-hoc alignment (permutation matching) is performed.

Recent work attempted to break these structural symmetries via architectural interventions (e.g., fixing random weights or masks) to achieve unaligned LMC (LMC without alignment). Yet, empirical results have been mixed: some symmetry-breaking methods yield unaligned LMC, while others do not. This paper investigates why structural symmetry breaking is sometimes insufficient and proposes that effective symmetry breaking depends on the interplay between architecture, data geometry, and the identifiability of neurons.

Methodology

1. Theoretical Framework: Effective Function Classes and Realization Costs

The authors develop a theoretical framework centered on effective function classes and realization costs.

• Effective Function Class ($H_i(X)$): For a neuron $i$ in a layer, this is the set of functions it can realize on the input support $X$ (the data distribution or learned representations).
• Realization Cost: The minimum weight norm required for a neuron to realize a specific function $h$. This cost is measured relative to the neuron's effective function class.
• Symmetry Breaking via Identifiability: The paper defines neuron identifiability as the consistent assignment of features to neurons across independent training runs. This occurs when architectural asymmetries bias training toward a unique, low-cost assignment of features to neurons.

The authors model the input support $X$ as lying on a low-dimensional subspace $U$ (Assumption 3.1). They analyze a symmetry-breaking intervention where the effective weight matrix is $W_{eff} = F + D \odot W$, where $F$ (fixed offsets) and $D$ (fixed diagonal masks) are architectural components, and $W$ is trainable.

2. Quantifying Symmetry Breaking

The core of the analysis involves quantifying the realization cost sensitivity to permutations.

• Permutation Sensitivity ($\Delta_{\pi, \tau}$): The change in total realization cost when features are reassigned to different neurons (output permutation $\pi$) or when input coordinates are reindexed (input permutation $\tau$).
• Center-Dominated Regime: The authors identify a regime where the fixed weights $F$ dominate the trainable weights $W$. In this regime, the cost of swapping neurons is primarily determined by the distance between their fixed centers projected onto the input subspace.
• Subspace Coherence ($\nu(U)$): The paper introduces the concept of subspace coherence to measure how aligned the data's principal directions are with the standard basis. High coherence implies that masking specific input coordinates (via $D$) has a pronounced effect on which features are realizable by which neurons.

3. Connection to Linear Mode Connectivity

The paper establishes a link between neuron identifiability and LMC.

• Chord Deviation: The authors analyze the deviation of the network's output along a linear interpolation path between two solutions. They show that if neurons are "center-dominated" (fixed weights dominate), the chord deviation is small ($O(\beta^{3/2})$), implying a low loss barrier.
• Merging without Alignment: If neurons are identifiable (i.e., their roles are fixed by the architecture and data), representations can be merged without explicit alignment, leading to unaligned LMC.

Key Contributions

1. Theoretical Framework for Neuron Identifiability: The paper formalizes neuron identifiability as the consistent assignment of features to neurons across training runs, driven by the minimization of realization costs. It distinguishes between structural symmetry breaking (breaking symmetries in raw parameter space) and effective symmetry breaking (breaking symmetries on the actual input support).
2. Characterization of Realization Costs: The authors derive closed-form expressions for realization costs as Mahalanobis norms dependent on the Gram matrices of the fixed architectural components ($F$ and $D$) projected onto the input subspace.
3. Conditions for Effective Symmetry Breaking:
   • Fixed Weights ($F$): In the center-dominated regime, effective symmetry breaking requires the projected centers of neurons to be sufficiently separated. The paper provides asymptotic bounds showing that this separation depends on the number of neurons ($m$) and the intrinsic dimension of the data ($k$). If $k$ is small relative to $m$, separation is difficult, and symmetries may remain effectively intact.
   • Input Subspace Coherence: The paper demonstrates that even without large fixed weights, high subspace coherence can enable effective symmetry breaking via the diagonal operator $D$, as it makes certain feature directions expensive to realize for specific neurons.
4. Trade-off between Identifiability and Feature Learning: The authors prove a hardness result suggesting that strong symmetry breaking (enforcing consistent neuron roles) restricts the set of features a network can learn within a given norm budget, potentially limiting expressivity compared to a fully symmetric random feature baseline.
5. Unified Explanation of Empirical Phenomena: The framework explains why some symmetry-breaking methods (like W-asymmetric networks with large fixed weights) yield unaligned LMC, while others (like sparse networks without fixed offsets) do not.

Experimental Results

The authors validate their theory through experiments on MLPs and ResNets (and a preliminary Transformer extension) using MNIST and CIFAR-10.

• Effect of Fixed Weight Scale ($\sigma_F$): Experiments sweeping $\sigma_F$ show that for small $\sigma_F$ (effectively symmetric), the activation matching objective (a proxy for identifiability) is low for identity permutations, and unaligned LMC barriers are high. As $\sigma_F$ increases, identity permutations become nearly optimal, and unaligned LMC barriers drop significantly.
• Intrinsic Dimension ($k$): Using synthetic Gaussian mixture data, the authors show that neuron swap costs (a measure of symmetry breaking) are low when the intrinsic dimension $k$ is small relative to the network width, even with symmetry-breaking architectures. As $k$ increases, swap costs rise, and unaligned LMC improves.
• Subspace Coherence: Experiments varying the coherence of the input subspace confirm that high coherence allows diagonal masking ($D$) alone to induce effective symmetry breaking and unaligned LMC, even when fixed weights ($F$) are zero.
• Transformers: Preliminary results on Vision Transformers suggest that while the W-asymmetric mechanism reduces interpolation barriers, the complex symmetries of attention mechanisms (beyond simple permutations) make achieving perfect unaligned LMC more challenging than in MLPs.

Significance and Claims

The paper claims to provide a unifying theoretical lens for understanding when and why parameter symmetry breaking leads to neuron identifiability and linear mode connectivity.

• Beyond Structural Symmetries: The primary significance is the shift from viewing symmetry breaking as a purely structural property of the architecture to an effective property determined by the interaction between the architecture, the data distribution, and the learned representations.
• Mechanism for Unaligned LMC: The work clarifies that unaligned LMC is not guaranteed by simply breaking permutation symmetries in the weight space; it requires the architecture to induce a unique, low-cost assignment of features to neurons that is stable across training runs.
• Role of Data Geometry: The paper highlights that the geometry of the data (specifically subspace coherence and intrinsic dimension) plays a critical role in determining whether a symmetry-breaking intervention is effective.
• Limitations and Trade-offs: The authors modestly note that strong symmetry breaking may come at the cost of reduced feature learning capabilities, as the network is constrained to specific feature-neuron mappings. They also acknowledge that their current framework focuses on permutation symmetries and may need extension for richer symmetry groups (e.g., in Transformers).

In summary, the paper argues that neuron identifiability is the key mechanism linking architectural interventions, data geometry, and the emergence of linear mode connectivity, offering a predictive framework for when unaligned model merging will succeed.