Technical Summary: Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference

Problem Statement

Large Language Models (LLMs) fine-tuned via Low-Rank Adaptation (LoRA) typically rely on point estimates (Maximum A Posteriori, MAP) or single-mode variational inference, which fail to provide principled estimates of epistemic uncertainty. While Deep Ensembles (DE) offer a discrete multi-mode approach, recent findings in the LoRA regime suggest they offer little benefit over single-mode methods, contradicting broader deep learning observations where ensembling independent optima improves generalization. A critical gap exists in understanding whether the LoRA weight space contains continuous, low-loss valleys connecting these independent optima. If such valleys exist, they could enable Bayesian Model Averaging (BMA) over a continuous manifold, potentially capturing functional diversity missed by both local variational methods and discrete ensembles.

Methodology: LoRA-Curve

The authors introduce LoRA-Curve, a framework that parameterizes a continuous, segmented path through the LoRA weight space using Bézier curves. This approach constructs low-loss valleys to connect independent optima.

Core Parameterization

The method defines a continuous path $t \mapsto \zeta_\vartheta(t)$ within the LoRA parameter space $\Theta$, where the updated weights are $W_\vartheta(t) = W_0 + B_\vartheta(t)A_\vartheta(t)$. The curve is constructed using Bernstein basis polynomials over control points $\vartheta = \{\theta_0, \dots, \theta_{m+1}\}$.

Two primary configurations are proposed:

1. Free LoRA-Curve (FLC): All control points (anchors and intermediate handles) are jointly optimized from scratch. The initialization ensures $B_i A_i = 0$ (recovering the base model $W_0$) at all points, and the entire curve is trained to minimize the expected loss along the path.
2. Anchored LoRA-Curve (ALC): The "anchor" points (segment boundaries) are frozen to weights of independently fine-tuned LoRA models (e.g., from a Deep Ensemble). Only the intermediate "handle" points are optimized to connect these fixed anchors via a smooth, low-loss trajectory.

Segmentation and Regularization

To navigate highly non-convex loss landscapes, the framework employs segmented Bézier curves. Instead of a single high-degree polynomial, the path consists of multiple segments connected at anchor points. This allows for sharp bends around loss barriers while maintaining $C^0$ continuity.

To prevent the curve from collapsing into a single point estimate and to encourage functional diversity, the authors introduce a Jensen–Shannon Divergence (JSD) regularizer. This penalty maximizes the divergence between predictive distributions at distant points along the curve ($t_1$ and $t_2$), forcing the trajectory to explore diverse functional regions of the loss valley.

Theoretical Properties

The paper establishes two key theoretical properties:

• Continuity: The output distribution $p(y|x, W_\vartheta(t))$ is continuous with respect to the curve parameter $t$. This contrasts with discrete ensembles, which exhibit jumps between modes.
• Regularity: The loss restricted to the curve is Lipschitz continuous and piecewise continuously differentiable. Changes in loss are controlled by the norm of the full loss gradient and the speed of the curve through parameter space.

Inference Strategy

For inference, the authors approximate the posterior predictive distribution via a grid approximation along the optimized curve. Crucially, they employ uniform averaging (equivalent to infinite temperature scaling, $T \to \infty$) rather than standard Bayesian weighting. The authors argue that standard weighting would lead to "double-conditioning" on the training data (since the curve was already optimized on $D$), collapsing the predictive diversity.

Key Results

Experiments were conducted using Qwen2.5 7B across reasoning and classification benchmarks (ARC-Challenge, ARC-Easy, OBQA, boolq, Winogrande).

1. Existence of Low-Loss Valleys:

   • Linear interpolation between independent optima encounters distinct loss barriers (peaks in negative log-likelihood).
   • Both FLC and ALC successfully navigate these barriers, constructing continuous low-loss valleys. FLC, by dynamically shifting anchors, often finds smoother paths than ALC on training data.

2. Functional Diversity and Performance:

   • ALC vs. Discrete Ensembles: ALC matches or outperforms Deep Ensembles (DE) of equivalent size in terms of Log-Likelihood and Accuracy. Crucially, ALC bridges the predictive gaps between discrete modes, providing smooth transitions in class probabilities that DE cannot offer.
   • FLC vs. ALC: While FLC discovers smooth valleys on training data, it exhibits a generalization gap on certain datasets (e.g., winogrande_m) compared to ALC. This suggests that jointly optimizing all points may lead to optimization dynamics that do not align perfectly with the test landscape, whereas ALC's reliance on pre-trained anchors provides a more robust inductive bias.
   • Segmentation: Multi-segment curves consistently outperform single-segment curves with an equivalent number of control points, confirming the necessity of flexible, non-linear routing.

3. Impact of Regularization:

   • Flat-LoRA: Combining the curve framework with Flat-LoRA perturbations (injecting noise to find wider minima) successfully forces the continuous paths into broader regions of the loss landscape, improving generalization.
   • JSD Regularization: Applying the JSD regularizer significantly increases Mutual Information (MI) between the curve parameter and the prediction, indicating higher functional diversity, without sacrificing predictive performance (Log-Likelihood).

Significance and Claims

The paper claims that the space between independent LoRA optima is not a high-loss barrier but a rich, navigable region of low loss. By constructing continuous paths through this space, LoRA-Curve:

• Unifies Extremes: It bridges the gap between single-mode variational inference and discrete ensembles, offering a continuous manifold for BMA.
• Captures Functional Diversity: It explicitly links continuous parameter-space traversal to measurable increases in mutual information, capturing diversity that discrete methods only partially exploit.
• Provides Interpretability: The continuous nature of the curve allows for the interpretation of how changes in low-rank weights translate to smooth changes in predictive probabilities.

The authors conclude that while FLC offers an efficient optimization strategy for discovering valleys, ALC provides a more robust mechanism for connecting known high-performing optima. The work establishes that continuous mode connectivity is a viable and effective strategy for probabilistic LoRA, provided that appropriate regularization (flatness and diversity) and inference strategies (uniform averaging) are employed.

Limitations

The authors acknowledge several limitations:

• The analysis is restricted to one-dimensional, globally coupled trajectories. Extending to higher-dimensional surfaces or independent curves per adapter degrades interpretability and yields no empirical gains in preliminary tests.
• FLC Generalization: The Free configuration shows a generalization gap compared to Anchored configurations on specific datasets.
• Computational Bottleneck: While the 1D case allows for uniform averaging to bypass posterior weight computation, higher-dimensional parameterizations would reintroduce the prohibitive cost of evaluating the full posterior.