Technical Summary: Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

1. Problem Statement

The paper addresses the Parameter Placement Problem within Low-Rank Adaptation (LoRA). While recent works have demonstrated that LoRA adapters can function with drastically reduced parameter counts (e.g., shared random bases or spectral parameterizations), these methods implicitly assume that performance depends primarily on the quantity of trainable parameters rather than their identity.

The authors formalize the question: Given a fixed budget of $k$ trainable entries within the $B$ matrix of a LoRA adapter (where $A$ is frozen and initialized randomly), does the specific choice of which $k$ entries to train significantly impact downstream performance? The study isolates parameter selection from parameter count by keeping $A$ frozen and restricting training to a subset of $B$.

2. Methodology

2.1 Parameter Placement Strategy

The authors propose a gradient-based selection procedure to identify the most critical parameters before training begins.

• Initialization: $B$ is initialized to zero, and $A$ is drawn from a random distribution.
• Scoring Phase: The model performs $N$ forward-backward passes on the training distribution. Since $B=0$, the model's output matches the pre-trained model, and the gradients $\nabla_B L$ directly measure the utility of each $B$ entry for reducing loss.
• Scoring Functions: Two metrics are defined for each element $(i, j)$ of $B$:
  1. Gradient Magnitude Score ($\hat{S}$): Estimates the magnitude of the population mean gradient ($|\mathbb{E}[g_{ij}]|$). This selects elements with directionally consistent gradients across examples.
     $$\hat{S}_{ij} = \left| \frac{1}{N} \sum_{k=1}^N g_{k,ij} \right|$$
  2. Diagonal Fisher Score ($\hat{F}$): Estimates the diagonal of the empirical Fisher information ($\mathbb{E}[g_{ij}^2]$). This selects elements where the loss is most sensitive to perturbations, capturing both mean and variance.
     $$\hat{F}_{ij} = \frac{1}{N} \sum_{k=1}^N g_{k,ij}^2$$
• Selection: The top-$k$ elements globally across all layers are selected based on these scores to form a "circuit."

2.2 Theoretical Analysis of Gradient Structure

The paper posits that the efficacy of parameter placement depends on the gradient structure of the training regime:

• Supervised Fine-Tuning (SFT): Gradients are low-rank, directionally stable, and highly aligned across steps. Consequently, any random subset of parameters accumulates coherent updates, making placement less critical.
• Reinforcement Learning (GRPO): Gradients are high-rank, near-orthogonal across steps, and exhibit high variance. In this regime, updates to random parameters tend to cancel out (random walk), while only parameters with consistently signed, large gradients accumulate meaningful displacement.

2.3 Experimental Setup

• Models: Qwen2.5 (1.5B, 3B, 7B), Llama-3.2 (3B), and Llama-3.1 (8B).
• Tasks:
  • SFT: Instruction following (Alpaca, Bespoke-Stratos) and Vision-Language (MathVista).
  • RL: Group Relative Policy Optimization (GRPO) on mathematical reasoning datasets (GSM8K, MATH-500).
• Baselines: Random placement, Full LoRA, and the proposed informed placement ($\hat{S}$ and $\hat{F}$).

3. Key Results

3.1 Regime Dependence

• Under SFT: Random and informed placements achieve comparable performance. The gap is modest (e.g., <1% on instruction following benchmarks), confirming that SFT gradients are distributed enough that arbitrary subsets capture useful signal.
• Under GRPO (Base Models): Random placement fails completely, performing at or near the base model level (e.g., ~9.5% accuracy on GSM8K for 1.5B models vs. ~64% for Full LoRA). In contrast, informed placement ($\hat{S}$ and $\hat{F}$) recovers standard LoRA accuracy, often matching or exceeding it with less than 0.1% of the adapter parameters.
• Under GRPO (Instruct Models): While random placement performs better than on base models (due to pre-existing reward signals), informed placement still provides a consistent edge, particularly in out-of-distribution (OOD) generalization.

3.2 Efficiency and Stability

• Cost: Circuit discovery requires only $N=50$ forward-backward passes, taking <10 seconds on a single GPU (<0.5% of the cost of full SFT training).
• Stability: The selected circuits are highly stable across different random seeds and data subsets. Top-10K elements show >95% overlap with reference circuits using small sample sizes.
• Generalization: Circuits discovered on one math dataset (e.g., MATH) transfer effectively to others (e.g., GSM8K, AMC, AIME), often outperforming Full LoRA in OOD settings. This suggests the method identifies a stable, task-general subnetwork rather than dataset-specific artifacts.

3.3 Circuit Anatomy

Analysis of the selected parameters reveals a non-random concentration:

• Modules: Selected parameters concentrate heavily on residual-stream writing projections ($V$, $O$, and $Down$).
  • $V$ (Value) and $K$ (Key) act as "attention readers," aligning with the pretrained weight's dominant singular vectors.
  • $O$ (Output) and $Down$ act as "residual writers," producing spectrally concentrated, low-rank updates.
• Layers: In base models, circuits concentrate in the early layers (0–4), absorbing 70–98% of the budget.
• Knockout Validation: Zeroing out the top-ranked parameters in a trained adapter degrades performance significantly faster than random knockout, causally confirming that these specific parameters mediate the adapter's effect.

4. Key Contributions

1. Formalization of Parameter Placement: The paper separates the effects of parameter count from parameter identity, demonstrating that placement is critical under RL but less so under SFT.
2. Lightweight Selection Procedure: Proposes a gradient-based scoring method ($\hat{S}$ and $\hat{F}$) that identifies effective sparse subsets in seconds with negligible overhead.
3. Empirical Validation: Shows that informed placement recovers full LoRA performance under GRPO on base models with extreme sparsity (e.g., 10K parameters on a 20M parameter adapter), whereas random placement fails.
4. Mechanistic Insight: Provides evidence that the failure of random placement in RL is due to gradient noise and orthogonality, which prevents coherent accumulation in arbitrary subsets, while informed placement targets the sparse, consistent signal.

5. Significance and Claims

The paper claims that the "how many" assumption in parameter-efficient fine-tuning is insufficient; "which" parameters are trained is the determining factor in high-variance training regimes like Reinforcement Learning.

• For SFT: The findings suggest that while random placement is a reasonable baseline, informed placement offers marginal gains and potential regularization benefits in extreme budget scenarios.
• For RL: The work demonstrates that successful sparse adaptation in RL requires informed parameter selection. Random placement is structurally incapable of learning in this regime because the gradient signal is too noisy and distributed for random subsets to accumulate coherent updates.
• Practical Impact: The proposed method enables training effective adapters with less than 0.05% of the adapter parameters, significantly reducing memory and compute requirements for RL fine-tuning without sacrificing performance.

The authors conclude that their approach identifies a "circuit" of parameters that are functionally critical for learning, acting as an implicit regularizer that preserves generalization capabilities often lost in dense adapters.