Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

Technical Summary: Finer Parameter Steps for Low-Rank PEFT

Problem Statement
Low-rank adapters, such as LoRA, are standard for Parameter-Efficient Fine-Tuning (PEFT). However, the rank parameter $r$ in LoRA fixes the resolution of the parameter budget. For large projection matrices (e.g., $2048 \times 2048$ in OPT-1.3B), a single rank increment adds 4,096 trainable scalars per projection. This creates large, discrete gaps in the low-budget region, resulting in a sparsely sampled accuracy–budget curve. The paper investigates whether an adapter family with finer capacity increments can reveal different operating points or improve the observed accuracy–budget trade-off compared to LoRA.

Methodology
The authors propose a controlled comparison between LoRA and fixed-component Canonical Polyadic (CP) tensor adapters.

1. Adapter Formulation:

   • LoRA: Uses the standard update $\Delta W = BA$, where the rank $r$ determines the parameter count. For a $2048 \times 2048$ matrix, one rank step equals 4,096 parameters.
   • CP Adapters: The matrix update is reshaped into a four-way tensor ($32 \times 64 \times 32 \times 64$). The update is parameterized as a sum of $c$ rank-one components with normalized factors:
     $$T(\Delta W) = \sum_{s=1}^{c} \lambda_s u^{(1)}_s \circ u^{(2)}_s \circ u^{(3)}_s \circ u^{(4)}_s$$
     where factors have unit norm. One CP component stores 193 trainable scalars (32+64+32+64 factors + 1 scalar), approximately 21 times smaller than a single LoRA rank step.

2. Experimental Protocol:

   • Model & Tasks: OPT-1.3B model adapted on three classification tasks: SST-2, RTE, and BoolQ.
   • Targets: Attention $q$-proj and $v$-proj modules (48 projections total).
   • Comparison Metric: Instead of comparing only at matched parameter counts, the authors define a "best-under-budget" curve $U_A(B)$. For a given budget cap $B$, this reports the highest accuracy achieved by any setting with a parameter count $\le B$. This approach makes the discretization of feasible budgets explicit.
   • Controls: Training protocols, data caps, seed schedules (base grid: seeds 0–2; confirmation: seeds 0–9), and checkpoint selection are held constant. Learning rates were fixed prior to the main sweep ($10^{-4}$ for LoRA, $2 \times 10^{-4}$ for CP).

Key Results
The study reveals that the benefit of finer parameter steps is highly task-dependent:

• SST-2 (Early Plateau): Small CP adapters (e.g., $c=2$) reach an accuracy plateau (~0.93) with a fraction of the LoRA $r=1$ budget. However, adding more CP components does not monotonically improve performance, and no CP setting exceeds the best LoRA result under the same budget caps. The low-budget region appears saturated early.
• BoolQ (Gradual Rise and Saturation): The CP curve shows a clear, gradual improvement from $c=1$ to $c=43$, revealing useful intermediate capacity growth that LoRA's coarse grid misses. However, the curve eventually saturates slightly below the best LoRA result ($0.739$ vs. $0.743$).
• RTE (Persistent Gap): LoRA consistently outperforms CP across all budget levels. Even with finer steps and additional components, CP fails to close the accuracy gap, suggesting that for this task, the expressivity of LoRA's unconstrained dense rank-one directions is more critical than budget granularity.

Key Contributions

1. Explicit Budget Discretization: The paper introduces a "best-under-budget" comparison framework that explicitly visualizes how different adapter families discretize the parameter space, moving beyond simple matched-parameter comparisons.
2. Controlled Tensorized Adapter Study: The authors instantiate this framework using fixed-component CP adapters, providing a controlled test case where the capacity parameter step size is significantly smaller than LoRA's.
3. Task-Dependent Diagnostics: The study demonstrates that finer parameter steps act as a diagnostic tool. They reveal the shape of the budget-response curve (e.g., early saturation vs. gradual growth) but do not inherently guarantee a superior accuracy–budget curve.

Significance and Claims
The authors explicitly state that they do not claim the fixed-component CP adapter is a new state-of-the-art PEFT method. Instead, the significance lies in isolating the effect of budget granularity.

The paper concludes that while finer parameter steps are useful for diagnosing PEFT budget sensitivity and revealing operating points between LoRA ranks, they do not automatically overcome expressivity gaps. In some cases (RTE), the flexibility of LoRA's update structure is more important than the resolution of the budget grid; in others (BoolQ), finer steps reveal intermediate performance gains; and in others (SST-2), the task saturates so early that the grid resolution is less critical. The work serves as a controlled study on how the discretization of the capacity parameter influences the observed performance landscape.