Technical Summary: DualSFT – One Algorithm, Two Goals for LLM Fine-Tuning

1. Problem Statement

Large Language Model (LLM) fine-tuning faces prohibitive costs at modern scales, driving research into two distinct efficiency strategies: Parameter-Efficient Fine-Tuning (PEFT) and Data-Efficient Fine-Tuning (DEFT). PEFT methods (e.g., parameter selection, LoRA) reduce trainable state memory, while DEFT methods (e.g., data selection, reweighting) reduce optimization workload.

Currently, these strategies operate independently. When both a parameter mask and a data subset are applied to restrict updates, existing approaches typically employ separate scoring mechanisms. This separation incurs redundant computational overhead and creates a coordination mismatch: a parameter mask selected without considering the specific data subset (and vice versa) may lead to suboptimal joint performance. The paper posits that since parameter masks and data subsets jointly determine restricted updates, they should be selected via a shared scoring rule rather than disconnected mechanisms.

2. Methodology: DualSFT

The authors propose DualSFT (Dual-Selection Fine-Tuning), a one-shot algorithm that co-extracts parameter importance and data utility from shared gradient statistics. The core theoretical insight is casting parameter and data selection as two views of a single local response-surrogate problem under a common validation objective.

2.1 Theoretical Foundation: Bilevel Formulation and Surrogates

The selection process is formulated as two bilevel optimization problems sharing an outer validation objective $\mathcal{L}_{val} = \mathcal{L}_{new} + \lambda \mathcal{L}_{prior}$, where $\mathcal{L}_{new}$ is the target-task loss and $\mathcal{L}_{prior}$ preserves pre-trained capabilities.

• Outer Level: Selects a budget-constrained mask (parameter mask $\mathbf{m}_\theta$ or data mask $\mathbf{m}_D$).
• Inner Level: Computes the optimal parameter update $\delta^*$ induced by the selected mask.

Solving these exactly is intractable for LLMs. The authors approximate the inner response using a one-step gradient update at a common warmup checkpoint $\bar{\theta}$. This yields local surrogates where the update direction depends on the selected subset.

2.2 The Shared Interaction Matrix

By expanding the validation objective around $\bar{\theta}$ using first- and second-order Taylor approximations, the authors derive a gradient interaction matrix $\mathbf{M} \in \mathbb{R}^{N \times D}$ (where $N$ is the number of samples and $D$ is the parameter dimension).

• Parameter Importance: Emerges as the column-wise aggregation of $\mathbf{M}$.
• Data Utility: Emerges as the row-wise aggregation of $\mathbf{M}$.

This structure reveals a row-column duality: both signals are derived from the same algebraic object. The closed-form scores admit a Shapley value interpretation, treating the selection as a cooperative game where the interaction matrix provides the credit assignment.

2.3 The DualSFT Algorithm Pipeline

The algorithm executes in three stages:

1. Pilot Warmup: A lightweight warmup (1 epoch, 5% of data) establishes a stable checkpoint $\bar{\theta}$ and estimates diagonal curvature $\hat{c}$ using AdamW's second-moment buffer.
2. One-Shot Dual Scoring:
   • A validation-improvement projection vector $\mathbf{u}$ is constructed: $\mathbf{u} = \eta_{sc}(\nabla \mathcal{L}_{new} + \lambda \nabla \mathcal{L}_{prior}) - \frac{\eta_{sc}^2}{2} \hat{c} \odot \mathbf{G}$, where $\mathbf{G}$ is the accumulated training gradient.
   • Parameter Scores: $s_{\theta, d} = u_d G_d$ (column aggregation).
   • Data Scores: $s_{D, n} = \langle \mathbf{u}, \mathbf{g}_n \rangle$ (row aggregation, computed via Ghost Dot Product to avoid materializing per-sample gradients).
   • Top-$k$ parameters and Top-$b$ samples are selected based on these signed scores.
3. Restricted Fine-Tuning: The model is fine-tuned on the selected data subset using only the selected parameters.

2.4 Preservation Mechanism

To mitigate catastrophic forgetting without access to pre-training data, DualSFT employs Confidence-Weighted Self-Distillation (CWSD). This constructs the preservation direction $\nabla \mathcal{L}_{prior}$ by minimizing the KL divergence between the current model and the original pre-trained model on an anchor set, weighted by the confidence of the original model's predictions.

3. Key Contributions

1. Unified Scoring Theory: The derivation of a shared response-surrogate scoring rule that transforms separate parameter and data selection problems into row-wise and column-wise aggregations of a single gradient interaction matrix, supported by a localized Shapley interpretation.
2. DualSFT Algorithm: A practical, one-shot dual-scoring algorithm that coordinates parameter and data selection simultaneously, eliminating redundant scoring passes and coordination gaps found in sequential hybrid baselines.
3. Preservation Direction: A corpus-free preservation direction using CWSD that effectively mitigates forgetting during restricted fine-tuning.
4. Empirical Validation: Comprehensive experiments on 3B–9B LLMs (Llama-3.2, Gemma-3, Qwen-3.5) across code generation and mathematical reasoning tasks.

4. Experimental Results

Experiments were conducted on Magicoder (code) and MetaMathQA (math) with matched budgets (10% data, 5% parameters).

• Performance: DualSFT outperforms single-axis baselines (e.g., LESS, S2FT, NanoAdam) and sequential hybrids (e.g., LESS + S2FT, SPICE + NanoAdam).
  • On Llama-3.2-3B (Magicoder), full DualSFT achieved 47.10% Overall score, surpassing the strongest sequential hybrid (SPICE + NanoAdam at 44.71%) by 2.39 points.
  • DualSFT variants consistently led their respective comparison groups in both plasticity (target task performance) and stability (retention of general capabilities).
• Efficiency: DualSFT reduces fine-tuning costs. Full DualSFT achieved its results with 7.4 GPU-hours and 28.5 GB peak selection memory, outperforming sequential hybrids in the joint-constrained trade-off.
• Ablations:
  • Removing the warmup step significantly hurt plasticity.
  • Removing CWSD significantly hurt stability.
  • Using only first-order scoring (ignoring diagonal curvature) caused the largest drop in Overall performance, validating the second-order approximation.
• Diagnostic Analysis: The paper confirms that one-shot scoring captures most of the joint structure, as fixed-side re-selection (re-scoring one axis after fixing the other) yields marginal gains (<0.12 points) at the cost of significantly increased computation.

5. Significance and Claims

The paper claims that DualSFT represents a shift from treating parameter and data selection as independent heuristics to viewing them as coupled adaptation resources. By deriving both signals from a single validation-improvement surrogate, the method:

• Reduces Redundancy: Eliminates the need for separate scoring mechanisms and repeated conditional passes.
• Improves Coordination: Aligns the selection axes before the joint update, avoiding the "coordination gap" where independent selections mix unverified signals.
• Balances Trade-offs: Achieves a more favorable stability-plasticity-efficiency frontier than existing sequential or single-axis methods under matched budgets.

The authors modestly note that the method relies on a local surrogate, meaning score fidelity may decay as optimization moves far from the warmup checkpoint. They suggest dynamic re-scoring as a future direction for longer adaptation horizons but emphasize that the one-shot approach is highly cost-effective for the studied budgets.