Technical Summary: SPARD

Problem Statement

Large Language Models (LLMs) are increasingly adapted to specific domains via fine-tuning-as-a-service. However, this process often undermines the safety alignment achieved during pre-training. This vulnerability is exploited by harmful fine-tuning attacks, where adversarial data is injected to strip away safety safeguards and induce unsafe behaviors.

Existing defense mechanisms face two primary limitations:

1. Weak Control: Methods like SafeInstr and Lisa treat safe data as a soft regularizer (penalty term), offering only indirect control over safety constraints, making it difficult to balance safety with downstream utility.
2. Suboptimal Data Selection: Current approaches typically select safe samples randomly. They overlook the fact that safe data highly relevant to the downstream task provides stronger corrective signals, while ignoring the need for diversity to cover broad risk spaces.

Methodology

The authors propose SPARD (Safety Projection with Relevance–Diversity Data Selection), a defense framework comprising two core components:

1. Safety-Projected Alternating Gradient (SPAG)

SPARD formulates fine-tuning as a safety-constrained optimization problem:
$$\min_{\theta} \mathcal{L}(D_{ft}, \theta) \quad \text{s.t.} \quad \mathcal{L}(D_{safe}, \theta) \leq \tau$$
where $D_{ft}$ is the fine-tuning dataset, $D_{safe}$ is the safety dataset, and $\tau$ is a safety threshold.

Instead of using penalty-based relaxation, SPAG employs an explicit projection strategy:

1. Utility Update: Perform a gradient step on the fine-tuning data ($D_{ft}$) to update parameters $\theta$.
2. Safety Projection: Project the updated parameters back into a feasible region defined by the safety constraint.
   • The constraint is linearized using a first-order Taylor expansion around the updated parameters.
   • The projection finds the point in the linearized safety half-space closest to the updated parameters.
   • A trust-region mechanism limits the step size of the projection to prevent arbitrarily large updates that could destabilize training.

This approach provides a closed-form solution that guarantees feasibility up to the first order without requiring manual tuning of penalty weights.

2. Relevance–Diversity Determinantal Point Process (DPP)

To ensure the safety projection is effective, SPARD introduces a principled method for selecting a compact subset of safe data ($D_{safe}$) from a larger candidate pool. The selection balances two factors:

• Relevance: Safe samples must align with the downstream task distribution to provide strong corrective signals.
• Diversity: Selected samples must cover a broad range of distinct harmful behaviors to prevent overfitting to a narrow risk space.

The authors construct a Relevance–Diversity DPP kernel:
$$\tilde{K}_{ij} = (q_i \cdot q_j)^\beta \cdot K(x_i, x_j)$$
where $q_i$ is a relevance score (maximum cosine similarity between a candidate and the fine-tuning data), $K(x_i, x_j)$ captures intrinsic similarity, and $\beta$ controls the influence of relevance.

• The term $(q_i \cdot q_j)^\beta$ rewards subsets containing highly relevant samples.
• The determinant term $\det(L_C)$ enforces diversity among selected samples.

An efficient greedy selection algorithm is used to approximate the Maximum A Posteriori (MAP) subset, avoiding the computational cost of exact DPP inference.

Key Contributions

1. SPARD Framework: A novel defense integrating safety-projected optimization with relevance-diversity aware data selection.
2. SPAG Algorithm: A principled optimization method that alternates between utility updates and explicit safety projections, solving safety-constrained fine-tuning without soft penalties.
3. Relevance–Diversity DPP: A data selection strategy that jointly optimizes for task alignment and behavioral coverage, addressing the redundancy and irrelevance issues in random sampling.
4. Empirical Validation: Extensive experiments demonstrating that SPARD outperforms state-of-the-art defenses in reducing Attack Success Rates (ASR) while maintaining high downstream accuracy.

Experimental Results

The authors evaluated SPARD on GSM8K (math reasoning) and OpenBookQA (science reasoning) using Qwen-2.5-7B-Instruct and LLaMA-3.2-3B-Instruct under four harmful fine-tuning attacks (BeaverTails, I-BeaverTails, LatHarmful, Q-LatHarmful).

• Safety Performance: SPARD consistently achieved the lowest average Attack Success Rate (ASR) and Harmfulness Score (HS) across all models and datasets. For instance, on GSM8K with Qwen, SPARD reduced the average ASR to 9.45% (compared to 19.12% for the strong baseline Lisa and 87.93% for standard SFT).
• Utility Preservation: SPARD maintained competitive downstream task accuracy, often outperforming baselines like Lisa and SafeGrad. On GSM8K with Qwen, SPARD achieved 85.77% accuracy, surpassing Lisa (78.45%).
• Ablation Studies:
  • Relevance vs. Diversity: Removing relevance (Random selection) or diversity (Max Quality selection) resulted in higher ASR, confirming that both factors are necessary.
  • Hyperparameters: The method is robust to moderate variations in the relevance exponent ($\beta$) and safety threshold ($\tau$).
  • Efficiency: SPARD adds only 8.1% computational overhead compared to standard SFT, significantly lower than SafeGrad (+54.2%).

Significance and Claims

The paper claims that SPARD offers a principled defense against harmful fine-tuning attacks by moving beyond soft penalties to explicit geometric constraints. By integrating SPAG, the method ensures safety constraints are strictly enforced during training. Furthermore, the Relevance–Diversity DPP addresses the critical insight that not all safe data is equally effective; selecting data that is both relevant to the task and diverse in coverage is essential for robust safety alignment.

The authors assert that SPARD provides a robust balance between safety and utility, outperforming existing methods (including PTST, SafeInstr, Lisa, and SafeGrad) without requiring structural constraints (like LoRA adapters) or manual weight tuning. The framework is shown to generalize across different model architectures (Qwen and LLaMA) and downstream tasks (math and science QA).