Technical Summary: LLM-VA

Problem Statement

Safety-aligned Large Language Models (LLMs) currently suffer from two distinct failure modes: jailbreak (answering harmful inputs) and over-refusal (unnecessarily declining benign queries). Existing efficient safety alignment methods, specifically vector steering, attempt to manipulate the model's latent space by adjusting the magnitude of the "answer vector" ($v_a$). However, the authors identify a fundamental trade-off in this approach: reducing the magnitude of $v_a$ suppresses jailbreaks but increases over-refusal, while amplifying it has the opposite effect.

The root cause of this trade-off is identified as a structural decoupling within the model. The authors demonstrate that LLMs encode the decision to answer ($v_a$) and the judgment of input safety ($v_b$, or "benign vector") as nearly orthogonal directions (approximately 90°) across layers. Because these vectors are orthogonal, the model treats the willingness to answer and the assessment of safety as independent processes. Consequently, magnitude-based adjustments affect both benign and toxic inputs equally, making it impossible to simultaneously suppress jailbreaks and over-refusal without fine-tuning or architectural changes.

Methodology: LLM-VA

The authors propose LLM-VA (Large Language Model Vector Alignment), a method that resolves this trade-off by aligning $v_a$ with $v_b$ through closed-form weight updates, requiring no gradient-based optimization, fine-tuning, or architectural modifications. The core insight is to make the model's willingness to answer causally dependent on its safety assessment.

The method proceeds in three steps:

1. Vector Identification via SVMs:
   Instead of extracting vectors from the final residual stream (which aggregates all layers), LLM-VA identifies vectors at each layer. It trains linear Support Vector Machines (SVMs) on layer outputs to separate:

   • Benign vs. Toxic inputs (yielding the benign vector $v_b$).
   • Answered vs. Refused samples (yielding the answer vector $v_a$).
     The normal vectors of the maximum-margin hyperplanes provide robust control vectors, less sensitive to extreme points than simple mean-difference directions.

2. Layer Selection:
   Not all layers contribute equally to safety decisions. The method computes a score for each layer based on:

   • Influence: The alignment of the layer's vectors with the final residual stream vectors.
   • Accuracy: The SVM classification accuracy for separating benign/toxic and answer/refuse samples.
     The top $L_{select}$ layers with the highest combined scores are selected for modification.

3. Vector Alignment (Closed-Form Update):
   For selected layers, the method modifies the down-projection matrix $W$ to align $v_a$ with $v_b$. The goal is to ensure that for any input, the projection onto the answer direction is proportional to the projection onto the safety direction.
   The update $\Delta$ is derived analytically using the pseudoinverse to minimize the norm of the weight change:
   $$\Delta = \left( \frac{\sigma_a}{\sigma_b} W v_b - W v_a \right) v_a^T$$
   where $\sigma_a$ and $\sigma_b$ are standard deviations of the projections. This process is performed iteratively (up to 30 times), re-extracting vectors from the modified model in each step to account for shifts in subsequent layers, with early stopping based on validation F1 scores.

Key Contributions

• Novel Vector Steering: LLM-VA is presented as the first vector steering method that simultaneously addresses jailbreak and over-refusal by aligning answer and benign vectors, rather than merely adjusting magnitudes.
• No Fine-Tuning or Architecture Changes: The method achieves safety alignment through closed-form weight updates, avoiding the computational cost of retraining and the complexity of inserting hook layers (unlike methods such as SCANS or CAST).
• Automatic Adaptation: The method automatically adapts to the specific safety bias of different models. For models prone to jailbreaks, it prioritizes safety; for overly conservative models, it prioritizes usability, without manual hyperparameter tuning for specific models.
• Broad Applicability: The approach is validated across 12 LLMs from 5 different model families (Llama, Gemma, Mistral, Phi, Qwen) with parameter sizes ranging from 3B to 14B.

Experimental Results

Experiments on 12 LLMs demonstrate the following:

• Trade-off Resolution: LLM-VA achieves an average F1 score of 0.77, representing a 37.02% relative improvement over original models and an 11.45% improvement over the best baseline (AlphaSteer). It simultaneously reduces the Attack Success Rate (ASR) by 18.50% and the Over-Refusal Rate (ORR) by 22.00%.
• Utility Preservation: The method preserves 95.92% of the original model's utility on average across diverse NLP tasks (grammar, inference, sentiment, math). This outperforms baselines like SCANS (which suffered ~40% utility loss) and VectorSteer.
• Ablation Insights:
  • Vector Accuracy: Accurate identification of vectors via SVMs is critical; randomizing vectors by 90° causes a significant F1 drop.
  • Iteration Count: Convergence varies by model; some models (e.g., Phi-4) are sensitive to over-modification, while others (e.g., Qwen3) benefit from more iterations.
  • Layer Selection: Modifying too many layers or early layers degrades utility, confirming that later layers are more critical for safety decisions while early layers maintain general capabilities.

Significance and Claims

The paper claims that LLM-VA fundamentally resolves the jailbreak-over-refusal trade-off by addressing the structural independence of safety and answer decisions in LLMs. By geometrically aligning these decision vectors, the method ensures that the model only answers when it deems the input safe, and refuses when it deems the input toxic, without the need for expensive retraining.

The authors position this as a significant step toward efficient safety alignment, offering a "plug-and-play" solution that preserves model utility while significantly improving safety metrics. However, they modestly acknowledge limitations, including the reliance on binary toxicity assumptions, the need for labeled training data for SVMs, challenges with reasoning models (Chain-of-Thought), and the current lack of transferability across unseen datasets without specific tuning. They also note that the static nature of the alignment may require periodic updates as threat landscapes evolve.