Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis

Technical Summary: Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis

Problem Statement
Large Language Models (LLMs) are increasingly deployed in software security workflows for vulnerability detection, triage, and repair. However, these workflows face a "boundary problem": the terminology and reasoning patterns required for legitimate defensive analysis (e.g., identifying exploits, analyzing attacker-controlled input) closely resemble those associated with malicious misuse. Consequently, safety-aligned models often over-refuse benign defensive requests.

Prior evaluations of this phenomenon suffer from two limitations:

1. Confounding Variables: Many studies compare unrelated model families, making it difficult to attribute performance differences to safety alignment rather than architecture, scale, or training data.
2. Refusal-Centric Metrics: Existing evaluations focus primarily on refusal rates, failing to capture whether non-refused answers are correct, specific, or actionable. A model that rarely refuses but provides incorrect localization or invalid patches offers little utility to defenders.

This paper investigates how a model's "safety state" affects the utility of legitimate vulnerability analysis, specifically examining whether the cost of alignment manifests not just in refusal, but in the degradation of answer quality and actionability.

Methodology
The study employs a matched same-lineage experimental design to isolate the variable of safety state. The authors compare two states derived from the same open-source model families:

• ALIGNED: The standard instruction-tuned model with intact refusal behaviors.
• ABLITERATED: A publicly released descendant where refusal behavior has been surgically reduced (ablated) via activation-space interventions (subtracting a learned refusal direction) or weight-space modifications.

Model Families:

• Primary Pair: Gemma (comparing gemma-4-31B-it vs. gemma-4-31B-CRACK).
• Supplementary Pair: Qwen (comparing Qwen3.6-27B-MXFP4 vs. Qwen3.6-27B-MXFP4-CRACK).

Evaluation Framework:
The study decomposes "defender-side utility" into three dimensions: Coverage (usability of the answer), Answer Quality (correctness among usable answers), and End-to-End Utility (practical usefulness across the workflow). The evaluation spans a "task-depth progression" from shallow diagnosis to executable repair:

1. Vulnerability Detection & CWE Attribution: Binary classification and taxonomy mapping.
2. Vulnerable-Line Localization: Identifying specific source lines containing vulnerabilities.
3. Root-Cause Localization: Explaining the causal mechanism of the vulnerability.
4. Executable Patch Validation: Generating patches that can be applied, compiled, and validated against proof-of-vulnerability (PoV) tests (using Java/Vul4J, Python/PatchEval, and C/C++/Vul4C).

Prompt Framing:
The study controls for prompt wording by varying two dimensions:

• Authorization Context: Neutral review vs. Explicit internal security audit context.
• Lexical Intensity: Varying the density of cybersecurity terminology (Neutral, Scenario, Tactic/Pattern, Dense Cyber) while keeping the underlying code and task fixed.

Key Results
The findings indicate that safety-state effects are context-dependent rather than monotonic:

1. Task Depth and Localization:

   • Shallow Tasks: For general detection and CWE attribution, the ALIGNED state often performs better, particularly under neutral code-review wording.
   • Code-Grounded Tasks: As tasks require specific code evidence (vulnerable-line localization), the ABLITERATED state consistently outperforms the ALIGNED state across both Gemma and Qwen pairs.
     • Gemma: Top-1 accuracy for line localization increased from 9.00% (ALIGNED) to 10.10% (ABLITERATED); Mean line-F1 increased from 3.84% to 4.00%.
     • Qwen: Top-1 accuracy increased from 4.10% to 6.95%; Mean line-F1 increased from 2.08% to 3.91%.

2. Executable Repair (Java/Vul4J):

   • The ABLITERATED state demonstrated significantly higher throughput in early executable gates.
   • Usable Answers: 67.8% (ABLITERATED) vs. 29.9% (ALIGNED).
   • Applied Patches: 65.0% vs. 24.9%.
   • Compiled Patches: 32.8% vs. 9.0%.
   • PoV Pass: 6.8% vs. 1.7%.
   • Nuance: In Neutral prompts, the ALIGNED state occasionally showed slightly better final validation success rates in Java, but the ABLITERATED state dominated in security-explicit frames (Authorized/Cyber-Term). In cross-language checks (Python, C/C++) under neutral prompts, the ALIGNED state remained competitive or superior in final execution gates.

3. Prompt Sensitivity:

   • Safety mechanisms appear sensitive to surface cues. Under neutral wording, ALIGNED models often retain an advantage in classification. However, when prompts include explicit authorization or dense cybersecurity terminology, the advantage shifts toward the ABLITERATED state, particularly for localization and repair tasks.
   • Output Stability: Prompt wording changes the actual content of the answer, not just the score. ALIGNED models exhibited higher "decision flip" rates (changing the predicted vulnerable line or CWE label) when moving from neutral to security-dense wording compared to ABLITERATED models.

Significance and Claims
The paper claims that current evaluations of LLM-based security assistants are insufficient if they rely solely on refusal rates. The authors argue that:

• Safety State Affects Utility Profile: Alignment changes how a model answers, not just if it answers. The cost of alignment is not a uniform refusal but a shift in the model's ability to provide actionable, high-quality defensive assistance depending on the task depth and prompt framing.
• Evaluation Must Be Holistic: Cyber-safety evaluations should jointly measure refusal, correctness, localization quality, and executable actionability. A model that refuses frequently may be less harmful than one that answers frequently but provides incorrect or unusable guidance.
• Design Implications: Safeguards should be designed to distinguish between authorized defensive analysis and exploit enablement without degrading the quality of legitimate defensive outputs. The goal is to preserve reliable support for defenders who require precise security language to analyze and repair real vulnerabilities.

The study does not claim that removing safeguards is the solution; rather, it highlights that current safety mechanisms may inadvertently penalize legitimate defensive work, suggesting a need for more nuanced evaluation metrics that account for the "defender-side utility" across the full engineering workflow.