💻 computer science

Willing but Unable: Separating Refusal from Capability in Code LLMs via Abliteration

This paper demonstrates that "abliteration," a low-rank weight editing technique that orthogonally projects out refusal directions, can effectively decouple safety-aligned code LLMs' refusal to generate vulnerable code from their actual generation capabilities, thereby enabling the scalable production of labeled vulnerable code for vulnerability detection research without compromising syntactic validity.

Original authors: Cristina Carleo, Pietro Liguori, Naghmeh Ivaki, Domenico Cotroneo

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cristina Carleo, Pietro Liguori, Naghmeh Ivaki, Domenico Cotroneo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Over-Protective" Robot

Imagine you are trying to teach a robot how to spot security holes in computer code. To do this, you need a huge library of examples showing both "safe" code and "broken" (vulnerable) code.

However, there's a catch. The robots (AI models) we have today are trained to be very safe. If you ask them, "Hey, can you show me how to break a database?" they immediately say, "No! I can't do that. That's dangerous."

This is a problem for researchers. They need the robot to make the mistake so they can study it and build better defenses. But the robot refuses to play the role of the "bad guy," even if the researcher just wants to study the mistake in a controlled lab setting.

The Experiment: Turning Off the "Stop" Button

The researchers in this paper wanted to see if they could turn off that "Stop" button without breaking the robot's brain. They used a technique called Abliteration.

Think of the AI's brain as a giant library of thoughts. When the robot sees a request to create a security hole, a specific "refusal signal" (like a red alarm light) flashes in its mind, and it shuts down the conversation.

Abliteration is like a surgeon carefully removing just that one red alarm light from the robot's wiring. They didn't retrain the robot or teach it new things; they just surgically edited the weights (the connections) to stop the refusal signal from firing.

The Three Key Findings

The researchers tested this on three different sizes of AI models (Small, Medium, and Large) using Python code and a specific type of security hole called SQL Injection (which is like tricking a database into revealing secrets).

1. The "Refusal" Depends on Size and Context

Before they did any surgery, they noticed something interesting:

The Giant Model (14B): It refused 100% of the time. No matter what you asked, it said "No."
The Medium Model (7B): It was picky. It refused most of the time on long, complex code, but it would sometimes say "Yes" on short, simple code snippets.
The Small Model (3B): It barely refused at all. It was willing to try almost anything.

The Analogy: Imagine three security guards. The big guard is so strict he stops everyone. The medium guard stops people with big bags but lets people with small envelopes through. The small guard is very relaxed and lets almost everyone pass.

2. The Surgery Worked (The "Willing" Part)

After they performed the "Abliteration" surgery:

The Refusal Disappeared: The Giant and Medium models stopped saying "No." They were now willing to try to create the security hole.
The Brain Was Still Healthy: Crucially, the surgery didn't break the robot. The code they produced was still grammatically correct and made sense. They hadn't lobotomized the AI; they just removed the "I won't do that" reflex.

The Analogy: It's like taking a "Do Not Enter" sign off a door. The door was always open and the hallway inside was fine; the sign was just telling people they couldn't walk through. Once the sign was gone, people could walk through, and the hallway was still perfectly intact.

3. Willingness $\neq$ Ability (The "Unable" Part)

This is the most important discovery. Just because the robot was now willing to make the mistake, didn't mean it was able to do it well.

The Giant Model (14B): Once the "No" sign was removed, it was great at making the security hole. It succeeded about 90% of the time.
The Medium Model (7B): Also very good, succeeding about 90% of the time.
The Small Model (3B): Even though it was willing and didn't refuse, it was bad at the actual task. It only succeeded about 25–48% of the time.

The Analogy: Imagine three artists asked to paint a picture of a broken vase.

The Giant Artist was told "You can't paint this!" but after being told "You can," they painted a masterpiece.
The Medium Artist was also told "You can," and they painted a great picture.
The Small Artist was never told "No" to begin with, but when they tried to paint the broken vase, they just couldn't get the details right. They wanted to do it, but they lacked the skill.

The Conclusion: Two Different Things

The paper proves that Refusal (willingness) and Capability (skill) are two separate things.

Refusal is a safety setting that can be turned off (via Abliteration).
Capability is a measure of how smart the model is. Turning off the safety setting doesn't make a "dumb" model "smart."

Why does this matter?

For Researchers: If you want to generate data to train security tools, you can't just use a small model and hope it works. Even if you turn off its safety filters, it might not be smart enough to create the realistic errors you need. You need a bigger model.
For Safety: It shows that safety filters are very specific. You can remove the "refusal" without breaking the model's ability to write good code. This means safety alignment is a specific layer on top of the model's intelligence, not the intelligence itself.

A Note on Ethics

The authors are very careful. They released the tools to measure this and the data, but they did not release the "surgically modified" models that can actually create these security holes. They believe that while the idea of how to do this is known, releasing the actual "hacked" models would be too dangerous. They want to help researchers study security, not give hackers a new weapon.

Technical Summary: Willing but Unable: Separating Refusal from Capability in Code LLMs via Abliteration

Problem Statement
The paper addresses a critical bottleneck in learning-based vulnerability detection: the scarcity of accurately labeled training data. Existing datasets suffer from significant label noise (20–71%) and duplication because they are mined from CVE-tagged commits. While Large Language Models (LLMs) offer a path for data augmentation, current methods typically transform existing vulnerable seeds, thereby inheriting the noise of the source corpus. A complementary approach involves starting with safe code and instructing an LLM to inject a specific Common Weakness Enumeration (CWE) vulnerability. However, modern safety-aligned, instruction-tuned code LLMs systematically refuse such prompts due to Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), even when the model possesses the underlying capability to perform the transformation. This refusal creates a barrier to generating large-scale, specification-driven vulnerability datasets.

Methodology
The authors propose a feasibility study using abliteration, a low-rank weight editing technique, to remove the refusal barrier while preserving generative capability. The study focuses on CWE-89 (SQL Injection) in Python as a case study.

Model Selection: The study utilizes the Qwen2.5-Coder-Instruct family at three parameter scales: 3B, 7B, and 14B. These models were chosen for their open weights, code specialization, and feasibility for CPU-based inference.
Abliteration Procedure: Based on the hypothesis that refusal is mediated by a specific linear subspace in the model's residual stream, the authors apply orthogonal projection to remove this "refusal direction."
- They compute a contrast matrix between activations on harmful (CWE-injection) and harmless prompts.
- Singular Value Decomposition (SVD) identifies the top- $k$ right singular vectors representing the refusal direction.
- These vectors are projected out of the output weight matrices ( $W_{out}$ ) of specific transformer layers (typically the upper third of the network) without retraining.
- Hyperparameters ( $k$ and refinement passes) are tuned per model size (e.g., $k=2$ for 3B, $k=8$ for 7B/14B).
Experimental Pipeline:
- Input: 70 safe Python samples from PromSec (20 samples, multi-route Flask apps) and SafeCoder (50 samples, single-route snippets).
- Prompting: A fixed instruction asks the model to rewrite the safe code to introduce CWE-89 (e.g., replacing parameterized queries with string concatenation).
- Validation: A three-step process evaluates outputs:
  1. Refusal Classification: Regex-based detection of refusal patterns.
  2. Syntactic Validation: AST parsing and compilation checks.
  3. Vulnerability Detection: An ensemble of three static analyzers (CodeQL, Semgrep, Bandit) flags potential vulnerabilities.
  4. Manual Adjudication: Two authors manually verify detector-positive outputs against the CWE-89 specification to resolve false positives/negatives.
Evaluation Metrics: The study measures refusal rates, syntactic validity, vulnerability injection rates, and code similarity (CrystalBLEU/CodeBLEU) against ground-truth vulnerable counterparts.

Key Contributions

Reproducible Pipeline: A validated workflow applying abliteration to instruction-tuned code LLMs, coupled with a multi-tool static analysis ensemble and manual adjudication.
Empirical Measurement: A controlled factorial study (3 models $\times$ 2 conditions $\times$ 2 datasets $\times$ 3 runs) providing the first empirical measurements of refusal and post-abliteration compliance on CWE-injection tasks.
Orthogonal Separation of Willingness and Capability: The study demonstrates that refusal (willingness) and code-generation capability are orthogonal axes. Abliteration successfully removes the refusal reflex without creating new capabilities; it merely unlocks capabilities that were already present but suppressed.
Open Artifacts: Release of the evaluation harness (refusal classifier, validators, scoring scripts) and aggregate data, while withholding specific abliterated checkpoints and vulnerable samples to mitigate misuse risks.

Results

Refusal is Size- and Context-Dependent:
- The 14B model refused 100% of prompts on both datasets.
- The 7B model refused 73% of PromSec prompts (multi-route) but only 5% of SafeCoder prompts (single-route).
- The 3B model exhibited negligible refusal (0–7%).
Abliteration Removes Refusal:
- Post-abliteration, refusal rates dropped to 0% (or near-zero, e.g., 1.7% for 3B on PromSec) across all model sizes and datasets.
- Capability Preservation: Syntactic validity remained above 93% for all abliterated models. The intervention did not degrade the models' ability to generate valid, coherent Python code.
Capability is Capacity-Bound:
- Once refusal was removed, the vulnerability injection rate scaled with model parameters:
  - 14B: ~88–97% injection success.
  - 7B: ~89–90% injection success.
  - 3B: ~25–48% injection success.
- The 3B model's low success rate persisted even after abliteration, indicating that the limitation was a lack of generative capacity (inability to perform the structural edit correctly), not a refusal mechanism.
Detector Concordance: Static analyzers showed varying levels of agreement. CodeQL was highly specific to certain idioms (e.g., flask_mysqldb), while Bandit had higher recall on syntactic patterns. Manual adjudication was essential for resolving discrepancies.

Significance and Claims
The paper claims to provide a preliminary feasibility study demonstrating that:

Refusal can be detached from capability: Safety alignment in code LLMs acts as a "refusal reflex" that can be surgically removed via low-rank weight edits without damaging the underlying generative substrate.
Data Generation Viability: The pipeline offers a method to generate labeled vulnerability data where the labeling burden shifts from open-ended detection (prone to noise) to bounded binary confirmation. The 14B and 7B models, once abliterated, show high potential for generating high-quality, specification-driven vulnerability datasets.
Dual-Use Implications: The authors highlight a security concern: the ability to remove refusal via simple weight edits implies that safety-aligned open-weight models can be easily converted into tools for generating malicious code. They argue that the "attack surface" exposed by abliteration on complex inputs is already present by default on simpler inputs (as seen with the 7B on SafeCoder).
Methodological Insight: The study suggests that evaluating safety-aligned models solely on "refusal + failure" rates conflates two distinct phenomena. A model's apparent failure on sensitive tasks may be entirely due to refusal, which is removable, rather than a lack of capability.

The authors maintain a modest tone, noting that the study is limited to a single CWE (89), a single language (Python), and one model family. They explicitly state that the findings do not guarantee generalizability to other CWEs or model architectures without further empirical validation.