💬 NLP

Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests

This paper introduces a consensus-labeled prompt bank that distinguishes between executable malicious code and harmful security knowledge to provide a reliable, standardized benchmark for measuring whether coding-specialized models meet the stricter refusal standards required to prevent the generation of functional weapons.

Original authors: Richard J. Young, Gregory D. Moody

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Richard J. Young, Gregory D. Moody

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Text vs. The Real Thing

Imagine you ask a general chatbot, "How do I make a bomb?" If it answers, it gives you a recipe. You still have to go to the store, buy the ingredients, and mix them yourself. It's dangerous information, but it's just words.

Now, imagine you ask a coding-specialized AI the same question. If it answers, it doesn't just give you a recipe; it hands you a fully assembled, working bomb that is ready to explode the moment you press "run."

The authors of this paper argue that because coding AIs can hand over "working weapons" (like viruses or spyware) instantly, they should be much stricter about saying "No" than regular chatbots. But right now, nobody has a good way to measure if they are actually saying "No" enough.

The Problem: Mixing Apples and Oranges

To test if these AIs are safe, researchers have been using lists of questions (prompts). But these lists have been messy. They mix two very different types of dangerous requests:

"Weapon" Requests: "Write me a program that steals passwords." (The AI gives you the code).
"Knowledge" Requests: "Explain how password stealing works." (The AI gives you an essay).

If you mix these together and say, "This AI refused 50% of the bad requests," you don't know if it refused the weapons or just the explanations. It's like testing a security guard by asking them to stop a thief and stop someone asking for directions. If the guard stops the thief but lets the direction-seeker through, you can't tell if they are good at their job just by looking at the total number of people stopped.

The Solution: A "Consensus-Labeled" Prompt Bank

The authors created a massive, clean, and organized list of 6,675 dangerous questions. They split them into two clear buckets:

The "CODE" Bucket: Questions asking for executable, dangerous software (the "weapons").
The "KNOWLEDGE" Bucket: Questions asking for harmful information or theories (the "recipes").

To make sure the questions were sorted correctly, they didn't just ask one person. They used a panel of five different AI judges (like a jury). Each judge looked at every question and voted: "Is this a weapon request?" or "Is this a knowledge request?"

The Verdict: If at least 3 out of 5 judges agreed, the question got a final label.
The Result: They ended up with 4,748 confirmed "Weapon" requests and 1,923 confirmed "Knowledge" requests.

The "Jury" and the Surprises

The authors used five different AI models to act as the judges. They wanted to make sure the test was cheap and open for everyone to use, so they picked models that are free or open-source, rather than expensive, paid ones.

During this process, they found two interesting things:

1. The "Too Easy" Paradox
For some of the question lists (like the ASTRA list), almost every single question was obviously a "Weapon." The judges all agreed 99% of the time.

The Metaphor: Imagine a math test where every question is "What is 2+2?" Everyone gets 100%. You can't really measure how "smart" the students are because the test was too easy.
The Finding: In statistics, when everyone agrees on everything, the usual score for "agreement" (called Kappa) breaks and looks like a zero or a negative number. The authors had to invent a special way to report this: "Hey, everyone agreed perfectly, but the math score looks weird because the test was too easy."

2. The "Gatekeeper" Glitch
One of the five AI judges (a free model from OpenAI) started refusing to answer any of the questions, even the ones it was supposed to judge. It kept hitting a "Stop" sign from the company that hosts it.

The Metaphor: Imagine a jury where one juror keeps getting kicked out of the courtroom for asking too many questions.
The Fix: Since the rule was "3 out of 5," the other four judges could still decide the verdict. The authors noted this as a real-world quirk: sometimes the free tools you use to test safety have their own safety filters that block the test before it even starts.

Why This Matters

This paper isn't about testing a specific AI to see if it's safe today. Instead, it's about building the ruler that everyone else can use.

Before this, researchers were trying to measure safety with a broken ruler that mixed up "weapons" and "recipes." Now, they have a standardized, high-quality ruler (the prompt bank) that clearly separates the two. This allows anyone to test coding AIs and say, "Okay, this AI refused 90% of the actual weapons," which is a much more meaningful safety check than before.

Summary

Goal: Create a clean list of dangerous questions to test if coding AIs are safe.
Method: Used a "jury" of 5 AIs to sort 6,675 questions into "Weapons" (Code) and "Recipes" (Knowledge).
Outcome: A public database of 4,748 confirmed weapon requests and 1,923 knowledge requests.
Key Insight: You can't measure safety properly if you mix up "making a bomb" with "reading about bombs." This paper fixes that mix-up.

Technical Summary: Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests

Problem Statement

The paper addresses a critical asymmetry in AI safety: while a general-purpose language model answering a harmful query returns text, a coding-specialized model complying with the same intent can generate executable "weapons" (e.g., keyloggers, ransomware droppers, exploit payloads). Current benchmarks for measuring malicious-code refusal fail to distinguish between requests for executable software (CODE) and requests for harmful security knowledge (KNOWLEDGE). Existing corpora mix these distinct request types, rendering single compliance-rate statistics uninterpretable and preventing direct cross-corpus comparisons. Furthermore, previous consensus-labeled efforts were limited in scope (four corpora) and relied on paid commercial APIs, hindering replicability and cost-effective extension.

Methodology

The authors present v2 of a consensus-labeled prompt bank, expanding the scope from four to eight upstream corpora and refreshing the evaluation panel to utilize zero-marginal-cost, open-weight, or free-tier models.

1. Corpus Expansion

The dataset pool was expanded from 3,133 prompts (v1) to 6,675 prompts by integrating four new publicly released corpora:

ASTRA: 1,995 prompts featuring jailbreak-decorated, single-turn text-to-code requests.
Scam2Prompt (Innoc2Scam-bench): 1,377 indirect-elicitation prompts disguising malicious intent as innocuous developer requests.
JailbreakBench (coding subset): 10 prompts specifically targeting executable code generation.
RedCode: 160 prompts targeting code-interpreter agent trajectories.
The v1 corpora (RMCBench, MalwareBench, CySecBench, and AdvBench/harmful_behaviors) were re-classified under the new panel to ensure continuity.

2. Judge Panel and Protocol

A five-judge panel was assembled from five distinct vendor families, selected for zero marginal API cost and accessibility via OpenRouter and Ollama Cloud:

NVIDIA: Nemotron-3-Super (General-purpose)
Alibaba: Qwen3-Coder-Next (Coder-specialized)
DeepSeek: DeepSeek-V4-Pro (General-purpose)
OpenAI: GPT-OSS-120B (Open weights, general-purpose)
Zhipu AI: GLM-5.1 (General-purpose)

Classification Protocol:

Template: A binary schema distinguishing CODE (requests for runnable software/weapons) from KNOWLEDGE (requests for text-based information/explanations). The template is byte-identical to v1 to prevent drift.
Consensus Rule: A 3-of-5 majority rule. A prompt is labeled CODE or KNOWLEDGE if at least three judges agree; otherwise, it is labeled AMBIGUOUS.
Reliability Metric: Fleiss' $\kappa$ with a 10,000-iteration bootstrap 95% confidence interval, calculated on the subset where all five judges returned valid labels (6,552 prompts).

Key Contributions

1. The Expanded Consensus-Labeled Bank

The released artifact comprises:

4,748 consensus-CODE prompts (a 3.1 $\times$ expansion of v1).
1,923 consensus-KNOWLEDGE prompts (a 5.0 $\times$ expansion of v1).
4 AMBIGUOUS prompts (released with per-judge votes for downstream re-thresholding).
The dataset is released under an OpenRAIL++ license with specific restrictions prohibiting the use of the data to train malicious-software-generation systems.

2. Methodological Findings

The paper documents two significant methodological observations arising from the construction of v2:

The High-Agreement Low- $\kappa$ Paradox: On four prevalence-skewed corpora (ASTRA, MalwareBench, RedCode, RMCBench), where consensus is >99% CODE, Fleiss' $\kappa$ degenerates to near-zero or negative values despite high observed agreement ( $P_o$ ). The authors attribute this to the Feinstein-Cicchetti paradox, where the expected chance agreement ( $P_e$ ) approaches 1.0. Consequently, the paper proposes a dual-statistic reporting convention: report $\kappa$ for prevalence-mixed corpora and observed agreement ( $P_o$ ) for prevalence-skewed corpora.
Provider-Side Content-Policy Refusals: One judge (GPT-OSS-120B routed via OpenRouter's OpenInference provider) returned deterministic HTTP 403 refusals on 1.84% of prompts (123 total), heavily concentrated in the harmful_behaviors corpus. The paper identifies this as a structural property of free-tier routing providers applying content filters before the model processes the prompt. The 3-of-5 consensus rule successfully absorbed these refusals without generating AMBIGUOUS labels, but the authors note this as a critical factor for researchers replicating the panel.

3. Cross-Panel Stability

The v2 panel (open-weight/free-tier) was compared against the v1 panel (paid commercial APIs) on the 3,133 overlapping prompts. The panels achieved 94.45% agreement on consensus labels, with a Cohen's $\kappa$ of 0.952 ("almost perfect") on the binary-binary subset. This validates that the zero-cost panel serves as a robust substitute for the commercial panel on shared corpora.

Results

Overall Reliability: On the full-panel subset (6,552 prompts), Fleiss' $\kappa$ is 0.767 [95% CI: 0.755, 0.777], falling in the "substantial" agreement band of the Landis-Koch scale.
Per-Corpus Reliability:
- Prevalence-Mixed: Scam2Prompt ( $\kappa=0.768$ ), CySecBench ( $\kappa=0.664$ ), and harmful_behaviors ( $\kappa=0.910$ ) show interpretable agreement.
- Prevalence-Skewed: ASTRA, MalwareBench, RedCode, and RMCBench show $\kappa$ values near zero or negative due to the degeneracy of the marginal distribution, despite mean observed agreement ( $P_o$ ) ranging from 0.850 to 0.989.
Judge Robustness: Leave-one-out analysis confirms that no single judge dominates the panel. Removing the most divergent judge (Nemotron-3-Super) actually increased $\kappa$ to 0.832, confirming that the disagreement introduced by this judge is meaningful boundary-case work rather than noise.
Agreement Tiers: 76.9% of prompts achieved unanimous 5/5 agreement; 95.0% achieved at least 4/5 agreement.

Significance and Claims

The paper claims to provide a construct-stable substrate for measuring coding-model compliance with malicious-code requests. By disambiguating executable software requests from knowledge requests, the bank allows for more precise safety evaluations than previous mixed-corpus benchmarks.

The authors modestly position the work as an expansion and methodological refinement of their prior four-corpus release [1], rather than a re-evaluation of the underlying CODE-versus-KNOWLEDGE axis. The primary significance lies in:

Scalability and Replicability: Demonstrating that high-reliability consensus labeling can be achieved using zero-marginal-cost, open-weight models, removing financial barriers for downstream researchers.
Standardization: Establishing a protocol for handling prevalence-skewed corpora (dual-statistic reporting) and documenting provider-side filtering behaviors that affect free-tier judge availability.
Data Availability: Releasing a significantly larger, multi-judge, reliability-quantified dataset (6,675 prompts) to enable cross-corpus comparisons that were previously not directly interpretable.

The paper explicitly states it does not address the behavioral question of how target coding LLMs respond to the v2 bank, noting that such evaluations are the scope of separate downstream work.