💻 computer science

A Validated Prompt Bank for Malicious Code Generation: Separating Executable Weapons from Security Knowledge in 1,554 Consensus-Labeled Prompts

This paper introduces a validated 1,554-prompt "CODE" bank and a consensus-labeled classification framework that rigorously separates executable malicious software requests from harmful security knowledge queries to address the conflation issues in existing language model safety benchmarks.

Original authors: Richard J. Young, Gregory D. Moody

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Richard J. Young, Gregory D. Moody

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to test how good a security guard is at stopping bad guys. You have a list of requests to give the guard. Some requests are like handing the guard a loaded gun and saying, "Shoot this person." Other requests are like handing the guard a textbook and saying, "Explain how a gun works and how to aim it."

For a long time, researchers testing AI safety have been mixing these two types of requests together in one big pile. They would ask the AI to "write a virus" (the loaded gun) and "explain how a virus spreads" (the textbook) and then just count how many times the AI said "No."

The authors of this paper, Richard J. Young and Gregory D. Moody, say this is a messy way to test. If an AI refuses to write the virus but happily explains the textbook, or vice versa, a single "No" count doesn't tell you what's actually happening inside the AI's brain. It's like trying to measure how well a guard stops bullets by also counting how well they stop people from reading about bullets. You can't tell if the guard is good at stopping weapons or just good at stopping information.

The Big Idea: Separating the "Weapons" from the "Knowledge"

To fix this, the researchers created a new tool: a Validated Prompt Bank. Think of this as a highly organized, double-checked filing cabinet. They took thousands of questions from four different existing test lists and sorted them into two distinct drawers:

The "Weapons" Drawer: These are prompts that ask the AI to actually build something dangerous, like a piece of malware or a script that can run on a computer. These are the "executable weapons."
The "Knowledge" Drawer: These are prompts that ask the AI to describe dangerous things, like explaining how a virus works or how to hack a system, without actually building the code.

How They Built the Filing Cabinet

They didn't just sort the papers themselves; they wanted to be absolutely sure the sorting was correct. So, they set up a "jury" of five different AI judges. These judges came from five different tech companies (Anthropic, OpenAI, Google, Zhipu AI, and Alibaba).

Imagine five different experts looking at a single request. They all have to vote: "Is this a request for a weapon, or is it just a request for information?"

If at least three out of the five agree, that's the final answer.
They checked their work using a statistical tool (called Fleiss' $\kappa$ ) to see how much they agreed. The result was "almost perfect" agreement (0.876 out of 1.0). This means the sorting is incredibly reliable.

The Result: A Clean, Verified List

After sorting through over 17,000 original requests and filtering out duplicates or confusing ones, they ended up with a clean, verified list of 1,554 prompts that are purely about asking for "weapons" (malicious code). They also kept a smaller list of 388 prompts about "knowledge" to use for comparison later.

Why This Matters

Before this paper, if a researcher wanted to test if an AI was safe, they had to build their own messy pile of questions, guessing which ones were "weapons" and which were "knowledge." This meant every study was slightly different, making it hard to compare results.

This paper provides a standardized, pre-sorted set of questions. Now, researchers can say, "We tested the AI on these specific 1,554 weapon requests," and everyone else knows exactly what that means. It removes the guesswork and allows for a fair, apples-to-apples comparison of how different AI models handle dangerous requests.

Important Safety Note

The authors are very clear: They did not create any new viruses or dangerous code. They only took existing questions from other academic studies and sorted them. The goal of this tool is to help researchers make AI safer by understanding exactly where it fails, not to help anyone build attacks. The data is locked behind a "gated" system, meaning only verified researchers working on safety can get access to the actual text of the questions.

In short, this paper built a better, cleaner ruler for measuring AI safety, separating the "do the bad thing" requests from the "talk about the bad thing" requests so we can measure them accurately.

Technical Summary: A Validated Prompt Bank for Malicious Code Generation

1. Problem Statement

Current benchmarks evaluating Large Language Model (LLM) refusal on malicious-coding tasks suffer from a critical methodological flaw: they conflate requests for executable malicious software (weapons) with requests for harmful security knowledge (information). Existing datasets mix prompts such as "write a keylogger" (executable code) with "explain how a keylogger works" (descriptive knowledge).

This conflation is problematic because these two request types likely trigger distinct refusal pathways in safety-aligned models. A single refusal-rate statistic computed over a heterogeneous mixture cannot distinguish between a model that refuses to write malware but explains cryptography, and one that writes malware but refuses to describe cryptography. Consequently, "code-safety refusal rates" in current literature measure an unknown construct, introducing unobserved variance that limits cross-study comparability and mechanism-level analysis. Furthermore, prior benchmarks often rely on single-annotator labels or pairwise agreement without rigorous multi-rater consensus, leaving residual class-boundary ambiguity.

2. Methodology

The paper introduces a weapons-versus-knowledge classification axis, operationalized through a multi-judge consensus protocol to separate executable-code requests from knowledge-based prompts.

2.1 Data Consolidation and Pre-Filtering

The study consolidated 3,133 candidate prompts drawn from four public benchmarks:

RMCBench: 473 prompts (all code-generation requests).
MalwareBench: 320 filtered prompts (subset of 3,520, excluding adversarial paraphrasing variants).
CySecBench: 1,820 prompts (filtered from 12,662 raw prompts using a regex-based screen for imperative code-generation phrasing).
harmful_behaviors (from AdvBench): 520 content-safety prompts.

A rule-based pre-filter removed exact duplicates and non-code-generation requests (e.g., conceptual questions), retaining only prompts plausibly requesting executable output for the initial classification pool.

2.2 Five-Model Consensus Classifier

Each of the 3,133 prompts was independently classified by a panel of five Large Language Models acting as judges. The panel was deliberately vendor-diverse (spanning four vendor families: Anthropic, OpenAI, Google, Zhipu AI, Alibaba) and included both general-purpose and coder-specialized models to mitigate correlated labeling errors:

Claude Sonnet 4.6 (Anthropic, General)
GPT-5.3-Codex (OpenAI, Coder)
Gemini 3 Flash (Google, General)
GLM-5 (Zhipu AI, General)
Qwen3-Coder-Next (Alibaba, Coder)

Classification Task: Judges were prompted to assign a binary label:

CODE: The prompt asks for the production of executable code, scripts, or functioning software artifacts.
KNOWLEDGE: The prompt asks for information, strategies, tutorials, or explanations in natural language.

Consensus Rule: A 3-of-5 majority rule was applied. A prompt was assigned to the consensus-CODE set if at least three judges labeled it CODE, and to the consensus-KNOWLEDGE set if at least three labeled it KNOWLEDGE. No prompts failed to reach a 3-of-5 majority; the "AMBIGUOUS" bucket remained empty.

2.3 Reliability Metrics

Inter-rater reliability was quantified using Fleiss' $\kappa$ with 10,000-iteration bootstrap 95% confidence intervals. This metric was calculated on the subset of prompts where all five judges returned valid labels (2,488 prompts).

3. Key Contributions

The primary contribution is the release of a consensus-labeled prompt bank that serves as a standardized, validated substrate for code-safety research.

The Code-Safety Bank: A dataset of 1,554 consensus-CODE prompts (representing 1,551 unique prompt strings). This set isolates requests for executable malicious software.
The Knowledge Comparison Set: A set of 388 consensus-KNOWLEDGE prompts (derived exclusively from the harmful_behaviors source) used for within-subject comparisons in companion studies.
Validation: The dataset includes per-prompt metadata, including individual judge labels, agreement tiers (e.g., 5/5, 4/5), and source taxonomy categories (27 malware categories).
Open Science: The consolidation code, consensus labels, and reliability statistics are released openly (MIT License) on GitHub. The prompt text is available as a gated-access dataset on Hugging Face to mitigate dual-use risks while enabling bona fide research.

4. Results

4.1 Filtering Cascade

The pipeline reduced 17,175 raw source prompts to the final artifact:

Pre-filtering: Reduced the pool to 3,133 candidates.
Consensus: 1,618 prompts were labeled CODE, and 1,515 were labeled KNOWLEDGE.
Final Artifact: After a length and metadata-quality pass, 1,554 CODE prompts and 388 KNOWLEDGE prompts were released.

4.2 Inter-Rater Reliability

Overall Agreement: The five judges achieved a Fleiss' $\kappa$ of 0.876 [95% CI: 0.862, 0.888], which corresponds to "almost perfect" agreement by the Landis & Koch convention.
Unanimity: 69.3% (2,170) of the 3,133 prompts received unanimous 5/5 agreement.
Zero Ambiguity: All 3,133 prompts reached the 3-of-5 threshold; zero prompts were excluded as ambiguous.
Per-Source Reliability:
- MalwareBench: $\kappa = 1.000$ (Perfect agreement).
- harmful_behaviors: $\kappa = 0.942$ ("Almost perfect").
- CySecBench: $\kappa = 0.775$ ("Substantial"), reflecting higher boundary ambiguity in this source.
- RMCBench: $\kappa \approx 0.001$ (A degenerate case due to lack of variance, as all prompts were CODE).
Pairwise Agreement: All ten pairwise comparisons between judges exceeded the "substantial" threshold ( $\kappa \ge 0.61$ ), with ten pairs exceeding "almost perfect" ( $\kappa \ge 0.81$ ).

4.3 Judge Performance

Error Rates: Error rates varied significantly by model (from 1 error for GLM-5 to 430 for Gemini 3 Flash). However, the 3-of-5 consensus rule successfully tolerated these errors, resulting in zero consensus errors across the dataset.
Coder vs. General: Coder-specialized judges (GPT-5.3-Codex, Qwen3-Coder-Next) produced slightly higher CODE counts on average than general-purpose judges, supporting the hypothesis that code-tuned pre-training aids in distinguishing executable requests from descriptive ones.

5. Significance and Claims

The paper modestly claims to provide the first multi-judge consensus-validated prompt bank that explicitly separates executable-weapon requests from harmful-knowledge requests across consolidated benchmarks.

Methodological Resolution: The work resolves the conflation of distinct request types that has confounded previous malicious-code generation literature. By isolating executable-code requests, it enables more precise measurement of the specific construct that code-safety evaluations aim to capture.
Standardization: The artifact allows downstream research to adopt a canonical, reliability-documented substrate rather than independently reconstructing filtering logic. This reduces unobserved variance in reported refusal rates and supports reproducible cross-model comparisons.
Scope of Contribution: The paper explicitly states that whether the "weapons-versus-knowledge" axis separates model behavior in practice is an empirical question left to a companion benchmark study. The current contribution is the artifact itself and the validation of the classification axis, not the behavioral results of models on this axis.
Ethical Stance: The dataset is intended for defensive research (measuring refusal, safety alignment, and interpretability). The authors emphasize that no new malware was created, and the release follows dual-use mitigation frameworks (gated access) to prevent the artifact from being used to operationalize attacks.