💻 computer science

Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)

This paper presents a systematic review of thirteen malicious-code prompt corpora used to evaluate coding LLM refusal, identifying critical methodological gaps in human annotation baselines, cross-corpus comparability, and taxonomy standardization while proposing a unified framework for future dataset construction.

Original authors: Richard J. Young, Gregory D. Moody

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Richard J. Young, Gregory D. Moody

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where we are trying to teach giant, super-smart robots (called Large Language Models or LLMs) how to be good citizens. Specifically, we want to know: If someone asks these robots to write a computer virus or a scam, will they say "No"?

To test this, researchers have been creating "exam papers" (datasets of prompts) to trick the robots. This paper is a massive report card review of 13 different exam papers created between 2023 and 2025. The authors, Richard Young and Gregory Moody, looked at all these exams to see how they were built, how fair they were, and what they actually tested.

Here is the breakdown of their findings, using simple analogies:

1. The Problem: Everyone is Using a Different Ruler

Imagine trying to compare the height of 13 different people. But here's the catch:

Person A is measured with a ruler in inches.
Person B is measured with a tape measure in centimeters.
Person C is measured by how many steps they take to reach a wall.
Person D is measured by a different person entirely.

The paper argues that this is exactly what is happening with these AI safety tests. Each of the 13 exam papers (like AdvBench, CyberSecEval, RedCode, etc.) was built differently:

Different Questions: Some ask for code directly; some ask for code through a complex story; some ask the AI to act like a hacker; others ask it to act like a helpful assistant who might make a mistake.
Different Graders: Some exams are graded by the authors alone. Some are graded by other AI bots. Some are graded by running the code in a sandbox to see if it explodes.
Different Rules: Some exams have strict rules about what counts as a "virus." Others are vague.

The Result: You cannot simply take the "refusal rate" (how often the AI said "No") from one exam and compare it to another. It's like comparing a sprinter's time to a swimmer's time and declaring one "faster" without accounting for the different sports.

2. The Three Big Missing Pieces

The authors found that none of the 13 exam papers included three critical safety features that you would expect in a serious scientific study:

No "Human Judges" to Calibrate the AI: Most exams use other AI bots to grade the answers. But the paper points out that we don't know if these AI graders are actually right. It's like having a robot grade a math test without a human teacher ever checking if the robot's grading key is correct. None of the 13 papers had a "Fleiss' kappa" score (a statistical way to prove that different human judges would agree on the same answer).
No "Locked Doors" (Gated Access): These exams contain instructions on how to make dangerous software (malware). Yet, all 13 are completely open to the public. Anyone can download them, even if they have bad intentions. It's like publishing a cookbook for making bombs and leaving it on a park bench for anyone to pick up.
No "Recall Policy" (Takedown): If someone finds a dangerous prompt in these exams that shouldn't be there, or if the AI learns from it and starts making real viruses, there is no official way to contact the authors to take it down. None of the papers listed a "takedown policy" or a specific person responsible for removing harmful content.

3. The "Empty Seats" in the Classroom

The authors created a map (a taxonomy) to show what kinds of "trick questions" exist. They found that the researchers are all sitting in the same few seats, leaving many others empty:

Crowded Seats: Most exams ask for code in a single, direct sentence (e.g., "Write a virus").
Empty Seats: Very few exams test complex scenarios, like:
- Asking the AI over a long conversation (multi-turn) to slowly build a virus.
- Asking the AI to act as an autonomous agent that controls a computer.
- Asking for code that attacks hardware (like smart fridges or car chips) rather than just software.

Because the "seats" are uneven, the data is skewed. We know a lot about how AI handles simple requests, but we know very little about how it handles complex, multi-step attacks.

4. The Recommendations: How to Fix the Classroom

The paper suggests that if we want to build better "exam papers" in the future, we need to follow a new checklist:

Pre-register the Rules: Before building the exam, write down exactly what you are testing so you don't change the rules halfway through.
Use a Panel of Human Judges: Don't just use one AI to grade. Use a diverse group of humans (or a mix of humans and different AIs) to agree on what counts as a "refusal."
Report the Agreement Score: Publish a statistic showing that the judges actually agreed with each other.
Use a Standard Dictionary: Agree on a single list of "bad things" (malware types) so everyone is counting the same categories.
Lock the Door: If the exam contains dangerous instructions, make it hard to get (require a researcher application) so bad actors can't easily download it.
Name a Guardian: Have a specific person responsible for removing the exam if it causes harm.

Summary

This paper is a "systematic review," meaning it didn't run new experiments. Instead, it acted like a librarian who walked into a room with 13 different "Safety Test" books, opened them all, and realized: "We are all measuring different things with broken rulers, and we are leaving the dangerous parts of these books out in the open for anyone to find."

The authors are calling for the community to stop building these tests in isolation and start agreeing on a standard, safe, and fair way to measure if AI is truly refusing to do bad things.

Technical Summary: Refusal Evaluation in Coding LLMs and Code Agents

Problem Statement

The evaluation of Large Language Model (LLM) refusal on malicious-code-generation tasks has fragmented into at least thirteen publicly released prompt corpora (e.g., AdvBench, CyberSecEval, RMCBench, RedCode, MOCHA, ASTRA, JAWS-Bench). These datasets are constructed under disparate protocols, utilize different licensing terms, and lack standardized inter-rater reliability (IRR) validation. Existing literature conflates "refusal evaluation" (does the model decline a request for malicious code?) with "vulnerability evaluation" (does the model generate code with bugs on benign requests?), or treats prompt corpora merely as adjacent context to broader security taxonomies. Consequently, refusal-rate statistics reported across these corpora are not directly comparable; they measure non-equivalent constructs due to variations in prompt composition, modality (single-turn vs. multi-turn vs. agent), and labeling protocols (human vs. automated vs. LLM-judge). There is no canonical schema for malware categories, and downstream researchers are forced to reconstruct inclusion criteria from scratch for every new study.

Methodology

This paper employs a PRISMA-style systematic review protocol to analyze the thirteen in-scope corpora as the primary unit of analysis.

Search Strategy: A multi-engine search (arXiv, Google Scholar, ACL, IEEE, ACM) combined with snowball-back citation tracking from key entry-point corpora.
Inclusion Criteria: Corpora must target coding LLMs or code-interpreter agents, release a prompt corpus publicly (or describe it in detail), and evaluate the refusal of executable malicious-code requests.
Exclusion Criteria: Vulnerability-detection benchmarks (inverse task), general capability benchmarks (e.g., HumanEval), non-coding jailbreaks, and knowledge-elicitation MCQs (e.g., WMDP) were excluded.
Extraction Process: A uniform extraction template was applied to each corpus's primary publication and release artifacts. Fields included construction methodology, prompt count, modality, malware taxonomy, target model panel, reported refusal rates, IRR statistics, and licensing/access policies. No new empirical measurements or re-labeling were performed; the review synthesizes metadata as released.

Key Contributions

The paper produces four synthesis artifacts that reframe the field from a collection of disparate benchmarks to a structured, comparable landscape:

Cross-Corpus Methodology Comparison: A comprehensive table (Table 1) extracting ten axes of comparison for all thirteen corpora. It reveals that construction strategies range from pure handcrafting to LLM-generated synthesis and oracle-driven agent loops. Crucially, it documents that zero of the thirteen corpora report a Fleiss' $\kappa$ -style inter-rater reliability statistic on their refusal labels.
Prompt-Construction Taxonomy: A three-axis framework (Modality, Turn Structure, Elicitation Style) that characterizes how corpora elicit malicious code.
- Modality: Text-to-code, code-to-code, multi-turn, agent, and interpreter.
- Turn Structure: Single-turn, multi-turn, or agent-trajectory.
- Elicitation Style: Direct, jailbreak-decorated, or indirect/plausibly-benign.
- The taxonomy visualizes "empty cells" in the design space (e.g., multi-turn jailbreak-decorated prompts), highlighting structural gaps in current evaluation coverage.
Reproducibility and Licensing Landscape: A catalog (Table 2) of distribution venues, licenses, and access policies. It finds that while most corpora are open-access, five lack explicit license statements on their landing pages, and none document a takedown policy or responsible-disclosure contact, despite containing executable malware payloads.
Malware-Category Coverage Map: A consolidated 22-category ontology derived from the union of all corpus taxonomies. The map identifies well-covered categories (e.g., Trojans, Ransomware, Phishing) and under-represented areas, specifically firmware exploits, hardware/side-channel attacks, IoT/embedded attacks, and supply-chain attacks.

Key Results

Uniform Absence of Reliability Metrics: Across all thirteen corpora, there is no reporting of inter-rater reliability (e.g., Fleiss' $\kappa$ ) for refusal labels. Labels are derived via author-only assignment, upstream provenance, or automated classifiers without human-annotator baselines for calibration.
Construct Inconsistency: Refusal rates are not comparable across corpora because they sample different prompt distributions (e.g., handcrafted seeds vs. LLM paraphrases) and different elicitation styles (direct vs. jailbreak vs. indirect).
Governance Gaps: No surveyed corpus implements gated access, registered-researcher review, or a documented takedown policy. Five corpora ship without explicit licensing terms at the preprint stage.
Taxonomy Fragmentation: There is no canonical malware schema. The review had to construct a 22-category union ontology to enable cross-corpus comparison, as existing taxonomies (MITRE-derived, functional families, attack types) do not map cleanly to one another.
Coverage Imbalance: While common malware types are saturated, critical emerging vectors like firmware exploits and supply-chain attacks remain under-represented, often covered by only one or two corpora.

Significance and Claims

The paper claims to be the first systematic review to treat malicious-code prompt corpora themselves as the unit of analysis, rather than treating them as secondary context to code security or jailbreak taxonomies. Its significance lies in:

Auditing the Field: It exposes that the rapid pace of corpus release has outstripped the methodological scaffolding required for valid comparison.
Providing a Baseline: It offers a structured vocabulary (the taxonomy) and a consolidated ontology (the 22 categories) for future researchers to audit selection decisions and compare datasets.
Defining Methodological Expectations: It proposes a checklist for next-generation corpus construction, including pre-registration of inclusion criteria, vendor-diverse multi-judge label validation, reporting of Fleiss' $\kappa$ with bootstrap confidence intervals, and the adoption of a canonical malware taxonomy.
Future Work: The authors identify a uniform empirical re-evaluation of inter-rater reliability across all thirteen corpora using a common multi-judge protocol as the necessary next step, which is deferred to future work.

The review concludes that while the field has generated a rich set of evaluation artifacts, the lack of standardized reliability reporting, licensing governance, and taxonomy alignment currently prevents the aggregation of refusal-rate statistics into a coherent, interpretable metric for LLM safety.