💻 computer science

A Study on Question-Answer Dataset for LLM Safety Evaluation with a Focus on Illegal Activities

This paper introduces an enhanced question-answer dataset for evaluating LLM safety regarding illegal activities by building upon the AnswerCarefully dataset with new information, creation methods, and an evaluation rubric, with results intended for the JAI-Trust project.

Original authors: Kenji Imamura, Masao Ideuchi, Atsushi Fujita

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kenji Imamura, Masao Ideuchi, Atsushi Fujita

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, super-fast robot librarian (a Large Language Model, or LLM) who can answer any question you ask. The problem is, sometimes people ask this librarian for help doing things that break the law, like stealing, making illegal drugs, or scamming people. If the librarian helps, it becomes an accomplice to the crime.

This paper is like a safety manual and a new training guide for that librarian. The authors, researchers from Japan's National Institute of Information and Communications Technology, are working on a project called "JAI-Trust" to make sure these AI robots stay on the right side of the law.

Here is the story of their work, broken down into simple parts:

1. The Old Map Had Holes

The researchers started by looking at an existing "map" of bad questions called the AnswerCarefully dataset. Think of this map as a list of "Do Not Answer" signs.

The Problem: They found the map was a bit messy. Some signs were vague, and some dangerous questions didn't have clear "legal reasons" attached to them.
The Fix: They decided to focus strictly on illegal activities (things that break specific laws), ignoring vague "unethical" behaviors for now, because laws are clearer than moral opinions. They also noticed the map missed some very common crimes, like theft or traffic violations, which are like the "common weeds" in a garden that the map forgot to list.

2. Building a Better Training Kit

To fix the map, they proposed a new, more detailed format for every question and answer. Imagine they are upgrading the librarian's training cards. Instead of just a question and a "No," they now add five new fields to every card:

Question Type: Did the person asking know they were asking something illegal? (e.g., "I know stealing is wrong, but how do I do it?" vs. "How do I make a cool glowing fish?" not realizing it's illegal).
The "Bad" Answer: They created examples of what the librarian should not say (e.g., "Here is the website to buy the stolen goods"). This is like showing the librarian a "Do Not Do" example so they know what to avoid. Note: The authors warn that sharing these "bad" answers publicly is dangerous, so they keep them very careful.
The Legal Basis: Every "No" answer must now cite the specific law (like "Article 5 of the Penal Code"). It's like the librarian saying, "I can't help you because the law says X," which makes the refusal stronger.
Who is the Criminal? Who is doing the bad thing? (The asker, the AI, or someone else?)
Who is the Victim? Who gets hurt? (The asker, a stranger, or a third party?)

3. How to Write New Training Cards

The researchers tried two ways to write these new cards:

Method A: The Human Detective. Humans read real court cases and news stories, then wrote new questions based on them. This is like a teacher writing a test based on real-life stories. It's very accurate but slow and requires experts to check the work.
Method B: The Robot Assistant. They asked another AI to draft the questions based on a specific law, and then humans edited them. This is like having a student write a first draft and a teacher polishing it. It's faster and creates more variety, but the human teacher still has to check for mistakes (hallucinations).

They tested these methods using Japan's Minor Offenses Act (a law covering small crimes like carrying a hidden knife or making false police reports) to see if they could cover a wide range of "small" illegal acts that often lead to bigger crimes.

4. The Grading Rubric (The Scorecard)

Finally, they created a scorecard to grade how well the librarian answered. It's not just a pass/fail; it's a math formula.

The Formula: Score = (Discouragement) × (Legal Reasoning + Risk Explanation) × (Quality)
How it works:
- If the librarian says "No, that's illegal" and explains why (citing the law) and what could go wrong (the risk), they get a high positive score.
- If the librarian says "No" but gives no reason, they get a lower score.
- If the librarian says "Yes, here is how you do it," they get a huge negative score.
- If the librarian ignores the question entirely, they get a zero.

The Bottom Line

The paper doesn't claim to have solved everything. They admit they only looked at Japanese laws so far and haven't built a massive dataset yet. However, they have laid the blueprint (the new format), the construction methods (how to write the cards), and the inspection tool (the scorecard) for the "JAI-Trust" project to build a much larger, safer, and smarter safety system for AI in the future.

In short: They are upgrading the AI's "rulebook" to make sure it knows exactly which laws it's breaking, who might get hurt, and how to say "No" in the most convincing way possible.

Technical Summary: A Study on Question–Answer Dataset for LLM Safety Evaluation with a Focus on Illegal Activities

Problem Statement
The widespread adoption of Large Language Models (LLMs) has raised practical concerns regarding their inappropriate use, particularly in facilitating illegal activities. While existing safety evaluation datasets like AnswerCarefully (AC) and Do-Not-Answer exist, the authors identify specific limitations in the current version of AC (v2.2) when applied to the "Malicious Uses" category. Key issues include:

Typological Ambiguity: The leaf-level categories under "Assisting Illegal Activities" are not mutually exclusive, leading to examples that could belong to multiple categories (e.g., copyright infringement overlapping with illegal digital activities) but are labeled with only one.
Lack of Legal Grounding: The dataset does not consistently link examples to specific statutory laws, making it difficult to verify the legal basis for safety refusals.
Coverage Gaps: The dataset lacks representation for high-frequency crimes (e.g., theft, traffic violations) and does not sufficiently distinguish between users who are aware of the illegality of their query versus those who are naive.
Absence of Unsafe Examples: The current dataset contains only "safe" answers, limiting its utility for contrastive learning or training models to recognize unsafe outputs.

Methodology
The study proposes an extension of the AC dataset focused specifically on illegal activities under Japanese law. The methodology involves three primary components:

Extended Data Format: The authors propose a new schema (Table 4) that augments the standard question-answer pair with five new fields:
- q-type: Classifies the questioner's awareness of illegality (e.g., aware, unaware/naive, ambiguous).
- unsafe-answer: Provides an example of an inappropriate response (e.g., one that incites or details the illegal act) to serve as a negative training signal, while noting the ethical risks of public distribution.
- basis: Explicitly cites the relevant legal grounds (laws and articles) justifying the safety refusal.
- risk-perpetrator: Identifies the agent responsible for the potential illegal act (e.g., the questioner, the LLM provider, or a third party).
- risk-victim: Identifies the potential victims (e.g., the questioner, third parties, or the public).
Data Creation Strategies: The authors explored two approaches for generating new examples based on the Minor Offenses Act (a law covering diverse, frequent, yet often overlooked offenses):
- Fully Manual Creation: Workers create examples by analogy to judicial precedents and web sources, ensuring no copyrighted or private information is reused directly. This allows for precise control over attributes like question type.
- LLM-Assisted Creation: A semi-automatic pipeline where an LLM (e.g., DeepSeek-R1) is prompted with specific legal provisions to draft candidate questions and answers. Human workers then refine these drafts, filter for hallucinations, and ensure legal accuracy. This approach aims to improve diversity and coverage.
Evaluation Rubric: A scoring rubric (Table 6) is proposed to evaluate LLM responses. The final score is calculated using the formula: $score = a \times (b + c) \times d$ .
- $a$ (Discouragement): Measures if the response discourages the act and offers alternatives.
- $b$ (Reference to Illegality): Measures if the response correctly cites legal grounds.
- $c$ (Reference to Risks): Measures if the response explains potential harms.
- $d$ (Validity): A binary check for response relevance and quality.
  This rubric allows for quantitative differentiation between safe and unsafe responses (e.g., a safe answer scoring +4 vs. an unsafe answer scoring -4).

Key Contributions

Dataset Extension Proposal: A detailed specification for extending the AnswerCarefully dataset to include legal grounding, risk analysis, and unsafe answer examples.
Creation Framework: A validated workflow for constructing safety datasets that balances manual rigor with LLM-assisted efficiency, specifically tailored to statutory laws.
Evaluation Metric: A structured rubric designed to assess not just the refusal of illegal requests, but the quality of the reasoning (legal citation and risk explanation) provided in the refusal.
Analysis of Existing Data: A critical review of AC v2.2 highlighting the need for multi-label categorization and better coverage of high-frequency offenses.

Results and Findings

Analysis of AC v2.2: In the "Assisting Illegal Activities" subset (316 examples), 85% could be linked to specific Japanese laws. The Penal Code was the most frequent legal basis (111 examples), followed by various special acts. However, significant gaps were found; for instance, theft (the most common crime in Japan) had only one instance in the dataset, and Road Traffic Act violations were entirely absent.
Question Awareness: Approximately 77% of the analyzed questions were from users explicitly aware of the illegality. The study argues that including "naive" questions (where users are unaware) is crucial, as safety refusals for these depend more on content understanding than on recognizing the user's malicious intent.
Pilot Dataset: The authors successfully created a small-scale dataset focusing on the Minor Offenses Act (e.g., impersonating police officers, handling explosives) using both manual and LLM-assisted methods. These examples demonstrated the feasibility of the proposed data format and rubric.

Significance and Claims
The authors state that the outcomes of this study are intended to be shared with the "JAI-Trust" project, led by the Japan AI Safety Institute. The paper claims that its findings and the proposed extensions will contribute to the development of a large-scale benchmark dataset for LLM safety.

The study positions itself as a foundational step, acknowledging that while it has analyzed existing data and created a small-scale pilot, the construction of a comprehensive, large-scale dataset and its full evaluation remain future work. The authors emphasize the ethical necessity of handling "unsafe answers" with extreme caution, noting that public distribution of such content could itself constitute aiding illegal behavior. Consequently, the paper focuses on the methodology of creation and evaluation rather than releasing a fully populated, public dataset of unsafe examples.

1. The Old Map Had Holes

2. Building a Better Training Kit

3. How to Write New Training Cards

4. The Grading Rubric (The Scorecard)

The Bottom Line

More like this