RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models

1. Problem Statement

As Large Language Models (LLMs) are deployed in safety-critical domains (e.g., healthcare, legal, finance), ensuring their robustness against adversarial inputs is paramount. However, the current landscape of red teaming (adversarial testing) faces three critical challenges:

• Inconsistent Risk Categorization: Existing datasets use fragmented, overlapping, or ambiguous definitions for risks, making cross-dataset comparisons difficult.
• Limited Domain Coverage: Most datasets focus narrowly on specific risks (e.g., toxicity) or general knowledge, lacking comprehensive coverage of specialized domains (e.g., military, nutrition, infrastructure).
• Outdated Evaluations: Many benchmarks evaluate older model architectures and fail to provide baselines for modern state-of-the-art (SOTA) models (e.g., Llama 3.1, Qwen2.5, Gemma 2).

2. Methodology

The authors introduce RedBench, a universal dataset designed to harmonize and standardize red teaming evaluations.

A. Data Collection & Aggregation

• Source: Aggregated 37 benchmark datasets from leading conferences (NeurIPS, ACL, ICML, ICLR, arXiv) and repositories.
• Scale: The final corpus contains 29,362 samples.
• Dual Focus: The dataset covers two distinct red teaming directions:
  1. Attack Prompts (33 datasets): Designed to elicit unsafe, biased, or erroneous responses (e.g., jailbreaking, toxicity).
  2. Refusal Prompts (4 datasets): Designed to test "over-refusal," where models incorrectly reject benign or legitimate requests, hindering usability.

B. Standardized Taxonomy

To resolve inconsistency, the authors developed a unified taxonomy applied to all 29,362 samples:

• 22 Risk Categories: Including Abusive Content, Cybersecurity Threats, CBRN (Chemical, Biological, Radiological, Nuclear) capabilities, Misinformation, Election Interference, and Confabulation.
• 19 Domains: Spanning specialized fields like Healthcare, Military, and Law, as well as General Knowledge and Technology.
• Annotation Process: A semi-automated pipeline using Qwen2.5-72B-Instruct for initial labeling, followed by human validation. The model achieved high agreement with human annotators (~85% for Risk Categories, ~98% for Domains).

C. Experimental Setup

• Target Models: Evaluated 6 SOTA LLMs (4 open-source: Llama-3.1-8B, Gemma-2-9B, Qwen2.5-7B, Ministral-8B; 2 closed-source: GPT-4o-Mini, GPT-4.1-Nano).
• Red Teaming Methods: Tested four attack strategies: Direct, Zero-Shot, Human Jailbreak, and RainbowPlus (an adaptive quality-diversity search method).
• Metrics:
  • Attack Success Rate (ASR): Percentage of prompts successfully bypassing safety filters (evaluated by Llama-Guard-3-8B).
  • Rejection Rate (RR): Percentage of benign prompts incorrectly refused (evaluated by GPT-4o).

3. Key Contributions

1. RedBench Dataset: A unified, open-source resource aggregating 37 datasets into a standardized framework with 22 risk categories and 19 domains.
2. Comprehensive Analysis: A detailed mapping of the current red teaming landscape, revealing significant imbalances (e.g., over-representation of "Abusive Content" and "General Knowledge," under-representation of "Election Interference" and "Religion").
3. Modern Baselines: Established robust performance baselines for current SOTA models, facilitating comparative studies.
4. Open-Source Release: Publicly available dataset and evaluation code to promote reproducibility.

4. Key Results

A. Refusal Benchmarks (Over-Defensiveness)

• Variability: Rejection rates varied significantly across models.
• Llama-3.1-8B-Instruct exhibited the highest average Rejection Rate (28.53%), indicating an overly conservative safety alignment that compromises usability, particularly in General Knowledge domains.
• Gemma-2-9B-It showed the lowest average Rejection Rate (13.46%), suggesting a more permissive stance.
• GPT-4.1-Nano showed high rejection on specific datasets (ORBench: 35.18%), highlighting dataset-specific prompt sensitivity.

B. Attack Benchmarks (Vulnerability)

• Open-Source vs. Closed-Source: Open-source models were significantly more vulnerable.
  • Ministral-8B and Llama-3.1-8B suffered near-total failure against advanced attacks, with RainbowPlus achieving ASRs of 97.81% and 96.25% respectively.
  • Closed-source models (GPT-4.1-Nano, GPT-4o-Mini) demonstrated robust resistance, with GPT-4.1-Nano maintaining an ASR of only 6.88% against RainbowPlus.
• Method Effectiveness: RainbowPlus was the most potent attack method across all models, outperforming Human Jailbreaks and Zero-Shot attacks.
• Domain Vulnerabilities:
  • Environment and Nutrition domains showed surprisingly high susceptibility (ASR > 66% for RainbowPlus), suggesting safety training gaps in these specialized contexts.
  • Economic Harm and Extremism were also high-risk categories.

5. Significance and Implications

• Standardization: RedBench provides the first unified framework for comparing LLM safety across diverse risks and domains, addressing the fragmentation in current research.
• Safety-Usability Trade-off: The results highlight a critical tension: models with high safety (low ASR) often suffer from high over-refusal (low usability), particularly in open-source models.
• Gap Identification: The analysis reveals that current datasets neglect critical areas like Election Interference and specific domains like Religion and Travel, necessitating targeted future research.
• Model Architecture Insights: The stark difference between open-source and closed-source performance suggests that advanced safety alignment techniques (likely used by closed-source providers) are crucial for real-world deployment.
• Future Research: The dataset enables the development of more robust defense mechanisms and the creation of balanced red teaming strategies that cover underrepresented domains.

Code and Data Availability: The dataset and evaluation code are open-sourced at https://github.com/knoveleng/redeval.