Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation

1. Problem Statement

While red teaming has become a standard practice for ensuring the safety and reliability of Large Language Models (LLMs), current research and industry practices are heavily skewed toward technical benchmarks and attack success rates (ASR). This focus leaves the socio-technical practices underlying red teaming under-examined. Specifically, there is a lack of understanding regarding:

• How adversarial datasets are defined, created, and evaluated by practitioners.
• How the conceptualization of "risk" and "harm" is constructed through data work rather than being a fixed, objective concept.
• The extent to which current red teaming practices overlook context, interaction types (e.g., multi-turn vs. single-turn), and user specificity (e.g., marginalized communities).

The authors argue that adversarial datasets are not neutral; they embody the assumptions, values, and disciplinary biases of their creators, which ultimately shapes what risks are detected and what mitigations are implemented.

2. Methodology

The study employs a qualitative research approach grounded in Human-Computer Interaction (HCI) and Critical Data Studies.

• Data Collection: The authors conducted 22 semi-structured interviews with AI practitioners who design, build, or reuse red teaming datasets for LLMs.
• Recruitment Strategy: Participants were recruited from open-source communities (Hugging Face, Papers with Code), academia, and industry. The researchers identified 107 potential candidates based on their public red teaming datasets, ranked them by visibility (downloads, stars, citations), and sampled 76.
• Participant Demographics:
  • Geography: 8 countries across 3 continents.
  • Background: Primarily Computer Science (PhD students, researchers, engineers).
  • Practice Types: 12 participants created datasets from scratch; 10 reused existing datasets.
  • Representation: 15 PhD students, 2 industry practitioners (notably low industry participation), and others from startups/academia.
• Analysis: The authors used thematic analysis on interview transcripts. Two authors independently coded the data, developed a codebook, and iteratively refined 47 sub-codes grouped under 21 parent codes to identify patterns in how practitioners conceptualize and execute red teaming.

3. Key Contributions

The paper makes three primary contributions to the fields of AI Safety and HCI:

1. Empirical Evidence of Practice: It provides the first systematic empirical study of how practitioners actually conceptualize, develop, and evaluate red teaming datasets, moving beyond theoretical frameworks to real-world workflows.
2. Identification of Blind Spots: It reveals how practitioners' disciplinary backgrounds (e.g., Reinforcement Learning vs. NLP) and motivations lead to specific blind spots, particularly the overlooking of context, interaction types (multi-turn), and user specificity.
3. HCI Research Opportunities: It proposes three concrete pathways for HCI researchers to expand red teaming practices, shifting the focus from purely technical exploits to socio-technical evaluations that center on real-world impact.

4. Key Results and Findings

A. Conceptualization of Red Teaming

Practitioners' approaches are heavily influenced by their disciplinary training:

• Exploration/RL Framing (Majority): Viewed red teaming as an open-ended search problem to navigate a vast prompt space. Success is measured by coverage, diversity, and efficiency (ASR). The goal is to find any failure in an unbounded space.
• Inappropriate Content Detection/NLP Framing (Minority): Viewed red teaming as a supervised classification task (e.g., hate speech detection). Success is measured against predefined ground-truth labels (e.g., F1 score).

B. Dataset Creation Strategies

Three distinct pathways for creating adversarial datasets were identified:

1. Repurposing Existing Datasets: Reusing benchmarks like AdvBench or BeaverTails. This leads to inherited definitions of harm and often results in redundant, low-diversity data that fails to capture novel risks.
2. Creating from Scratch: Designing original data. This allows for more control but requires significant resources.
3. Deriving from Human Interactions: Collecting "in-the-wild" jailbreak attempts. This captures authentic, context-rich adversarial cases but introduces subjectivity in categorization.

Critical Finding: Reusing datasets often locks practitioners into the original dataset's taxonomies, limiting the scope of risks examined. For example, datasets often lack specific categories for LGBTQ+ experiences, youth concerns, or dialect-specific biases.

C. Evaluation Challenges

• Uncertainty in Harm: Determining what counts as "adversarial" is difficult due to inconsistent model behaviors, mixed-content outputs, and the difference between potential harm and actual actionable harm (e.g., a model giving fictional bomb instructions vs. real ones).
• Automation vs. Human Judgment:
  • Automation: LLMs and classifiers are used for scale but suffer from inconsistency, cost, and API restrictions (e.g., researchers losing API access for generating harmful content).
  • Human Evaluation: Perceived as more trustworthy for nuance but suffers from a lack of standardized selection criteria for annotators and high recruitment costs.
• The "Black Box" of Evaluation: The reliance on automated judges means that the definitions of harm encoded in those judges (often trained on specific cultural or linguistic data) dictate what is considered a safety failure.

D. Socio-Technical Blind Spots

The study identifies three critical areas where current practices fail:

1. Overlooking Context: Risks are treated as abstract and generic. Practitioners rarely consider the deployment environment, cultural norms, or legal frameworks (e.g., copyright laws vary by country).
2. Overlooking Interaction Type: Most research focuses on single-turn prompts. However, multi-turn conversations can reveal vulnerabilities (e.g., gradual jailbreaking) that single-turn tests miss.
3. Overlooking User Specificity: Risks are assessed for a "generic" user. Specific vulnerabilities of marginalized groups (youth, non-native speakers, specific dialects) are rarely targeted or evaluated.

5. Significance and Recommendations

The paper argues that red teaming is not merely a technical audit but a socio-technical practice that constructs the very definition of safety. The current narrow focus on technical exploits limits the ability to detect systemic harms.

Three Recommendations for HCI Researchers:

1. Design Situated Scenarios: Move away from agnostic "search" problems. Design red teaming scenarios grounded in specific communities (e.g., minors, elderly) and real-world deployment contexts to capture "compositional risks" (harm arising from combinations of actions).
2. Integrate Domain Expertise: Replace generic, broad taxonomies with risk categories defined in collaboration with domain experts and affected communities. This ensures taxonomies reflect real-world vulnerabilities rather than abstract technical categories.
3. Evaluate Compositional Risk: Shift evaluation from isolated single-turn outputs to interaction-level assessment. Analyze the dynamics of multi-turn conversations to identify escalation patterns, emotional manipulation, and cumulative harms that single-turn metrics miss.

Conclusion:
By treating red teaming as a socio-technical practice, the authors highlight the need for more inclusive, context-aware, and participatory evaluation methods. This shift is essential for developing AI systems that are safe not just in a technical sense, but in a way that respects diverse human values and real-world complexities.