SCOPE: A Dataset of Stereotyped Prompts for Counterfactual Fairness Assessment of LLMs

1. Problem Statement

Large Language Models (LLMs) are increasingly integrated into critical decision-making systems, raising concerns about fairness and the potential amplification of societal biases. While previous research has demonstrated that LLM outputs can shift based on demographic references in prompts, existing resources for evaluating these disparities suffer from significant limitations:

• Limited Scale and Diversity: Many datasets (e.g., CrowS-Pairs, StereoSet, WinoBias) rely on small, manually crafted sentence pairs or fixed templates, offering narrow linguistic coverage.
• Lack of Counterfactual Rigor: Many benchmarks do not strictly maintain semantic equivalence while varying only the sensitive attribute, making it difficult to isolate bias from content changes.
• Absence of Communicative Intent: Existing resources rarely account for how different interaction styles (e.g., asking a question vs. giving a direction) influence model behavior, despite evidence that prompt formulation significantly impacts LLM outputs.
• Narrow Scope: Most datasets focus on a limited set of bias dimensions (often just gender or race) and lack the combinatorial diversity needed for large-scale, reproducible analysis.

2. Methodology

The authors introduce SCOPE (Stereotype-Conditioned Prompts for Evaluation), a large-scale dataset generated through a multi-stage pipeline designed to ensure semantic alignment, linguistic diversity, and intent conditioning.

A. Knowledge Base Construction

• Source: The process began with CrowS-Pairs, a dataset of 1,508 sentence pairs contrasting stereotypical and anti-stereotypical formulations across nine bias dimensions.
• Abstraction: Instead of using raw sentences, the authors used GPT-4O Mini to abstract each pair into a structured triple: <Topic, Disadvantaged Group, Advantaged Group>.
  • Example: ("Gay men are emotionally fragile", "Straight men are emotionally fragile") $\rightarrow$ <emotional sensitivity, gay men, straight men>.
• Validation: The extracted triples underwent rigorous manual validation by the authors to ensure fidelity to the original meaning and correct identification of groups. This resulted in a validated knowledge base of 1,508 triples.

B. Counterfactual Prompt Generation

• Intent Conditioning: To capture realistic user interactions, the generation process was conditioned on four distinct communicative intents derived from a developer-agent taxonomy:
  1. Question
  2. Recommendation
  3. Direction
  4. Clarification
• Generation Process: For each triple and intent, the system generated 10 prompts for the disadvantaged group and 10 counterfactual prompts for the advantaged group using GPT-4O Mini.
• Constraints: The generation prompts were iteratively refined to enforce:
  1. Strict Semantic Equivalence: The two prompts in a pair must differ only in the referenced social group.
  2. Lexical Diversity: Prompts within a triple must vary syntactically and lexically to avoid repetition.
  3. Reproducibility: The process must be deterministic and structured.
• Output: This yielded 20 prompt pairs per triple.

C. Dataset Processing

• The raw tabular data was post-processed into a structured JSONL format.
• Each record encapsulates a complete counterfactual pair, including metadata (bias type, intent, topic, group labels) and the specific prompt sentences, facilitating easy integration into fairness assessment pipelines.

3. Key Contributions

• Scale and Scope: SCOPE contains 241,280 prompts organized into 120,640 counterfactual pairs.
• Comprehensive Coverage:
  • Topics: 1,438 distinct topics.
  • Bias Dimensions: 9 dimensions (Race-Color, Gender, Socioeconomic Status, Nationality, Religion, Age, Sexual Orientation, Physical Appearance, Disability).
  • Demographic Groups: 1,536 unique group labels.
  • Intents: 4 communicative intents (Question, Recommendation, Direction, Clarification).
• Intent-Aware Design: Unlike prior work, SCOPE explicitly conditions prompts on communicative intent, allowing researchers to study how bias manifests differently across interaction styles.
• Open Resource: The dataset, generation scripts, and transformation tools are publicly available to support reproducible research.

4. Results and Statistics

The paper presents descriptive statistics confirming the dataset's balanced and diverse nature:

• Distribution: Prompts are evenly distributed across intents (60,320 per intent).
• Bias Dimensions: The largest segments focus on Race-Color (82,480 prompts, 454 groups) and Gender (41,920 prompts, 238 groups), but all nine dimensions are substantially represented.
• Granularity: The dataset supports fine-grained analysis beyond binary groupings (e.g., distinguishing between specific nationalities or socioeconomic statuses).

Illustrative Experiment:
The authors conducted a pilot study using the Gemini-2.5-flash model on a subset of the data (Question/Race and Recommendation/Gender).

• Findings: Even with identical topics and intents, the model produced outputs with distinct explanations, emphases, and lexical overlaps.
• Observation: In the "Recommendation" intent, the model's reformulations and interpretive choices were not fully symmetric across gender variants, suggesting potential differential treatment that varies by interaction style.

5. Significance and Usage Scenarios

SCOPE provides a controlled, semantically aligned foundation for three primary research scenarios:

1. Counterfactual Fairness Testing: Systematically detecting asymmetric behaviors (e.g., toxicity, stance shifts) when only the sensitive attribute changes.
2. Robustness Evaluation: Assessing whether models behave consistently when sensitive attributes are perturbed, identifying behavioral drift across model versions.
3. Intent-Conditioned Bias Detection: Investigating how stereotypes surface differently depending on whether the user is asking a question, seeking a recommendation, or requesting clarification.

Conclusion:
SCOPE addresses the critical gap in current fairness benchmarks by offering a large-scale, intent-aware, and linguistically diverse dataset. It enables researchers to move beyond static sentence pairs to dynamic, realistic interaction simulations, thereby advancing the evaluation of fairness, robustness, and counterfactual consistency in Large Language Models.