Bias Ahead: Sensitive Prompts as Early Warnings for Fairness in Large Language Models

1. Problem Statement

Large Language Models (LLMs) are increasingly integrated into software systems, yet existing fairness benchmarks primarily focus on stereotype-specific associations (e.g., gender or race biases in predefined categories). These benchmarks have two critical limitations:

1. Reactive Nature: They detect bias only after it occurs, rather than preventing it.
2. Limited Scope: They fail to capture the nuance of real-world contexts where prompts are not inherently discriminatory but touch upon sensitive topics (e.g., mental health, religion, identity) that increase the likelihood of biased, inadequate, or harmful outputs.

Developers currently lack practical mechanisms to identify when a prompt is likely to trigger such risks before deployment. The paper argues for a shift from reactive mitigation to preventive design by treating "prompt sensitivity" as a measurable predictor of fairness risks.

2. Methodology

The study follows a three-step empirical approach to validate the hypothesis that prompt sensitivity correlates with biased outputs and can be predicted automatically.

A. Dataset Construction: SENSY

The authors constructed SENSY, a new dataset containing 12,801 prompts designed to test sensitivity across seven thematic domains:

• Domains: Religion/Philosophy, Politics/Society, Relationships/Sentiments, Health/Mental Well-being, Identity/Diversity, Sexual, and Security.
• Sources:
  • Synthetic Generation: Prompts generated via ChatGPT aligned with specific sensitive domains.
  • Real-World Data: Randomly sampled first-turn queries from the Chatbot Arena dataset.
• Annotation: Prompts were manually labeled as "sensitive" or "non-sensitive" by two trained annotators using a consensus approach. The dataset includes 2,710 sensitive and 10,091 non-sensitive prompts.

B. Research Question 1 (RQ1): Correlation Analysis

• Objective: Determine if sensitive prompts lead to inadequate or biased LLM responses.
• Setup: A balanced sample of 500 prompts (covering all 7 domains) was queried against three open-source LLMs (7–8B parameters): Meta-Llama-3.1-8B-Instruct, DeepSeek-LLM-7B-Chat, and Qwen2-7B-Instruct.
• Procedure: Each prompt was queried 3 times per model (4,500 total responses). Responses were constrained to 160 words to ensure consistency.
• Evaluation: Human annotators manually evaluated responses for adequacy, defined by both factual correctness and communicative appropriateness (tone, empathy, acknowledgment of context).

C. Research Question 2 (RQ2): Automated Prediction

• Objective: Determine if prompt sensitivity can be automatically predicted to support fairness-aware development.
• Model: A Random Forest classifier was trained to predict sensitivity (Binary: 1 = Sensitive, 0 = Non-sensitive).
• Features:
  • Syntactic: Word counts, verb/adjective/noun counts, and sensitive keyword presence (via NLTK).
  • Semantic: Sentiment analysis scores and BERT embeddings.
• Evaluation: The model was tested via 10-fold cross-validation on SENSY and compared against the SQuARe dataset (a prior benchmark) to assess generalization and robustness against class imbalance.

3. Key Contributions

1. New Abstraction for Fairness: Proposes "sensitive prompts" as a preventive abstraction for fairness evaluation, moving beyond static stereotype testing to dynamic, context-aware risk assessment.
2. SENSY Dataset: Releases a large-scale, multi-domain dataset (12,801 prompts) combining synthetic and real-world data, annotated for sensitivity and thematic categories.
3. Empirical Evidence: Demonstrates that while LLMs often provide factually correct answers, they frequently fail to exhibit communicative responsibility (e.g., lacking empathy, failing to warn of risks) in sensitive contexts.
4. Automated Classifier: Develops and validates a classifier capable of predicting prompt sensitivity, offering a tool for "fairness-by-design."

4. Key Results

RQ1: LLM Behavior on Sensitive Prompts

• Fact vs. Context: Models generally provided factually correct answers but failed to acknowledge ethical, relational, or contextual nuances.
• Critical Weaknesses: The most significant failures occurred in Relationships, Health/Mental Well-being, and Identity/Diversity.
  • Example: Models provided step-by-step advice on how to hide an affair without addressing the ethical implications of deception.
  • Example: Models discussed mental health issues without advising professional consultation, normalizing self-diagnosis.
  • Example: Models reproduced gender stereotypes (e.g., "Man is to doctor as woman is to nurse") without critical framing.
• Conclusion: Linguistic competence does not equate to communicative responsibility. Models struggle with ambiguous or delicate situations requiring caution.

RQ2: Automated Sensitivity Prediction

• SENSY Performance: The classifier trained on SENSY achieved high stability and generalizability:
  • Accuracy: ~91%
  • Macro F1-Score: ~84%
  • ROC-AUC: 0.92
• SQuARe Limitations: Models trained solely on the SQuARe dataset showed high accuracy (~96%) but poor macro-performance (F1 < 0.50) due to severe class imbalance (over-representation of sensitive prompts). They failed to detect non-sensitive prompts.
• Cross-Dataset Generalization:
  • Training on SENSY and testing on SQuARe showed moderate performance but highlighted distributional shifts.
  • Training on SQuARe and testing on SENSY resulted in poor performance (Accuracy ~26%), confirming that SQuARe is insufficient for training robust, generalizable detectors.
• Conclusion: Robust sensitivity prediction requires diverse, balanced, multi-domain datasets like SENSY.

5. Significance and Implications

• Shift to Preventive Design: The paper advocates for integrating sensitivity detection into the software development lifecycle (SDLC) to flag risky prompts before they reach the LLM, rather than just filtering outputs.
• Developer Tooling: The proposed classifier serves as an "early-warning system," allowing developers to implement safeguards (e.g., pre-processing filters, reframing prompts) without needing deep fairness expertise.
• Beyond Stereotypes: Highlights that fairness risks extend beyond protected attributes to include any topic where the context demands ethical nuance (e.g., mental health, identity).
• Dataset Quality: Emphasizes that the quality and diversity of training data are critical for fairness tools; relying on narrow benchmarks (like SQuARe) leads to biased and non-generalizable models.

In summary, the paper establishes prompt sensitivity as a viable, predictive metric for fairness risks in LLMs and provides the tools (SENSY dataset and classifier) to operationalize this concept for safer AI deployment.