Empowering Affected Individuals to Shape AI Fairness Assessments: Processes, Criteria, and Tools

Technical Summary: Empowering Affected Individuals to Shape AI Fairness Assessments

1. Problem Statement

Current AI fairness assessments are predominantly "expert-centric," conducted by AI researchers or regulators using predefined protected attributes (e.g., race, gender) and standardized mathematical metrics (e.g., Demographic Parity). This approach suffers from several critical gaps:

• Lack of Nuance: Existing metrics often fail to capture the diverse, context-specific, and qualitative fairness notions held by the individuals actually affected by AI decisions (decision subjects).
• Divergence of Values: Stakeholders often hold fairness values that differ significantly from the statistical metrics used by experts.
• Knowledge Gap: While recent literature calls for stakeholder involvement, there is a lack of empirical evidence regarding how laypeople formulate fairness notions, how they translate these into quantifiable criteria, and what specific types of criteria they produce.

2. Methodology

The researchers conducted a qualitative user study to bridge this gap, employing a human-centered design approach.

• Scenario & Dataset: The study used a credit rating scenario based on the German Credit dataset (1,000 instances, 20 features). A logistic regression model was trained to predict "Good" vs. "Bad" credit.
• Participants: 18 lay stakeholders (decision subjects) with no formal AI training or professional experience.
• Experimental Setup: The study utilized a custom-designed four-module interactive prototype designed to scaffold the transition from abstract values to operational metrics.
  • Module 1: Using existing outcome fairness metrics (selecting features and metrics).
  • Module 2: Combining multiple existing metrics using Boolean operators (AND/OR) or weights.
  • Module 3: Creating entirely new outcome metrics via probability-based formulas involving ground-truth, predictions, and feature conditions.
  • Module 4: Creating procedural fairness metrics (defining feature exclusion, importance, or ranking).
• Procedure: Participants first articulated fairness notions in plain language, then used the prototype to quantify these notions into operational criteria, and finally viewed real-time fairness assessment results.

3. Key Contributions

• Process Discovery: Identification of the two-phase cognitive process stakeholders use to move from "notion" to "criterion."
• Taxonomy of Criteria: An empirical characterization of the diverse types of fairness criteria produced by non-experts.
• Design Implications: A set of actionable guidelines for developing inclusive AI assessment tools and processes that support value-sensitive design.

4. Results

The study revealed two distinct phases in the creation process and a high degree of complexity in the resulting criteria.

A. The Creation Process (RQ1):

• Phase 1 (Grounding): Participants grounded their notions through four actions: (1) exploring model features, (2) establishing an evaluative standpoint (often through empathy or historical context), (3) identifying specific features to protect or exclude, and (4) binning feature values to define protective boundaries.
• Phase 2 (Translating): Participants converted these notions into measurable forms by creating metrics, setting custom fairness thresholds (mostly 0–20%), and establishing criteria priorities (e.g., "all-pass" vs. "top-k" rules).

B. Types of Criteria Produced (RQ2):
Participants created a total of 47 criteria (averaging 2.6 per person), demonstrating that a single metric is insufficient for lay stakeholders.

• Outcome Fairness:
  • Existing Metrics: Using standard metrics like Demographic Parity on non-protected features (e.g., "Job" type).
  • Combined Metrics: Using Boolean logic to merge group and individual fairness (e.g., Counterfactual Fairness AND Demographic Parity).
  • New Metrics: Developing custom probability formulas or incorporating error rates (e.g., False Negative Rate).
• Procedural Fairness: A significant portion of participants (12/18) focused on the process, defining rules for feature exclusion (unawareness), feature importance (weighting), or feature ranking.

5. Significance

This research shifts the paradigm of AI fairness from a purely mathematical optimization problem to a participatory socio-technical process.

• For AI Experts: It provides evidence that "fairness" is multi-dimensional, suggesting that mitigation strategies should move beyond single-metric optimization toward multi-objective approaches that include procedural constraints.
• For Tool Designers: It highlights the necessity of providing model transparency (XAI), allowing for custom group definitions (beyond legal protected classes), and supporting the creation of complex, non-linear criteria.
• For Policy & Ethics: It demonstrates that empowering affected individuals to define their own "fairness thresholds" and "protected groups" can lead to more inclusive and socially acceptable AI systems.