Technical Summary: Same System, Opposite Verdicts: Metric Discretion in AI Ethics Audits and the Limits of Disclosure

1. Problem Statement

The paper addresses a critical gap in AI governance: while regulators increasingly mandate quantified fairness audits for high-risk AI systems (e.g., the EU AI Act, NYC Local Law 144), no standard exists to dictate how those fairness numbers are computed. This lack of a canonical measurement standard creates a "discretion window" where an honest auditor, using defensible methods, can report vastly different fairness verdicts for the same system.

The core problem is that current disclosure practices document the computation of a single figure but remain silent on the selection of that figure. Consequently, a published fairness score is a selection from a wide range of defensible possibilities, and readers cannot determine if the reported number represents a representative finding or a favorable outlier. This issue is distinct from "fairwashing" (deceptive explanations) or "audit-washing" (cosmetic processes); it concerns the structural ambiguity inherent in honest audits where multiple valid metrics yield contradictory results.

2. Methodology

The paper employs a three-part empirical strategy to measure the magnitude of this discretion and evaluate potential remedies.

Study 1: The Discretion Window (Multiverse Analysis)

• Design: The audit process is treated as a "multiverse." The study enumerates every defensible specification for four decision systems: one deployed instrument (COMPAS pretrial risk) and three canonical benchmarks (Census income, German credit, ACS employment).
• Scope: A total of 91,572 specifications were generated by varying seven axes:
  1. Metric Family (Selection): 8 families (e.g., statistical parity, equal opportunity, predictive parity).
  2. Decision Threshold (Definition): 5 rules (e.g., fixed 0.5, top decile).
  3. Reference Population (Definition): 3 defensible filters per system.
  4. Group Comparison Rule (Definition): 4 conventions for handling multi-level attributes.
  5. Reporting Period (Consistency): 5 years × 3 states (for census data).
  6. Model Class (Benchmark): 3 learners (logistic regression, gradient boosting, random forest).
  7. Resampling Seed (Benchmark): 3 independent draws to measure sampling error.
• Outcome Measures: The study calculates the "discretion window" (range of impact ratios), verdict instability (pass/fail against the 0.80 four-fifths rule), and direction reversal (which group is identified as disadvantaged).

Study 2: Analysis of Published Disclosures

• Corpus: 6,797 documents from two regimes: 25 mandatory Algorithmic Impact Assessments (Canada) and 6,772 voluntary Model Cards (Hugging Face).
• Procedure: An automated screen flagged 76 candidates containing quantified fairness claims, which were manually verified. The study coded whether these disclosures stated the metric name, evaluation population, compared groups, decision threshold, and alternative measures not chosen.

Study 3: Quantifying the Value of Disclosure

• Logic: Using the enumerated space from Study 1, the study simulates progressive disclosure. It calculates how much the range of possible reported figures narrows as an auditor discloses specific elements (e.g., metric family, then threshold, then population).
• Goal: To determine what percentage of the "discretion window" could be closed by mandatory disclosure requirements similar to those in financial reporting (reconciliation and consistency).

3. Key Results

The Width of the Discretion Window

• Magnitude: Across all 7 system/attribute pairs, the reported impact ratio spans a median range of 0.94 on a scale bounded at 1.0. Even the narrowest window (German credit) spans 0.79.
• Verdict Instability: In 100% of the 7 pairs, the space of defensible specifications contains both a "pass" and a "fail" against the regulatory four-fifths rule (0.80 threshold).
• Direction Reversal: In 100% of the 7 pairs, the identity of the "disadvantaged" group reverses depending on the specification chosen.
• Comparison to Sampling Error: The movement caused by varying specifications is 6 to 30 times larger than the movement caused by sampling error (redrawing the data). The median ratio is 17:1.

Drivers of Variation

• Metric Selection Dominates: The choice of metric family accounts for 14% to 94% of the variance in the reported figure (median 45%). This is the single largest driver and the least disclosed element.
• Definition Choices: Threshold, population, and group rules contribute 0–23% of the variance.
• Consistency: Reporting period contributes at most 2.4%.

Current Disclosure Practices

• Rarity: Only 0.56% (38 out of 6,797) of documents contained a confirmed quantified fairness claim.
• Omissions: Among the 38 confirmed disclosures:
  • 100% named the measure.
  • 68.4% stated the evaluation population.
  • 71.1% identified compared groups.
  • 0% stated a decision threshold, reported a prior-period figure, or named the alternative measure not chosen.
• Mandatory Regimes: None of the 25 Canadian Algorithmic Impact Assessments reported a quantified fairness measure.

The Value of Disclosure (Remedy)

• Metric Family Disclosure: Disclosing the metric family alone closes 44% of the discretion window.
• Full Reconciliation: Disclosing the metric family, threshold, reference population, and group rule closes 78% of the window.
• Consistency: Adding reporting period consistency brings the total closure to 82%.
• Residual: A residual range of 0.16 to 0.21 remains even with full disclosure, representing variation due to sampling error and model class choices that transparency alone cannot resolve.

4. Key Contributions

1. First Measurement of Audit-Level Metric Discretion: The paper provides the first empirical quantification of how much the reported fairness figure moves across defensible specifications on a deployed instrument and canonical benchmarks.
2. Decomposition of Variance: It empirically weights the auditor's choices, demonstrating that metric selection is the dominant source of variance, far exceeding sampling error.
3. Empirical Baseline for Disclosure: It establishes that current practice documents the "how" (computation) but not the "what" (selection), leaving readers unable to reconstruct the range of possible outcomes.
4. Quantified Bounds on Transparency: It places a measured bound on the efficacy of disclosure, showing that while reconciliation can close ~78% of the discretion, a non-zero residual remains that requires substantive standards or tighter estimation to address.

5. Significance and Claims

The paper argues that the "same system, opposite verdicts" phenomenon is not an anomaly but a structural feature of current AI ethics auditing. The significance lies in shifting the regulatory focus from the impossible task of agreeing on a single "correct" fairness metric to the achievable task of governing the disclosure of choices.

• For Regulators: The paper claims that regulators can immediately mandate the disclosure of four elements (metric family, operational definition, reference population, and comparison rule) without needing to settle which metric is correct. This would render the discretion "legible" and close the majority of the range where verdicts flip.
• For Theory: It reframes the "impossibility results" of fairness (which state metrics cannot coexist) as a policy problem of disclosure rather than a technical failure. The magnitude of the discretion window transforms the issue from a theoretical debate into a practical governance challenge.
• For Practice: It proposes a "minimum disclosure set" that costs nothing beyond recording choices already made, allowing boards and auditors to challenge reported figures based on the range of alternatives.

The paper concludes modestly that while transparency cannot eliminate all uncertainty (the residual remains), making the discretion legible is a necessary and immediate step that does not require waiting for a canonical standard that may never exist.