Technical Summary: Fair and Calibrated Toxicity Detection with Robust Training and Abstention

Problem Statement

Toxicity classifiers frequently learn spurious correlations between identity mentions (e.g., "white," "muslim," "gay") and toxicity labels. Because hate speech often co-occurs with mentions of protected groups, models treat identity terms as signals for toxicity. This leads to systematic false positives where neutral or anti-racist content mentioning these identities is disproportionately penalized.

Standard evaluations typically focus on ranking metrics (such as BPSN/BNSP AUC) while treating calibration and abstention as separate quality concerns. This paper argues that such a separation is misleading for three reasons:

1. Calibration Disparity is a Fairness Violation: If confidence scores correspond to different accuracies across groups, threshold-based decisions are inherently biased.
2. Abstention Efficacy is a Fairness Property: Confidence-based deferral only functions as a safety mechanism if confidence tracks correctness uniformly across all subgroups.
3. Post-hoc Dependency: Interventions like temperature scaling or threshold optimization succeed or fail based on how the training method shaped the underlying probability distribution.

The paper proposes an integrated framework evaluating toxicity classifiers across three axes: ranking, calibration, and abstention.

Methodology

Data

The study utilizes the Civil Comments dataset (Jigsaw), stratified and downsampled to 200,000 examples. Groups are assigned based on the first identity mentioned in the comment. The test set includes 18,217 background examples (no identity mentioned) and eight identity groups with sufficient support (e.g., white, black, muslim, gay/lesbian, jewish, christian, female, male). Groups with fewer than 50 examples are excluded to ensure stable bootstrap estimates. HateXplain is used for zero-shot transfer evaluation to assess generalization.

Training Methods

Three training-time interventions are compared:

1. Empirical Risk Minimization (ERM): A baseline fine-tuned on DistilBERT-base-uncased, minimizing average loss uniformly without subgroup awareness.
2. Reweighted ERM: Applies per-example weights based on group frequency ($w_i = N / (G \cdot n_{g_i})$), clipped to prevent rare groups from dominating gradients.
3. Group Distributionally Robust Optimization (Group DRO): Uses adaptive per-group weights updated via a minimax objective to focus on the highest-loss group.

Post-hoc Interventions

Three mechanisms are applied to trained outputs:

1. Temperature Scaling: Learns a scalar $T$ to correct uniform miscalibration.
2. Confidence-based Abstention: Computes risk-coverage curves by retaining the top $c$-fraction of predictions based on confidence ($max(p, 1-p)$).
3. Per-identity Threshold Optimization: Grid searches thresholds to minimize absolute error gaps between subgroups and background.

Evaluation Metrics

The framework employs paired bootstrap 95% confidence intervals ($n=1000$) for all metrics:

• Ranking: Subgroup AUC, BPSN (Benign Positive, Subgroup Negative), and BNSP (Benign Negative, Subgroup Positive).
• Calibration: Expected Calibration Error (ECE) per subgroup and the Calibration-Fairness Gap ($\Delta ECE = ECE_{subgroup} - ECE_{background}$).
• Tail Behavior: Percentage of benign examples with high predicted toxicity ($p > 0.9$ and $p > 0.99$).
• Abstention: Risk-coverage curves and residual error gaps at various coverage levels.

Key Results

1. Hidden Calibration Disparity in ERM

While ERM achieves near-perfect aggregate calibration (ECE $\approx$ 0.013) and strong overall AUC (0.940), it exhibits significant calibration disparity across all identity subgroups.

• Every identity subgroup is significantly miscalibrated compared to the background (e.g., Jewish subgroup ECE gap +0.134; White subgroup +0.077).
• The model is overconfident on identity-mentioning content. A prediction of $p=0.85$ on a comment mentioning a protected group does not carry the same accuracy as $p=0.85$ on background content.
• This disparity is invisible to standard ranking metrics like BPSN.

2. Training Interventions Reshape, Not Eliminate, Disparity

Fairness methods alter the trade-offs between axes rather than solving all issues simultaneously:

• Reweighted ERM: Improves ranking fairness (BPSN AUC increases by +0.06 to +0.12) but worsens the calibration-fairness gap significantly (up to +0.232). It creates a "bimodal" distribution where most benign identity comments are scored near zero, but a small tail is pushed to extreme confidence ($p > 0.99$).
• Group DRO: Eliminates calibration disparity (gaps cross zero) but only by becoming uniformly miscalibrated globally (ECE rises to 0.118). It decouples confidence from correctness, causing the risk-coverage curve for abstention to rise rather than fall.

3. Post-hoc Interventions Inherit Training Failure Modes

• Temperature Scaling: Fails for all methods. ERM is already globally calibrated; Reweighted ERM has selective, subgroup-specific miscalibration that a scalar cannot fix; DRO has a structural decoupling of confidence and correctness.
• Threshold Optimization: Cannot repair the "tail-heavy" bias where a small fraction of benign comments are assigned extreme toxicity scores.
• Abstention: Works well under ERM for background content but is unfair. It fails to protect identity-mentioning content because the errors are "confidently-wrong" (high confidence, incorrect label) rather than uncertain. Under DRO, abstention breaks entirely as deferring low-confidence predictions does not preferentially defer wrong ones.

4. Qualitative Failure Cases

Benign anti-racist or descriptive comments mentioning "white" are frequently assigned high toxicity scores ($p > 0.9$) by ERM and Reweighted ERM. These "confidently-wrong" predictions survive confidence filters, explaining why abstention gaps persist for white-mentioning content even at high coverage levels.

Contributions and Significance

The paper makes four primary contributions:

1. Multi-Method Comparison: A rigorous, bootstrap-validated comparison of ERM, Reweighted ERM, and Group DRO combined with post-hoc interventions.
2. Calibration as Fairness: Introduction of the calibration-fairness gap, demonstrating that aggregate calibration can mask severe subgroup-level violations.
3. Non-Dominated Trade-offs: Characterization of training methods as occupying distinct positions in the fairness space:
   • ERM: Worst ranking fairness, but cleanest abstention behavior.
   • Reweighted ERM: Best ranking fairness, but worst calibration disparity and introduction of confidently-wrong errors.
   • Group DRO: Fixes calibration parity but breaks abstention utility.
4. Post-Hoc Dependency: Demonstration that the efficacy of safety mechanisms (scaling, abstention) is determined by the training method's specific failure modes.

Significance:
The paper argues that Socially Responsible AI (SRAI) fairness requires a multi-axis framework. Methods that appear equivalent on aggregate ranking metrics differ sharply in their failure modes, which determine real-world harm. Specifically, fairness interventions that improve ranking often introduce "confidently-wrong" predictions on benign identity content that no post-hoc intervention can repair. The authors conclude that on this benchmark, Reweighted ERM is the most defensible practical choice due to its balance of ranking fairness, calibration, and abstention compatibility, though it still introduces new failure modes not present in the baseline ERM.