Reducing Biases in Record Matching Through Scores Calibration

1. Problem Definition

Record matching (or entity resolution) is the task of identifying records from different datasets that refer to the same real-world entity. While often framed as a binary classification problem (match vs. non-match), practical systems typically output a real-valued matching score (e.g., probability of a match) rather than a hard binary label. These scores are used for thresholding, ranking, or human-in-the-loop review.

The Core Issue:
Existing fairness research in record matching primarily focuses on binary outcomes at a fixed threshold (e.g., Demographic Parity at $\theta=0.5$). The authors argue this approach is insufficient because:

1. Threshold Dependence: A model may appear fair at one threshold but exhibit significant bias at another.
2. Hidden Disparities: Aggregated metrics like AUC can mask systematic differences in score distributions between groups (e.g., minority vs. majority). A model might have identical AUCs for two groups but produce systematically lower scores for the minority group across the entire distribution, leading to unfair downstream decisions.
3. Score Quality: In applications where scores are used directly (e.g., ranking top candidates), disparities in score quality cause unfair treatment even without explicit thresholding.

Goal: The paper aims to define a threshold-independent notion of score bias and propose methods to calibrate these scores to reduce disparities across demographic groups without retraining the underlying model.

2. Methodology

The authors propose a two-stage approach: defining a new bias metric and developing model-agnostic post-processing calibration algorithms.

A. Threshold-Independent Bias Metric

The paper extends standard fairness criteria (Demographic Parity - DP, Equal Opportunity - EO, Equalized Odds - EOD) from binary outputs to continuous score functions.

• Definition: The bias of a score function $s$ with respect to a metric $\Phi$ (e.g., TPR, PR) is defined as the integral of the absolute difference between group-wise metrics over all possible thresholds:
  $$\text{bias}(s, \Phi) = \int_{0}^{1} |\Phi_b(s, \theta) - \Phi_a(s, \theta)| \, d\theta$$
• Interpretation: This measures the cumulative area between the performance curves (e.g., TPR vs. Threshold) of the minority and majority groups. A value of 0 implies perfect fairness across all thresholds.

B. Calibration Algorithms

The authors propose two model-agnostic post-processing methods that require only an unlabeled calibration set (or the query set itself) and the original score function.

1. Calib (for Demographic Parity)

• Objective: Align the score distributions of the minority and majority groups to a common target distribution to satisfy DP.
• Mechanism: Uses Wasserstein Barycenter alignment.
  • It treats the score distributions of the two groups as probability measures.
  • It computes a barycenter distribution $\tilde{P}$ that minimizes the average Wasserstein distance to both group distributions.
  • It maps individual scores to this barycenter using a quantile-based optimal transport map.
  • Formula: $\tilde{Q}(p) = \alpha Q_a(p) + (1-\alpha)Q_b(p)$, where $Q$ is the quantile function and $\alpha$ is the group proportion.
• Key Feature: It is label-agnostic (does not require ground-truth match labels) and works on the raw score distribution.

2. C-Calib (for Equal Opportunity / Equalized Odds)

• Objective: Satisfy label-dependent fairness criteria (EO/EOD), which require aligning error rates (TPR/FPR) across groups.
• Challenge: Ground-truth labels ($Y$) are often unavailable during calibration.
• Mechanism: Conditional Calibration.
  • It first estimates a decision threshold $\gamma$ (using an unsupervised method like MeanShift on the score distribution) to generate pseudo-labels ($\hat{Y}$).
  • It splits the data into strata based on these predicted labels (predicted matches and predicted non-matches).
  • It applies the Calib algorithm separately within each stratum.
  • This ensures that the score distributions for the minority and majority groups are aligned conditional on the predicted label, thereby approximating EO and EOD.

3. Key Contributions

1. New Fairness Formulation: Introduced a threshold-independent bias metric for record matching scores that captures cumulative disparities missed by single-threshold or AUC-based evaluations.
2. Calib Algorithm: A model-agnostic post-processing method using Wasserstein barycenters to eliminate Demographic Parity (DP) bias in scores. Theoretical analysis proves it converges to the optimal fair score at a parametric rate ($O(n^{-1/2})$).
3. C-Calib Algorithm: An extension of Calib that handles label-dependent fairness (EO/EOD) by conditioning on estimated labels. The paper provides theoretical bounds showing how fairness depends on sample size and label estimation error.
4. Empirical Validation: Demonstrated that state-of-the-art deep learning matchers exhibit significant score bias even when appearing fair at specific thresholds, and that the proposed methods effectively reduce this bias with minimal accuracy loss.

4. Experimental Results

The authors evaluated their methods on standard record matching benchmarks (e.g., DBLP-ACM, AMZ-GOO, FOD-ZAG) using five deep learning matchers (DeepMatch, HierGAT, DITTO, etc.).

• Bias Detection: Traditional metrics (DP/EO at fixed thresholds) often showed low bias, while the proposed threshold-independent metrics revealed substantial hidden disparities (e.g., bias values varying from 0% to 14% depending on the threshold).
• Bias Reduction:
  • Calib: Successfully reduced DP bias to near-zero across all datasets and models.
  • C-Calib: Significantly outperformed Calib on EO and EOD metrics. For example, on the AMZ-GOO dataset, C-Calib reduced EOD from ~30% (after Calib) to ~7%, whereas Calib alone failed to reduce EOD effectively.
• Accuracy Preservation (Risk):
  • The calibration introduced minimal distortion to the original scores.
  • The drop in AUC (Area Under Curve) was negligible (typically < 1.0%, often < 0.3%), indicating that the ranking quality and utility of the matchers were preserved.
  • C-Calib generally showed slightly better stability in AUC preservation compared to Calib.
• Dataset Sensitivity: The methods work best when the calibration dataset $D$ matches the distribution of the query data. Using training data as $D$ when the test distribution differs slightly resulted in less effective bias reduction.

5. Significance and Impact

• Beyond Binary Fairness: This work shifts the paradigm of fairness in record matching from binary decision fairness to score distribution fairness, which is critical for systems relying on ranking, threshold tuning, or human review.
• Practical Deployment: The proposed methods are post-processing and model-agnostic. They do not require access to the model's internal parameters or training data, making them highly applicable to proprietary black-box models and existing production systems.
• Theoretical Rigor: The paper provides formal theoretical guarantees regarding the convergence of the bias and the risk (score distortion), grounding the empirical success in optimal transport theory.
• Addressing Label Scarcity: The C-Calib approach offers a novel solution for achieving label-dependent fairness (EO/EOD) in settings where ground-truth labels are unavailable, a common constraint in real-world entity resolution.

In summary, the paper provides a robust framework for detecting and mitigating subtle, threshold-independent biases in record matching scores, ensuring that fairness is maintained across the entire spectrum of decision-making processes, not just at a single cutoff point.