Technical Summary: Perturbation Effects on Robustness and Individual Fairness

Problem Statement

Deep neural networks (DNNs) are vulnerable to adversarial perturbations that can degrade both prediction robustness and individual fairness. However, existing evaluation protocols typically assess these dimensions in isolation. This separation obscures critical failure modes where a model may appear robust but unfair, or fair but unrobust. Specifically, single-axis metrics often miss coupled failure regimes:

• Robust Biased (RB): The model remains correct (robust) but produces inconsistent predictions for semantically equivalent individuals (unfair).
• Unrobust Fair (UF): The model produces consistent predictions for similar individuals (fair) but both are incorrect (unrobust).
• Unrobust Biased (UB): The model fails on both counts.

Current methods often rely on geometric proximity (e.g., cosine similarity) or white-box gradient access, which may fail to preserve semantic truth conditions or are inapplicable to black-box API deployments. Furthermore, many fairness tests allow semantic drift, making it impossible to distinguish between a model failure and a change in the ground truth.

Methodology

1. Formal Definition: Robust Individual Fairness (RIF)

The authors propose Robust Individual Fairness (RIF) as a unified criterion grounded in truth-conditional semantics.

• Semantic Equivalence: Two inputs $v$ and $v'$ are considered semantically equivalent if they share identical truth conditions. The paper operationalizes this via bidirectional entailment using a Natural Language Inference (NLI) model. An edit is admissible only if $v \models v'$ and $v' \models v$.
• The Criterion: A classifier $f$ satisfies $\epsilon$-RIF if, for every semantically equivalent counterpart $v'$ and every admissible perturbation $\delta$, the prediction error remains bounded by $\epsilon$. This requires the model to be both correct (anchored to ground truth) and consistent (invariant across similar individuals) under meaning-preserving perturbations.

2. Framework: RIFair

To expose RIF violations, the authors introduce RIFair, a black-box adversarial framework.

• Decoupled Substitution Strategy: Unlike coupled attacks that apply identical substitutions to similar pairs, RIFair applies distinct but semantically equivalent substitutions to each member of a pair (e.g., mapping "talk" $\to$ "discuss" for one input and "talk" $\to$ "chat" for the other). This isolates failures caused by heterogeneous surface-form sensitivity.
• Two-Stage Pipeline:
  1. Similarity Construction: Sensitive attributes (e.g., gender, race) are masked and replaced with alternatives to create semantically matched pairs $(v, v')$ while keeping non-sensitive content identical.
  2. Generate-Filter Perturbation: A Large Language Model (LLM) generates candidate paraphrases for tokens, which are then filtered by an NLI model to ensure bidirectional entailment (semantic preservation).
• Optimization Objective: RIFair minimizes a joint objective function combining robustness loss (cross-entropy with ground truth) and fairness discrepancy (distance between predictions). By varying weights ($\alpha, \beta$), the framework targets specific quadrants: RB, UF, or UB.
• Query Efficiency: The method uses occlusion saliency to rank tokens and searches only extreme candidates (min/max confidence) to minimize model queries, making it suitable for black-box settings.

Key Contributions

1. Unified Criterion (RIF): Formalizes a joint requirement for robustness and individual fairness under semantic-preserving perturbations, addressing the misalignment of existing single-axis metrics.
2. Semantic Rigor: Grounds perturbations in formal semantics via bidirectional entailment, ensuring that generated adversarial instances preserve original truth conditions, unlike methods relying solely on geometric proximity.
3. Black-Box Framework (RIFair): Proposes a decoupled substitution strategy that effectively exposes hidden vulnerabilities (RB and UF) that coupled attacks miss.
4. Empirical Evidence: Demonstrates that robustness-only and fairness-only metrics frequently fail to detect multi-dimensional adversarial failures, providing a case for joint evaluation.

Experimental Results

Experiments were conducted on diverse transformer architectures (BERT, RoBERTa, DistilBERT, DeBERTa-v3) across textual datasets (Bias in Bios, Jigsaw, SST-2) and a verbalized tabular dataset (Census Income).

• Misalignment of Metrics: Table 3 shows that model rankings vary significantly depending on the evaluation axis. For instance, DistilBERT ranked highest in robustness ($R$), BERT in fairness ($F$), but DeBERTa-v3 achieved the strongest joint resistance ($RIF$).
• Detection of Hidden Failures: RIFair successfully identified failure modes missed by single-axis metrics.
  • Robustness ($R$) was insensitive to RB scenarios (where predictions are stable but biased).
  • Fairness ($F$) was insensitive to UF scenarios (where predictions drift together, maintaining consistency but losing accuracy).
  • RIF provided a consistently sensitive signal across all failure modes (RB, UF, UB).
• Semantic Preservation: Ablation studies confirmed that entailment-based constraints yield significantly higher semantic equivalence scores ($S_{eq} \approx 0.97$) and human-aligned judge scores compared to geometric (cosine) constraints, which often result in semantic drift ($S_{eq} \approx 0.33$).
• Decoupling Efficacy: The decoupled strategy was found to be $2.3\times$ more effective at exposing RB violations and $3.0\times$ more effective at exposing UB violations compared to coupled attacks.
• Defense: Adversarial retraining using RIFair-generated instances substantially reduced failure rates across all categories (e.g., reducing UB failure rates by $\approx 16\%$).

Significance and Claims

The paper argues that robustness and fairness are not independent properties and that evaluating them in isolation leads to an incomplete and potentially misleading assessment of model trustworthiness.

• Necessity of Joint Evaluation: The authors claim that RIF is a necessary criterion for trustworthy model assessment, particularly in high-stakes applications (e.g., toxic content identification, sentiment analysis, income prediction) where both reliability and equity are essential.
• Limitations: The authors modestly acknowledge that their reliance on NLI models for semantic equivalence is a proxy that may encode biases or struggle with rare entities and long contexts. They suggest future work should incorporate human-in-the-loop oversight and improved masking functions.
• Core Insight: The primary contribution is the demonstration that models can exhibit "compound failures" under perturbations that single-axis metrics cannot detect, necessitating a unified framework like RIF and RIFair to surface these vulnerabilities.