Bias Leaves a Gradient Trail: Label-Free Bias Identification via Gradient Probes on Concept Decompositions

Technical Summary: Bias Leaves a Gradient Trail

Problem Statement
Modern vision classifiers often achieve high in-distribution accuracy by exploiting spurious correlations (shortcuts) rather than causal features. While these models perform well on training data, they fail under distribution shifts where these correlations break. Existing bias mitigation and analysis methods typically require curated datasets with known spurious attributes or group labels, or they necessitate retraining the model. However, in real-world deployment scenarios, practitioners often face a "frozen" model with no access to the original training pipeline, bias annotations, or group labels. The challenge is to identify these hidden spurious concepts and mitigate them post-hoc without retraining or additional supervision.

Methodology
The authors propose a label-free, post-hoc method to identify and mitigate spurious concepts in frozen vision models using only standard class labels from a held-out audit dataset. The approach consists of three main stages:

1. Non-Negative Concept Decomposition:

   • The model $f$ is decomposed into $h \circ g$, where $g$ maps inputs to intermediate representations $a$.
   • For a target class $y$, the authors collect patches from inputs predicted as $y$ and apply Non-Negative Matrix Factorization (NMF) to the activations $A_y$.
   • This yields a class-conditional bank of interpretable concept vectors ($W_y$) and corresponding coefficients ($U_y$), decomposing the representation into non-negative, semantically meaningful directions.

2. Gradient-Based Bias Identification:

   • The core hypothesis is that spurious attributes manifest as distinct concept directions that react asymmetrically to gradient updates on misclassified examples.
   • For misclassified examples (False Negatives and False Positives), the authors compute the gradient of the loss with respect to the representation ($\nabla_a L$) and apply a single gradient step to create a perturbed representation $a'$.
   • They observe how concept coefficients change between $a$ and $a'$.
   • Intuition: To correct a False Negative (missing the target class), the model tends to add the spurious cue; to correct a False Positive (incorrectly predicting the target), it tends to remove the spurious cue. Intrinsic concepts (true evidence) do not show this strong asymmetric pattern.
   • A Bias Score ($S_{y,k}$) is calculated for each concept $k$ based on the consistency of these additions and removals across the audit set. Concepts with high scores are ranked as candidate bias directions.

3. Inference-Time Mitigation:

   • Once bias concepts are identified, they are suppressed at inference time without updating model parameters.
   • The method merges class-conditional concept banks, clusters similar concepts, and projects the input representation onto the subspace orthogonal to the identified bias concepts.
   • The suppressed representation is rescaled to maintain the original activation norm before classification.

Key Contributions

• Label-Free Discovery: A method to identify spurious concepts in frozen models without requiring bias or group annotations, relying solely on standard class labels.
• Gradient Probes: A novel scoring mechanism that uses the interaction between concept activations and gradient steps to distinguish bias concepts from intrinsic evidence.
• Actionable Mitigation: A practical inference-time suppression technique that improves worst-group accuracy without retraining or parameter updates.
• Interpretability: The method outputs concept vectors with associated activating patches, providing an interpretable audit of what the model is relying on.

Experimental Results
The method was evaluated on three standard benchmarks: Colored MNIST (CMNIST), Waterbirds, and CelebA.

• Concept Alignment: On Waterbirds and CMNIST, the NMF decomposition reliably recovered concepts aligned with known spurious attributes (e.g., background type for birds, color for digits).
• Bias Scoring: The proposed gradient-based score successfully ranked bias-aligned concepts higher than intrinsic ones. A threshold of 0.55 was found to effectively separate bias concepts.
• Mitigation Performance:
  • Waterbirds: Suppressing top-ranked concepts improved worst-group accuracy by 17.9 percentage points (from 45.9% to 63.8%) and worst-class accuracy by 8.5 points.
  • CelebA: Despite the identified concepts only partially coinciding with the annotated gender attribute (capturing hair shape/length rather than just color), suppression improved worst-group accuracy by 10.4 percentage points (from 43.4% to 53.8%).
  • CMNIST: Results were mixed; while bias concepts were identified, suppression did not consistently improve worst-group accuracy, likely due to the tight entanglement of color and shape in the specific decomposition rank used.
• Comparison: The method outperformed a random suppression ablation, confirming that the identified directions are specifically bias-aligned. It operates without the group labels or retraining required by supervised baselines like Group DRO or DFR, though it does not reach the absolute performance levels of those retrained models.

Significance and Claims
The paper claims to provide both an interpretable auditing tool and an actionable debiasing handle for deployed vision models. Its primary significance lies in addressing the "discovery problem" where the bias is unknown and retraining is infeasible.

• The authors emphasize that the method identifies decision-relevant spurious directions that need not perfectly coincide with annotated attributes (as seen in CelebA), capturing subtle proxies like hair length or background texture that drive model errors.
• The approach is presented as a zero-cost, post-hoc intervention that can be applied before deciding whether more invasive retraining is warranted.
• The paper acknowledges limitations, noting that single-scale patch decomposition may struggle with spatially distributed biases (like gender on CelebA) and that the method requires a sufficient number of misclassified examples in the audit set to function effectively.