Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) have become foundational for multimodal tasks but inherit and amplify demographic biases (e.g., gender, race, age) from their training data. These biases manifest as skewed retrieval rankings, stereotypical captions, and unfair predictions.

While post-hoc debiasing methods (operating on frozen models) are preferred over retraining due to efficiency, the current state-of-the-art method, SFID (Selective Feature Imputation for Debiasing), relies on a flawed geometric assumption: that bias is localized to a small, discrete set of embedding coordinates. The authors argue that this coordinate-wise approach suffers from three critical limitations:

1. Feature Entanglement: Different attributes (e.g., gender and race) share overlapping embedding dimensions. Removing coordinates for one attribute inadvertently distorts others.
2. Poor Cross-Dataset Generalization: The specific indices of "important" dimensions shift across different datasets, making coordinate-based imputation values learned on one dataset (e.g., FairFace) ineffective on another (e.g., FACET).
3. Incomplete Bias Removal: Bias is not concentrated in the top-$m$ dimensions but is distributed across a broader linear subspace. Replacing only the top coordinates leaves significant linearly decodable bias signals intact.

2. Methodology: Subspace Projection Debiasing (SPD)

The authors propose Subspace Projection Debiasing (SPD), a geometrically principled framework that treats bias as a continuous linear subspace rather than discrete coordinates. The method consists of three stages:

A. Bias Subspace Identification (via INLP)

Instead of selecting specific coordinates, SPD uses Iterative Null-space Projection (INLP) to identify the linear subspace spanned by directions predictive of sensitive attributes.

1. A linear classifier is trained to predict the sensitive attribute from the embeddings.
2. The weight matrix of this classifier is decomposed (via QR decomposition) to extract an orthonormal basis ($U^{(t)}$) representing the discriminative directions.
3. The embeddings are projected onto the null space of these directions to remove the attribute signal.
4. This process repeats iteratively ($T$ times) to capture multiple correlated directions, forming a final bias subspace $U \in \mathbb{R}^{d_b \times D}$.

B. Subspace Projection

Once the bias subspace $U$ is identified, the model projects the input embedding $x$ onto the orthogonal complement of $U$:
$$x' = x(I - U^\top U)$$
This operation removes the component of the embedding lying within the bias subspace, effectively suppressing linearly decodable attribute information.

C. Neutral Reinjection

Pure projection can cause "off-manifold drift," degrading semantic fidelity and task performance. To counteract this, SPD re-injects a neutral mean component ($\bar{x}_{low}$) along the removed subspace.

1. A Random Forest classifier estimates prediction confidence for each sample.
2. The mean embedding of low-confidence samples (those where the classifier is uncertain) is computed as $\bar{x}_{low}$.
3. This neutral mean is projected back onto the bias subspace and added to the debiased embedding:
   $$x'' = x' + U^\top (U \bar{x}_{low})$$
   This step re-centers the embeddings without reintroducing attribute-specific variance, preserving semantic structure while maintaining fairness.

3. Key Contributions

1. Geometric Insight: The paper establishes that demographic bias in VLMs is subspace-structured and entangled, not coordinate-sparse. This challenges the fundamental assumption of existing coordinate-level debiasing methods.
2. SPD Framework: A novel, training-free, post-hoc method that removes bias via continuous subspace projection rather than discrete coordinate replacement.
3. Neutral Reinjection Mechanism: A technique to preserve semantic fidelity by re-centering embeddings using low-confidence sample statistics, mitigating the utility loss often associated with aggressive debiasing.
4. Systematic Analysis: A rigorous reproduction study of SFID that empirically demonstrates its failure in handling feature entanglement, cross-dataset transfer, and complete bias removal.

4. Experimental Results

The authors evaluated SPD across three downstream tasks using multiple VLM backbones (CLIP ResNet-50, CLIP ViT-B/32, XVLM) and generative models (SDXL, CoDi).

• Zero-Shot Classification (FACET Benchmark):
  • SPD achieved the lowest demographic parity gaps ($\Delta DP$) across all backbones.
  • It improved fairness metrics by an average of 13.8% to 16.0% compared to baselines while maintaining or slightly improving accuracy.
• Text-to-Image Retrieval (Flickr30K):
  • SPD significantly reduced retrieval skew ($Skew@100$) compared to SFID and other baselines.
  • For the CLIP ViT-B/32 model, SPD achieved a 59.4% improvement in fairness while simultaneously improving Recall@1.
• Text-to-Image Generation:
  • SPD reduced gender mismatch rates for gender-specific prompts and produced more balanced distributions for neutral prompts compared to adversarial or prompt-based methods.
  • It maintained high perceptual quality, avoiding the instability seen in adversarial residual methods.
• Ablation Studies:
  • Removing the "neutral reinjection" step resulted in lower accuracy, confirming its role in preserving semantic fidelity.
  • The projection depth ($r$) acts as a tunable knob for the fairness-utility trade-off; moderate values (e.g., $r=5$) offered the best balance.

5. Significance

This work fundamentally shifts the paradigm of post-hoc debiasing from coordinate editing to subspace geometry. By recognizing that bias is distributed and entangled, SPD offers a more robust, generalizable, and interpretable solution. It demonstrates that effective debiasing does not require expensive retraining or complex prompt engineering; instead, it can be achieved through principled linear algebra operations on frozen embeddings. This approach provides a unified framework for mitigating multi-attribute bias while preserving the generalization integrity of multimodal systems.