Technical Summary: BioPro – Towards Difference-Aware Gender Fairness for Vision-Language Models

1. Problem Definition

Vision-Language Models (VLMs) inherit significant social biases from their training data, particularly regarding gender representation. Existing fairness interventions often adopt a "difference-unaware" perspective, enforcing uniform treatment across all demographic groups. The authors argue that this approach fails to distinguish between contexts where neutrality is required (e.g., describing an ambiguous subject) and contexts where group-specific attributes are legitimate and must be preserved (e.g., describing a clearly visible female doctor).

The paper formalizes the problem of difference-aware gender fairness for VLMs, specifically targeting two tasks:

1. Image Captioning: The model should avoid inferring gender for ambiguous subjects (neutral fairness) while faithfully preserving gender information for clearly identifiable subjects (explicit faithfulness).
2. Text-to-Image Generation: The model should generate a balanced gender distribution for neutral prompts (e.g., "a photo of a person") while strictly adhering to specified gender identities in explicit prompts (e.g., "a photo of a female doctor").

The core challenge is to achieve selective debiasing: mitigating unwanted bias in neutral contexts without erasing valid distinctions in explicit ones, all while maintaining semantic fidelity.

2. Methodology: BioPro (Bias Orthogonal Projection)

The authors propose BioPro, an entirely training-free framework that operates via orthogonal projection in the representation space. The method identifies a low-dimensional gender-variation subspace and selectively neutralizes gender-related information based on the input's context.

2.1 Construction of the Gender-Variation Subspace

BioPro utilizes counterfactual embeddings to define the bias direction.

• Data Source: The method employs the SCFs dataset [19], which generates image pairs that are identical in all features except for the gender attribute (e.g., a male vs. a female subject in the same pose/scene).
• Subspace Extraction: For a pair of embeddings $(h_m, h_f)$, the difference vector encodes high-purity gender variation. By collecting these differences into a matrix $D$, the authors perform Singular Value Decomposition (SVD) to extract the principal directions of gender variation. The top $k$ singular vectors form the gender-variation subspace $S$.

2.2 Task-Specific Implementation

A. Image Captioning (Selective Debiasing)

For captioning, the goal is to remove gender information only when the input is ambiguous.

1. Orthogonal Projection: The model projects the fused text-image embedding onto the orthogonal complement of the gender subspace ($P^\perp$). This removes the gender component ($H_{bias}$) while preserving the semantic component ($H_{sem}$).
2. Projection-Based Selection: To prevent over-correction of explicit samples, BioPro introduces a selection mechanism. It analyzes the magnitude of the projection of embeddings onto the gender subspace.
   • Explicit samples (clearly gendered) exhibit larger projection magnitudes.
   • Neutral samples (ambiguous) exhibit smaller magnitudes.
   • A threshold $\delta_c$ is learned (via optimization of a trade-off coefficient $\lambda_c$) to distinguish between the two distributions (modeled as skew-normal distributions).
   • Action: If a sample's projection magnitude is below $\delta_c$, orthogonal projection is applied. If above, the original embedding is retained to preserve explicit gender faithfulness.

B. Text-to-Image Generation (Calibration)

For generation, the output space is inherently binary (images must exhibit gender attributes), making simple projection insufficient as it may still result in imbalanced distributions.

1. Calibration Term: BioPro introduces an optimization objective that combines the orthogonal projection term with a calibration term.
   • The objective minimizes the distance between the projection matrix $P$ and the orthogonal projector $P^\perp$ (to preserve semantics).
   • Simultaneously, it minimizes the distance between projected female embeddings and male embeddings (and vice versa) to balance the generative direction.
2. Closed-Form Solution: The authors derive a closed-form solution for the optimal projection matrix $P_{f \to m}$, allowing the model to shift representations toward underrepresented genders in neutral prompts without erasing intended semantics in explicit prompts.

2.3 Generalization to Continuous Bias

The framework is extended to handle continuous bias variables, such as scene brightness. By constructing prompt pairs with varying intensities (e.g., "light photo" vs. "dark photo"), BioPro identifies a continuous bias subspace. A calibration term allows for the adjustable control of the generated scene's brightness level via a hyperparameter $\lambda_g$, demonstrating applicability beyond discrete categorical attributes.

3. Key Contributions

1. Conceptual Extension: The paper extends the concept of difference-aware fairness from text-only Large Language Models (LLMs) to multimodal Vision-Language Models, providing a principled formulation for selective bias mitigation.
2. BioPro Framework: It introduces BioPro, a training-free method that effectively balances neutral fairness and explicit gender faithfulness through orthogonal projection and adaptive selection/calibration mechanisms.
3. Continuous Bias Handling: To the best of the authors' knowledge, this is the first method to effectively handle continuous bias variables (like scene brightness) in text-to-image generation tasks, moving beyond binary or categorical bias mitigation.

4. Experimental Results

The authors evaluated BioPro on image captioning (using LLaVA-1.5 and LLaVA-NeXT) and text-to-image generation (using FLUX.1-dev and FLUX.1-schnell).

• Image Captioning:

  • BioPro achieved the best Composite Bias Rate (CBR), indicating a superior balance between reducing bias in neutral cases ($BR_n$) and maintaining faithfulness in explicit cases ($BR_e$).
  • It significantly outperformed baselines like Prompting, LIBRA, and SFID.
  • Semantic preservation metrics (METEOR and CLIP-S) remained consistent with the base models, confirming that debiasing did not degrade caption quality.
  • Ablation studies showed that removing the "Selection" module led to a significant drop in explicit faithfulness, validating the necessity of the selective mechanism.

• Text-to-Image Generation:

  • BioPro achieved the lowest Skew scores (indicating better gender balance) for both male-stereotyped and female-stereotyped professions.
  • It maintained a near-zero Misclassification Rate (MR) for explicit prompts, ensuring that specified genders were respected.
  • Image quality metrics (CLIP-S and MUSIQ) showed negligible degradation compared to the base models.
  • The "w/o Calibration" variant showed significantly weaker debiasing, highlighting the importance of the calibration term for generation tasks.

• Continuous Bias (Scene Brightness):

  • BioPro successfully adjusted the luminance of generated images (e.g., making sky scenes darker) based on the parameter $\lambda_g$.
  • The method preserved semantic alignment (CLIP-S) and image quality (MUSIQ) while effectively mitigating the bias toward overexposed scenes.

5. Significance and Claims

The paper claims that BioPro provides a controllable and effective paradigm for bias-aware generation in VLMs. Its significance lies in:

• Context Sensitivity: Unlike "blind" fairness approaches that treat all groups identically, BioPro respects contextual nuances, preserving legitimate distinctions while removing spurious stereotypes.
• Training-Free Efficiency: As a post-hoc inference-time intervention, it avoids the computational cost and data requirements of retraining or fine-tuning large models.
• Generalizability: The framework demonstrates that bias mitigation can be generalized from discrete attributes (gender) to continuous variables (brightness), offering a unified solution for diverse bias types.

The authors conclude that while their linear intervention approach may not fully account for nonlinear entanglement between attributes, it offers a practical, flexible, and high-quality solution for achieving selective fairness in multimodal systems.