💻 computer science

A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks

This paper proposes a training-free, closed-form debiasing method for Vision-Language Models that achieves Pareto-optimal fairness with bounded utility losses across visual and textual modalities without requiring annotated data.

Original authors: Tangzheng Lian, Guanyu Hu, Yijing Ren, Dimitrios Kollias, Oya Celiktutan

Published 2026-03-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tangzheng Lian, Guanyu Hu, Yijing Ren, Dimitrios Kollias, Oya Celiktutan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian named VLM (Vision-Language Model). This librarian has read millions of books and looked at billions of photos from the internet. Because of this, they are incredibly good at connecting pictures with words. If you ask, "Show me a doctor," they can instantly find or draw a picture of a doctor.

The Problem: The Librarian's Bad Habits
However, because the librarian learned from the internet, they picked up some bad habits (biases).

If you ask for a "doctor," the librarian almost always shows you a man.
If you ask for a "nurse," they almost always show you a woman.
If you ask for a "CEO," they show a man in a suit.

This happens because the internet data they learned from had these stereotypes built-in. The librarian isn't "evil"; they just learned that "man" and "doctor" often appear together in the same sentences and photos.

The Old Solutions: The Heavy Hammers
Previous attempts to fix this librarian were like using a sledgehammer:

Retraining: You try to teach the librarian new lessons from scratch. This takes years, costs a fortune in electricity, and is practically impossible.
Adding a Filter: You hire a new assistant to stand in front of the librarian and correct their answers. This makes the system slower, more complicated, and sometimes the assistant accidentally throws away the real information you wanted (like the difference between a "doctor" and a "lawyer").
The "Sensitive List": Many methods require you to give the librarian a list of every single person's gender or race in the photos. This is a privacy nightmare and very hard to do.

The New Solution: The "Closed-Form" Magic Wand
This paper introduces a new, elegant method that acts like a magic wand rather than a sledgehammer. Here is how it works in simple terms:

1. No New Training, No New Data

The method doesn't need to retrain the librarian or ask for private data. It works instantly on the librarian's existing brain (the "embeddings"). It's training-free and data-free.

2. The "Stereotype Axis" vs. The "Meaning Axis"

Imagine the librarian's brain is a giant 3D map.

One direction on the map represents Meaning (e.g., "This is a medical professional").
Another direction represents Stereotypes (e.g., "This is a man").

Old methods tried to cut off the whole "Stereotype" direction, but they accidentally chopped off part of the "Meaning" too. If you cut off "man," you might lose the concept of "doctor" entirely.

This new method is smarter. It uses a large language model (like a super-smart AI writer) to generate many different ways to say "male doctor" (e.g., "man doctor," "masculine doctor," "doctor who is a man"). It then draws a precise line (an axis) that represents only the stereotype.

3. The "Closed-Form" Shortcut

Here is the mathematical magic: Instead of guessing and checking millions of times to find the perfect fix, the authors found a closed-form solution.

Analogy: Imagine you are trying to find the shortest path across a river. Old methods were like walking back and forth, testing different bridges. This new method is like having a map that instantly tells you the exact coordinates of the perfect bridge.
It calculates the perfect angle to "tilt" the librarian's answer so that it ignores the gender bias but keeps the job description perfectly intact.

4. The "Pareto-Optimal" Balance

The paper talks about "Pareto-optimal fairness." Let's translate that:

Usually, if you make something fairer, it becomes less useful (e.g., if you force the librarian to show equal men and women doctors, they might start showing you pictures of random people who aren't doctors).
This method finds the perfect sweet spot. It removes the bias as much as possible without making the librarian forget what a doctor looks like. It guarantees that the "utility" (usefulness) doesn't drop.

5. Fixing the "Intersection" Problem

Most old methods only looked at one thing at a time (e.g., just gender). But in real life, people are a mix of things (e.g., an "older Black woman").

This method is smart enough to handle intersectional fairness. It can fix biases for specific combinations of traits, ensuring that a "young female doctor" isn't treated differently than an "older male doctor."

Summary of Benefits

Fast: No waiting for training.
Private: Doesn't need sensitive data labels.
Versatile: Works for finding images, classifying photos, and even generating new images (like drawing a doctor).
Safe: It has a mathematical guarantee that it won't break the librarian's ability to do its job.

In a nutshell: The authors built a precise, instant "bias-removal tool" that surgically removes the librarian's stereotypes while keeping all their knowledge intact, ensuring that when you ask for a doctor, you get a doctor—regardless of whether they are a man, a woman, or anything else.

1. Problem Statement

Vision-Language Models (VLMs), such as CLIP, have achieved state-of-the-art performance by training on massive web-scraped image-text pairs. However, these datasets inherently contain social biases (e.g., gender stereotypes linking "nurse" to "female" and "doctor" to "male") and spurious correlations. When deployed in downstream tasks (e.g., image classification, retrieval, generation), these biases propagate, leading to discriminatory outputs.

Existing debiasing methods face five critical limitations:

Multi-Modal Bias: Many methods only debias text or image embeddings, ignoring that bias is encoded in the cross-modal alignment.
Training Cost: Most approaches require retraining or adding auxiliary networks (e.g., adversarial heads), which increases computational cost and model complexity.
Data Dependency: Many methods rely on datasets annotated with sensitive attributes (gender, race), which are difficult to collect due to privacy and ethical constraints.
Task Specificity: Current solutions often target a single task (e.g., zero-shot classification) and fail to generalize across diverse tasks like text-to-image retrieval or generation.
Utility Loss: Debiasing often degrades model performance (utility). Existing methods lack theoretical guarantees that semantic utility or cross-modal alignment is preserved.
Intersectional Blindness: Most methods focus on group fairness (single attributes) rather than intersectional fairness (combinations of attributes, e.g., gender $\times$ race).

2. Methodology

The authors propose a training-free, data-free, closed-form solution that operates directly in the cross-modal embedding space. The method consists of two main stages:

A. LLM-Guided Group Prototype Construction

Instead of relying on pre-annotated datasets, the method uses a Large Language Model (LLM) to construct "group prototypes" dynamically.

Input: A neutral prompt (e.g., "a photo of a doctor").
LLM Processing: For a defined sensitive attribute (e.g., gender) and a specific group (e.g., "male"), the LLM:
- Inserts the group specification into the prompt (e.g., "a photo of a male doctor").
- Generates multiple linguistic variants of this prompt (e.g., "a photo of a man doctor," "a photo of a masculine doctor") to capture the semantic diversity of the group.
Prototype Aggregation: The text encoder generates embeddings for the anchor prompt and its variants. The Group Prototype ( $\vec{p}_g$ ) is computed as the spherical mean of these embeddings on the unit hypersphere.
Attribute Subspace ( $\mathcal{A}$ ): The method defines the "attribute direction" as the difference between group prototypes (e.g., $\vec{a} = \vec{p}_{male} - \vec{p}_{female}$ ). The attribute subspace $\mathcal{A}$ is the span of these directions.

B. Closed-Form Debiased Embedding Search

The goal is to find a debiased embedding $\vec{u}$ that is orthogonal to the attribute subspace $\mathcal{A}$ (removing bias) while remaining close to the original embedding $\vec{e}$ (preserving utility).

Decomposition: Using the orthogonal decomposition theorem, any embedding $\vec{e}$ is split into a component parallel to $\mathcal{A}$ (bias leakage, $\vec{e}_{A\parallel}$ ) and a component orthogonal to $\mathcal{A}$ (neutral content, $\vec{e}_{A\perp}$ ).
Optimization Formulation: The problem is formulated as minimizing a weighted sum of Attribute Leakage (fairness) and Self-Utility Loss (semantic preservation).
- Self-Utility: Cosine similarity between $\vec{u}$ and $\vec{e}$ .
- Cross-Utility: Preservation of alignment between image and text pairs.
Pareto-Optimal Solution: The authors prove that the optimal solution lies on the Pareto front. To avoid task-specific hyperparameter tuning, they derive a closed-form solution using Chebyshev scalarization (minimizing the worst-case loss).
- The optimal parameter $\alpha^*$ is derived analytically, balancing the trade-off between removing bias and keeping the original semantic content.
- The final debiased embedding is: $\vec{u}^* = \sqrt{1-(\alpha^*)^2}\frac{\vec{e}_{A\perp}}{\|\vec{e}_{A\perp}\|} + \alpha^*\frac{\vec{e}_{A\parallel}}{\|\vec{e}_{A\parallel}\|}$ .

Key Theoretical Guarantee: The method provides a theoretical upper bound on utility loss, proving that the proposed solution achieves Pareto-optimal fairness with bounded degradation in task performance.

3. Key Contributions

Closed-Form Solution: The first method to provide a theoretical, closed-form solution for debiasing VLMs in the cross-modal space, ensuring Pareto-optimal fairness with bounded utility loss.
Training-Free & Data-Free: The approach requires no retraining of the VLM, no additional network parameters, and no datasets annotated with sensitive attributes. It relies solely on the VLM's existing encoders and an LLM for prompt engineering.
Joint Modality Debiasing: Unlike prior works that focus on text or images separately, this method jointly debiases both visual and textual modalities, addressing the root cause of bias in cross-modal alignment.
Intersectional Fairness: The framework naturally extends to intersectional fairness (e.g., gender $\times$ race) by defining group prototypes for intersectional combinations.
Task Agnosticism: The method is applicable across diverse downstream tasks, including zero-shot image classification, text-to-image retrieval, and text-to-image generation.

4. Experimental Results

The method was evaluated on CLIP (ViT-L/14 and ResNet50) and BLIP across three tasks:

Zero-Shot Image Classification (CelebA, FACET):
- Achieved the best or second-best fairness metrics (Equal Opportunity violation) across all baselines.
- Crucially, it maintained or improved task utility (F1 score), whereas other methods (e.g., SFID, FairerCLIP) suffered significant performance drops.
- Successfully handled intersectional fairness (Gender $\times$ Age) without annotated data.
Text-to-Image Retrieval (Flickr30K, COCO2017):
- Significantly reduced bias (MaxSkew@M) in retrieved results.
- Preserved high Recall@K scores, outperforming training-based methods like PromptArray and FairerCLIP.
Text-to-Image Generation (Stable Diffusion v2.1):
- Generated balanced gender distributions for neutral prompts (e.g., "a photo of a doctor").
- Utility Preservation: Unlike Orth-Proj and Orth-Cali, which degraded the model's ability to generate gender-specific images when explicitly requested, this method maintained high CLIP scores and generation accuracy (AccG) for explicit prompts.
Ablation Studies: Confirmed that using LLM-generated variants (rather than single anchors) and debiasing both modalities are essential for optimal performance.

5. Significance

This paper represents a paradigm shift in VLM debiasing by moving away from complex, data-hungry, training-based approaches toward a geometric, analytical solution.

Practicality: By being training-free and data-free, it lowers the barrier to deploying fair AI, making it feasible for organizations without access to sensitive attribute datasets or massive compute resources.
Theoretical Rigor: The provision of closed-form solutions and utility bounds addresses a major gap in the literature, where debiasing was often treated as an empirical heuristic with unpredictable side effects on model performance.
Holistic Approach: By tackling intersectional fairness and joint modality bias, the method offers a more robust and realistic solution for real-world AI deployment where individuals belong to multiple sensitive groups simultaneously.

In summary, the authors demonstrate that it is possible to mathematically "subtract" bias from VLM representations without "subtracting" the model's intelligence or utility, offering a scalable and theoretically grounded path toward fair AI.