🤖 AI

A Woman with a Knife or A Knife with a Woman? Measuring Directional Bias Amplification in Image Captions

This paper introduces Directional Bias Amplification in Captioning (DBAC), a novel language-aware metric that overcomes the limitations of existing methods by accurately measuring and identifying the sources of bias amplification in image captioning models, particularly regarding gender and race attributes.

Original authors: Rahul Nair, Bhanu Tokas, Hannah Kerner

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rahul Nair, Bhanu Tokas, Hannah Kerner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot assistant whose job is to look at photos and describe what's happening. You teach this robot by showing it millions of photos with human-written descriptions. The problem is, humans aren't perfect; sometimes our descriptions contain hidden stereotypes (biases). For example, we might often describe a woman holding a knife as "a woman with a knife" (implying danger) but a man holding a knife as "a man cooking" (implying safety).

When the robot learns from these human descriptions, it doesn't just copy the bias; it often amplifies it. It might start predicting "knife" whenever it sees a woman, even if she's just holding a flower, because it learned that "woman" and "knife" are a dangerous pair in its training data.

This paper is about building a better bias detector to figure out exactly how and why the robot is getting these stereotypes wrong.

The Problem with Old Detectors

The authors explain that previous tools for measuring bias were like bad translators or broken scales:

The "Blind" Scales: Some old tools only counted how often words appeared together (like "man" + "ball"). They didn't understand the meaning of the sentence. They couldn't tell the difference between "He has a ball" and "He does not have a ball," even though the bias is totally different.
The "One-Way" Mirrors: Other tools could tell you that a bias existed, but they couldn't tell you which way it was pointing.
- The Analogy: Imagine a room where people are shouting. A "one-way" mirror tells you, "There is a lot of shouting." But it doesn't tell you if the people are shouting at the door or from the door.
- In the paper's terms: Did the robot see a knife and decide, "Ah, this must be a woman"? (Task $\to$ Attribute) OR did it see a woman and decide, "Ah, she must be holding a knife"? (Attribute $\to$ Task).
- Without knowing the direction, you can't fix the problem. If you try to fix the wrong direction, you might make the bias worse.

The New Solution: DBAC

The authors introduce a new tool called DBAC (Directional Bias Amplification in Captioning). Think of DBAC as a super-smart detective that can:

Read the story: It understands the full meaning of the sentence (Language-Aware).
Trace the arrows: It can pinpoint exactly which way the bias is flowing (Directional).
Ignore the noise: It gives consistent answers no matter which "dictionary" (sentence encoder) it uses to read the text.

How DBAC Works (The Detective's Toolkit)

To catch the bias, DBAC uses a clever trick involving masking and substitution:

The Masking Game: Imagine the robot writes a caption: "A woman is holding a knife."
- DBAC hides the word "woman" and asks: "Can you still guess the gender just by looking at the rest of the sentence?" If the answer is "Yes, it's definitely a woman," that means the other words (like "knife") are carrying a heavy bias.
- It does the reverse too: Hide "knife" and ask, "Can you guess the object just by knowing it's a woman?"
The "Contextual" Swap: This is a major upgrade.
- Old Method (Constant Substitution): If the robot used a rare word like "saddle," and the dictionary didn't have it, the old tool would just replace it with "UNKNOWN." This destroys the meaning.
- DBAC Method (Contextual Substitution): DBAC is smarter. If it sees "saddle," it looks for the closest match in the dictionary, like "seat" or "horse gear," and swaps that in. This keeps the story intact so the bias measurement is accurate.

What They Found

The authors tested this new detective on 13 different image-captioning robots using a massive photo dataset called COCO. Here are the key takeaways:

The Only Reliable Detective: DBAC was the only tool that could consistently and accurately measure bias amplification. The other tools either missed the bias entirely or gave confusing, inconsistent results depending on which "dictionary" they used.
The Direction Matters: They discovered that while many robots got better at predicting tasks when they knew the gender (e.g., "If it's a man, he's likely playing sports"), they got worse at predicting gender when they saw a task (e.g., "If he's holding a knife, he's definitely a man").
The "Fix" Can Break Things: They found that a technique called "Equalizer," which was supposed to fix bias, actually made one type of bias worse while fixing another. This proves you need a directional tool like DBAC to see the full picture before you try to fix the robot.

The Bottom Line

If you want to build an AI that describes images fairly, you can't just guess where the bias is. You need a tool that understands the story and can point exactly at the source of the prejudice. DBAC is that tool. It tells you not just that the robot is biased, but how it is biased, so you can fix the specific problem without breaking the rest of the system.

1. Problem Statement

Bias Amplification in Image Captioning:
When image captioning models are trained on biased datasets (e.g., human-written captions containing societal stereotypes), they often reproduce and even amplify these biases in their generated outputs. For example, a model might predict "a woman holding a knife" more frequently than "a man holding a knife" when the visual evidence is ambiguous, or vice versa.

Limitations of Existing Metrics:
Current methods for measuring bias amplification fall into two categories, both of which fail for image captioning:

Classification-based metrics (e.g., BA→, DPA): These measure bias in classification tasks but ignore the semantic meaning of natural language sentences. They cannot distinguish between "He has a ball" and "He does not have a ball" if the keywords are the same.
Language-aware metrics (e.g., LIC, ImgCap2):
- LIC (Leakage in Captioning): While it understands semantics, it is non-directional. It cannot distinguish which attribute caused the bias. It cannot tell if the model amplified the bias of predicting a "woman" when seeing a "knife," or predicting a "knife" when seeing a "woman."
- ImgCap2: Confounds biases inherent in the image with biases in the caption, making it unsuitable for isolating captioning model behavior.
- Consistency Issues: LIC scores vary significantly depending on the choice of sentence encoder (a hyperparameter), making results unreliable.

The Gap: There is no existing metric that is simultaneously language-aware (understands semantics), directional (identifies the source of bias: Attribute $\to$ Task vs. Task $\to$ Attribute), and consistent across different model configurations.

2. Methodology: Directional Bias Amplification in Captioning (DBAC)

The authors propose DBAC, a new metric designed to measure bias amplification in two specific directions:

$A \to T$ (Attribute to Task): Does the presence of an attribute (e.g., Gender) amplify the prediction of a task/object (e.g., Knife)?
$T \to A$ (Task to Attribute): Does the presence of a task/object amplify the prediction of an attribute (e.g., Gender)?

Core Mechanism

DBAC uses an adversarial attack framework combined with vocabulary alignment:

Masking:
- To measure $A \to T$ : Gender tokens are masked in the caption (e.g., "A with a guitar").
- To measure $T \to A$ $T \to A$ : Task tokens are masked (e.g., "A man with a
  ").
  Attacker Function ( $f$ ):
  - A model (sentence encoder + classification head) is trained to predict the masked attribute (e.g., gender) using only the remaining context.
  - High prediction accuracy implies that the remaining context (the task) contains strong bias signals about the attribute.
  Probabilistic Formulation:
  - The bias is quantified by the probability of the attribute given the masked sentence.
  - To ensure a fixed prior (a requirement for valid bias amplification measurement), the authors use Bayes' theorem to approximate the bias as $P(\text{Task} | \text{Attribute})$ .
  - Bias Amplification Score: Calculated as the difference between the bias in Model-Generated Captions (MGC) and Human-Generated Captions (HGC), normalized to a range of $(-1, 1)$ .
  - Formula: $DBAC = \frac{\omega_M - \omega_H}{\omega_M + \omega_H + \epsilon}$ , where $\omega$ represents the bias strength.
  Key Technical Improvements over LIC
  1. Directionality: Explicitly separates $A \to T$ and $T \to A$ amplification, allowing for targeted mitigation strategies.
  2. Contextual Substitution (Vocabulary Alignment):
    - Problem: Human captions (HGC) have larger vocabularies than model captions (MGC). Simple "constant substitution" (replacing unknown words with <unk>) loses semantic nuance.
    - Solution: DBAC uses contextual substitution. If a word in HGC is missing in MGC, it is replaced by the semantically closest word from the MGC vocabulary (using embeddings like GloVe or FastText). If no close match exists, it becomes <unk>.
    - Benefit: This preserves the semantic bias signal better than constant substitution.
  3. Encoder Consistency: The metric is designed to be robust across different sentence encoders (LSTM, RNN, Transformers, Pre-trained models).
  3. Key Contributions
  1. DBAC Metric: The first metric that is simultaneously language-aware, directional, and consistent.
  2. Source Identification: DBAC can pinpoint whether a model amplifies bias from attributes to tasks or vice versa, enabling specific retraining strategies (e.g., retraining only on "women with non-knife objects" if $A \to T$ bias is high).
  3. Contextual Substitution: Demonstrated that replacing words with semantically similar alternatives (rather than generic <unk>) yields more accurate bias estimates.
  4. Robustness: Proved that DBAC scores are stable across various sentence encoders, eliminating the need for hyperparameter tuning to find a "suitable" encoder.
  4. Experimental Results
  
  Experiments were conducted on the COCO Captions dataset using 13 different captioning models (including BLIP, OSCAR, LLaVA, and Transformer-based models) and attributes for Gender and Race.
  - Capturing Language Bias: In controlled datasets where gender and task were balanced but verb usage was imbalanced, non-language-aware metrics (BA→, DPA) reported zero bias. DBAC and LIC correctly detected the linguistic bias.
  - Directionality Validation:
    - The authors progressively masked people in images. As the person became less visible, the model relied more on background objects (tasks) to guess gender/race.
    - Result: Only DBAC captured the increasing trend in $T \to A$ bias as masking increased. LIC failed to detect this trend because it lacked directionality.
  - Consistency (Sensitivity Analysis):
    - DBAC showed significantly lower Coefficient of Variation (CV) across different sentence encoders compared to LIC.
    - Gender: DBAC had 84.5% less variation (scratch encoders) and 69.9% less variation (pre-trained encoders) than LIC.
    - Race: DBAC had 59.5% less variation (scratch) and 83.6% less variation (pre-trained) than LIC.
  - Substitution Impact: Contextual substitution improved METEOR similarity scores (retaining original meaning) by ~2.7% to 3.4% over constant substitution. It also resulted in higher, more accurate bias detection scores.
  - Model Analysis:
    - Most models reduced $A \to T$ bias (predicting tasks given gender became fairer).
    - However, 8 out of 13 models increased $T \to A$ bias (predicting gender given a task became more biased).
    - The "Equalizer" mitigation technique reduced $T \to A$ bias but inadvertently worsened $A \to T$ bias, highlighting the need for directional metrics to avoid unintended consequences.
  5. Significance
  - Reliability: DBAC is established as the only reliable metric for measuring bias amplification in image captioning, addressing the critical flaws of previous methods.
  - Actionable Insights: By identifying the direction of bias, DBAC allows researchers to apply targeted mitigation. Instead of retraining a model on a massive dataset, researchers can specifically curate data to break specific biased associations (e.g., "knife $\to$ woman").
  - Standardization: The robustness of DBAC against sentence encoder choices removes a major source of experimental noise, making bias measurement more reproducible and comparable across studies.
  - Safety: In high-stakes applications like assistive technology for the visually impaired, understanding and correcting directional bias is crucial to prevent misleading or harmful descriptions.
  ShareTwitter LinkedIn Email
  Drowning in papers in your field?
  Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.
  Try Digest →
  Get the best AI papers every week.
  Trusted by researchers at Stanford, Cambridge, and the French Academy of Sciences.
  Check your inbox to confirm your subscription.
  Something went wrong. Try again?
  No spam, unsubscribe anytime.
  More like this
  View all in this category 🤖 cs.AI →
  Weekly digest — the most novel research, explained simply.Subscribe
  Source paper
  arXiv
  2503.07878
  View on arXiv ↗PDF ↗
  Read in
  AR DE EN ES FR HI IT JA KO NL PT ZH
  Drowning in papers in your field?
  Try Digest
  ggist.science
  About Testimonials Privacy Terms Contact
  Explanations are AI-generated and may contain inaccuracies. Authors can suggest corrections via mail@gist.science.
  Thank you to arXiv, bioRxiv, medRxiv, and Research Square for use of their open access interoperability.
  Gist.Science is a product of Bition B.V.
  Verdunplein 17, 5627SZ Eindhoven
  KvK: 95743731 | BTW-ID: NL867271966B01
  mail@gist.science
  Made in the Netherlands 🇳🇱
  © 2026 gist.science

The Problem with Old Detectors

The New Solution: DBAC

How DBAC Works (The Detective's Toolkit)

What They Found

The Bottom Line

1. Problem Statement

2. Methodology: Directional Bias Amplification in Captioning (DBAC)

Core Mechanism

Key Technical Improvements over LIC

3. Key Contributions

4. Experimental Results

5. Significance

More like this