💬 NLP

Vision-Language Models Suppress Female Representations Under Ambiguous Input

This paper reveals that while vision-language models often internally encode female associations for ambiguous images, their outputs systematically default to male representations due to an asymmetric filtering mechanism that amplifies male signals and suppresses female ones during generation.

Original authors: Arnau Marin-Llobet, Simon Henniger, Mahzarin R. Banaji

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arnau Marin-Llobet, Simon Henniger, Mahzarin R. Banaji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Polite" Robot vs. The "Biased" Brain

Imagine you have a very polite robot assistant who has been trained to be fair and not make assumptions about people. If you show it a picture of a person clearly wearing a dress, it says, "That's a woman." If the person is clearly in a suit, it says, "That's a man." It passes the "politeness test" perfectly.

But this paper asks a tricky question: What happens when the robot can't see the person's face or gender clearly? (For example, a construction worker seen from behind in a helmet, or a nurse seen from the back).

The researchers found that while the robot's mouth (its final answer) stays polite and neutral, its brain (its internal thinking process) is actually making a very specific, biased guess: It assumes the person is a man.

Even for jobs that are statistically mostly done by women (like babysitters or florists), if the robot has to guess, it guesses "man." The scary part? The robot knows the job is usually female, but it overrides that knowledge and forces a "male" answer anyway.

The Tool: "LALS" (The X-Ray for Thoughts)

How do we know what the robot is thinking if it's not saying it? The authors invented a tool called LALS (Latent Association Leaning Score).

Think of LALS as an X-ray for the robot's brain.

Normally, we only see the robot's final sentence (the output).
LALS lets us peek inside the robot's "neurons" at every single step of its thinking process.
It translates the robot's internal electrical signals into words, allowing us to see if, at a specific moment, the robot is thinking "female," "male," or "neutral."

The Three Types of Jobs

When the researchers tested 15 different jobs with "faceless" images, they found three distinct patterns:

The "Agree" Men (e.g., Firefighters):
- Inside the brain: The robot thinks "Man."
- The answer: It says "Man."
- Verdict: No surprise here. The bias is consistent.
The "Agree" Women (e.g., Makeup Artists):
- Inside the brain: The robot thinks "Woman."
- The answer: It says "Woman."
- Verdict: Also consistent.
The "Divergence" Zone (e.g., Florists, Nurses, Babysitters):
- Inside the brain: The robot actually thinks "Woman." (It correctly associates the job with women).
- The answer: It says "Man."
- Verdict: This is the hidden bias. The robot's internal thought says "female," but something in its final step forces it to flip the switch and say "male."

The "Asymmetric Filter": A One-Way Street

The researchers discovered why this flip happens. They traced the robot's thoughts layer by layer (like checking the floors of a skyscraper).

The Male Signal: Imagine a strong, loud voice shouting "MAN!" This voice starts at the bottom floor and gets louder as it goes up. By the time it reaches the top floor (where the answer is spoken), it is very strong.
The Female Signal: Imagine a different voice shouting "WOMAN!" This voice starts strong, gets even louder in the middle of the building, but then hits a soundproof wall on the top floors. By the time it reaches the top, the voice has been silenced or turned into a whisper.

The robot has an asymmetric filter. It lets the "Male" idea pass all the way through, but it actively suppresses the "Female" idea right before it speaks.

The "Pink vs. Blue" Clue

To prove the robot was actually looking at the picture and not just guessing randomly, the researchers played a trick with colors.

They showed a picture of a nurse in blue scrubs. The robot's internal "Male" signal was strong.
They showed the exact same picture but changed the scrubs to pink.
Result: The robot's internal "Male" signal dropped significantly, and the "Female" signal grew stronger.

This proves the robot isn't just blindly guessing "Man" for everything. It is reacting to cultural cues. It has learned that in our world, pink often means "female" and blue often means "male." When the visual cue (pink) is strong, the robot's internal bias shifts. But without that strong cue, it defaults to "Man."

The Root Cause: It's Not Just "Training"

The researchers checked if this bias was taught to the robot to be "safe" (a process called alignment). They compared the robot before it was taught to be polite and after.

The Finding: The bias was there before the robot was taught to be polite.
The Conclusion: The "polite training" didn't fix the bias; it just taught the robot to hide it. The robot learned to say "I don't know" or "Man" to be safe, but its internal brain still carries the old, biased associations.

Summary

The paper reveals that Vision-Language Models have a hidden double life.

Publicly: They are neutral and careful.
Privately: When they can't see clearly, they default to assuming people are men, even when their own internal logic suggests the person is a woman.

The "Male Default" isn't a glitch; it's a deep-seated habit in the robot's brain that survives even when the robot is told to be fair. The robot knows the truth (that the person might be a woman), but it filters that truth out before it speaks.

Technical Summary: Vision-Language Models Suppress Female Representations Under Ambiguous Input

Problem Statement
While modern Vision-Language Models (VLMs) have been aligned to avoid expressing demographic biases in their outputs when gender is visually explicit, significant gaps remain regarding their behavior under ambiguous inputs (e.g., figures seen from behind, obscured faces, or full protective gear). Standard output-level auditing suggests these models are neutral, often refusing to assign gender or defaulting to "a person." However, this paper posits that output neutrality may mask persistent internal biases. Specifically, the authors investigate whether VLMs encode gendered associations internally that differ from their final textual outputs, particularly when visual cues are insufficient to determine gender.

Methodology
To probe internal representations without relying on output generation, the authors introduce LALS (Latent Association Leaning Score), a zero-shot metric designed to measure concept associations at the granularity of individual visual tokens and network layers.

LALS Mechanism: The method projects hidden states of visual tokens from any layer $\ell$ into the model's text-embedding space using the LatentLens procedure. By comparing these projected vectors against a balanced reference corpus of gender-associated terms (e.g., "man," "father" vs. "woman," "mother"), LALS computes a continuous score in $[-1, +1]$ for each token.
Aggregation: Image-level scores are derived by aggregating the top 5% of tokens with the strongest signal magnitude.
Experimental Setup: The study evaluates four open-weight VLMs (Qwen2-VL, Qwen2.5-VL, LLaVA-v1.6, InternVL2.5) on a dataset of 800+ gender-ambiguous images across 15 occupations (male-stereotyped, female-stereotyped, and neutral).
Comparative Analysis: The authors compare LALS internal scores against model outputs generated under two prompting conditions: open-ended description and forced-choice (FC) gender guessing. They also conduct ablation studies involving color manipulation (e.g., changing clothing from blue to pink) and comparisons between base and instruction-tuned models.

Key Contributions and Results

Decoupling of Internal Representations and Outputs:
The study identifies a systematic divergence where models internally encode female associations for certain occupations but output male defaults under forced-choice prompting. This "divergence regime" includes occupations like florists, preschool teachers, and babysitters. While open-ended prompts yield neutral responses, forced-choice prompts trigger a "male-mode collapse," where models default to male even for strongly female-stereotyped roles (e.g., nurses, hairdressers), despite internal signals suggesting otherwise.
Asymmetric Filtering Mechanism:
Layer-wise analysis of LALS scores reveals an asymmetric filter within the network architecture:
- Male Signals: Amplify or remain stable from early to late layers, passing through the network end-to-end.
- Female Signals: Peak in the mid-to-late layers but are systematically suppressed or collapsed toward zero (or negative/male territory) before the final generation layer.
  This suppression explains why female-associated internal representations fail to manifest in the final output for ambiguous inputs.
Influence of Culturally Loaded Visual Cues:
Color ablation experiments demonstrate that internal associations are modulated by specific visual cues. Changing a construction worker's hat or a nurse's scrubs from blue to pink significantly reduces the internal male signal or increases the female signal, respectively. This suggests models have internalized cultural gender-color associations present in training data.
Origin of Bias:
The asymmetric filtering is not an artifact of instruction tuning (RLHF). Comparisons between base and instruction-tuned models show that the suppression of female signals exists in the base model and is merely amplified by alignment, not created by it. Furthermore, the bias originates in the visual pathway; text-only prompts for the same occupations show female signal amplification in late layers, the opposite of the visual pathway's collapse.

Significance and Claims
The paper argues that output-level auditing is insufficient for assessing VLM fairness, particularly in high-stakes scenarios involving ambiguous inputs (e.g., surveillance, assistive tools). The authors claim that:

Alignment masks rather than eliminates bias: Models learn to suppress the expression of biased internal representations without removing the representations themselves.
Internal representations drive downstream risk: Since VLM embeddings are increasingly used as features for image search, content ranking, and hiring tools, biased internal signals propagate through these systems regardless of the model's final textual output.
LALS as an auditing tool: LALS provides a zero-shot, label-free method to detect these hidden biases, offering a more granular view of how demographic associations evolve through the network.

The authors conclude that the "male default" observed in ambiguous scenarios is a structural property of the model's visual processing, rooted in pretraining data and reinforced by the network's asymmetric filtering of gender signals, rather than a simple failure of alignment.