StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs

Technical Summary: StylisticBias

Problem Statement
Multimodal Large Language Models (MLLMs) are increasingly deployed in high-stakes domains such as hiring, content moderation, and judicial support. While prior research has established that these models inherit and amplify societal biases from training data, the specific visual cues driving these judgments remain poorly understood. Existing studies often compare different individuals or demographic groups, making it difficult to disentangle the effects of specific appearance attributes from inherent identity differences. Furthermore, previous work has largely focused on latent aggregate constructs like "attractiveness" or broad demographic categories (race, gender), leaving a gap in understanding how specific, controlled visual attributes—such as clothing style, grooming, or accessories—shift social perceptions when identity is held constant.

Methodology: The StylisticBias Benchmark
To address this, the authors introduce StylisticBias, a controlled benchmark designed to evaluate attribute-level social bias in MLLMs. The methodology relies on a two-stage generation pipeline to isolate visual variables:

1. Base Face Generation: The authors generated 500 photorealistic base faces using Imagen 4. These faces cover a Cartesian product of four demographic dimensions: age (young, middle-aged, elderly), gender (male, female), ethnicity (Asian, African, European, Middle Eastern, Latino), and body type (thin, normal, obese). These base images serve as identity anchors, featuring standardized studio lighting, neutral expressions, and no accessories or makeup.
2. Controlled Variation Generation: Using Nano Banana (Gemini 2.5 Flash Image), the authors created approximately 50 single-attribute variations for each base face, resulting in roughly 25,000 images. Each variation modifies exactly one visual attribute (e.g., changing hair style, adding tattoos, altering clothing style) while preserving the base identity, pose, and lighting. This design ensures that any shift in model judgment can be attributed to the specific altered attribute rather than identity changes.
3. Evaluation Protocol: Six open-source MLLMs (ranging from 4B to 14B parameters, including LLaVA-v1.6, Qwen3-VL, and Gemma variants) were evaluated in a zero-shot setting. The models performed binary forced-choice judgments across 25 social scenarios grounded in the warmth–competence framework (e.g., "Stylish vs. Unstylish," "Wealthy vs. Poor," "Trustworthy vs. Untrustworthy"). To mitigate prompt sensitivity, each image-scenario pair was evaluated under 12 variations (4 prompt orderings × 3 random seeds), totaling approximately 28.3 million judgment calls.

Key Contributions

1. StylisticBias Benchmark: A dataset of 500 base identities and ~25,000 synthetic images with single-attribute edits, enabling fine-grained bias evaluation where identity is fixed.
2. Large-Scale Evaluation: A comprehensive assessment of six MLLMs across 25 binary social judgment scenarios, providing a granular view of how different architectures respond to visual cues.
3. Empirical Findings on Bias Concentration: The study identifies that bias is not uniformly distributed but concentrated in a small set of visual cues, particularly those related to self-presentation.

Results
The experiments yielded several consistent patterns across the evaluated models:

• Demographic Drivers: Age and body type emerged as the strongest demographic drivers of social judgment (Variation Strength, VS = 0.075 and 0.069, respectively). Obese and elderly identities were systematically associated with less favorable trait attributions. In contrast, ethnicity and gender showed significantly smaller effects (VS = 0.038 and 0.030).
• Attribute-Level Shifts: Approximately 15 visual attributes accounted for nearly 80% of the total bias variation. Fashion style produced the largest shifts, followed by facial hair and makeup. Conversely, attributes like skin irregularities and hair color showed near-zero impact.
• Asymmetry of Cues: Negative cues (e.g., worn/distressed clothing, messy hair) produced significantly larger shifts in model judgment than their positive counterparts, mirroring human negativity bias.
• Semantic Alignment Bias: Model sensitivity was highest when the judged trait was semantically aligned with visible appearance. Socioeconomic judgments (e.g., "Wealthy vs. Poor") and style-related judgments (e.g., "Stylish vs. Unstylish") were disproportionately sensitive to visual changes compared to internal traits like "Honest" or "Loyal."
• Model Consistency: While models differed in the magnitude of their responses (e.g., Pixtral was the most reactive, Qwen3 the most conservative), they largely agreed on which cues mattered. Larger models tended to attenuate effect magnitudes but preserved the overall sensitivity structure.
• Contextual Interactions: The impact of cues was moderated by demographic context. For instance, the positive effect of formal clothing was amplified for elderly faces and obese bodies, while the negative effect of "streetwear" shifted from negative for young faces to neutral/positive for elderly faces.

Significance and Claims
The paper claims that StylisticBias provides a necessary foundation for moving beyond coarse demographic comparisons toward controlled visual attribution. The authors argue that MLLMs are systematically sensitive to how a person looks (specific cues like fashion and grooming) rather than just who they are represented as being.

The study highlights that bias in multimodal systems is concentrated in a small set of self-presentation cues, particularly those linked to socioeconomic status and style. This suggests that standard evaluations focusing only on broad demographic groups may miss the primary drivers of appearance-based bias. By releasing the benchmark and dataset, the authors aim to support fairness auditing and the development of mitigation strategies for appearance-driven bias in consequential AI applications.

Limitations
The authors acknowledge that the study uses controlled synthetic images rather than real photographs, which limits the generalizability to all real-world deployments but was a deliberate choice to ensure identity isolation and ethical safety. Additionally, the study focuses on a curated subset of demographic groups and visual attributes to maintain interpretability, meaning it identifies which cues drive shifts but does not exhaustively cover all socially meaningful identities or explain the underlying mechanisms of these effects.