💻 computer science

Whose Art Counts? Model- and Prompt-Dependent Associations in Vision-Language Judgments of Museum Art

This paper introduces an archive-conditioned audit protocol to evaluate how vision-language models and prompt structures interact with museum metadata when assessing artistic value, demonstrating through a Metropolitan Museum of Art case study that "influence" prompts yield the most consistent category-based differences while highlighting the limitations of cross-model and cross-prompt generalization.

Original authors: Manpreet Singh, Nandakishor Reddy Pulagam, Rhythm Bhatia, Rahul Joshi

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Manpreet Singh, Nandakishor Reddy Pulagam, Rhythm Bhatia, Rahul Joshi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've just built a super-smart robot librarian who has read millions of books and seen billions of pictures. You ask it, "Which of these paintings is a masterpiece?" or "Which one changed history?" You expect the robot to look at the art and give a fair answer based on beauty and skill. But here's the catch: this robot doesn't just "see" art like a human does. It sees patterns. It learned by matching pictures to words on the internet. If the internet mostly said "great artists" were men, the robot might learn that "great art" looks like a man's work, even if it's wrong. This is the world of Vision-Language Models (VLMs): computers that connect images to words. The big question isn't just "Is the robot smart?" but "What kind of world did the robot learn from?" and "Does the robot's answer change if we ask the question differently?" Museums are full of these robots now, helping people find art, but if the robot is biased, it might hide the stories of women and other underrepresented creators, making them seem less important than they really are.

So, a team of researchers decided to put these robot librarians to a very specific test. They didn't just ask the robots to guess; they set up a strict game to see if the robots were playing fair. They grabbed 445 digital pictures of art from the Metropolitan Museum of Art. But here's the twist: they knew that museum records are messy. Many old artworks don't have a clear name attached to them. So, they split the group into two: a small group of 61 pieces where the creator's name was known (and they guessed the gender based on the name), and a huge group of 384 pieces where the creator was a mystery. They wanted to see if the robots gave "higher scores" to the named male artists compared to the named female artists.

But they didn't just ask one question. They asked the robots four different ways: "Is this a masterpiece?", "Is this quality work?", "Is this influential?", and a control question just to check if the robot was paying attention. They ran this test on four different robot brains (OpenAI CLIP, OpenCLIP B/32, OpenCLIP L/14, and SigLIP).

Here is what they found, and it's a bit of a plot twist: The robots didn't agree.

It wasn't a simple story where "all robots are sexist." Instead, the answer depended entirely on which robot you asked and how you asked.

When they asked about "Influence" (how much an artist changed history), two of the robots (OpenAI CLIP and OpenCLIP L/14) gave a clear signal: they rated the male-inferred works higher than the female-inferred ones. It was like the robot was saying, "Oh, history books say men changed things, so I'll give them a higher score."
However, when they asked about "Quality" or "Masterpiece," the results were a mess. One robot might say "Men are better," another might say "Women are better," and a third might say "I don't see a difference."
One robot, SigLIP, actually gave the opposite result, rating the female-inferred works higher, though the researchers noted the numbers were a bit wobbly and hard to pin down.

The most important thing the paper says is that you cannot just average these answers together. If you took the scores from all four robots and mashed them into one big number, you would get a lie. The robots are too different. One robot's "high score" is not the same as another robot's "high score." The researchers argue that the "bias" isn't a fixed flaw in the technology; it's a measurement problem. The result changes depending on the tool you use and the exact words you type.

The study also points out that the museum's own records played a huge role. Because 384 out of the 445 artworks had no known creator, the robots couldn't even compare them. The "named" artists were a tiny, specific slice of the museum's collection. The researchers warn that we can't say these robots hate women artists in general; we can only say that in this specific test, with these specific words and this specific set of pictures, some robots showed a pattern of favoring male names.

In the end, the paper doesn't say "Robots are broken" or "Robots are perfect." It says, "Be careful." If a museum uses a robot to rank art, the ranking might just be a reflection of how the robot was trained and how the question was asked, not the true value of the art. The researchers suggest that instead of looking for one "truth," we need to listen to the disagreement between the robots. If one robot says "This is a masterpiece" and another says "Nope," that disagreement is actually the most honest answer we can get: it tells us that the question is harder than we thought, and that the robot's opinion is just one noisy voice in a very loud room.

Technical Summary: Whose Art Counts? Model- and Prompt-Dependent Associations in Vision-Language Judgments of Museum Art

Problem Statement
Vision-language models (VLMs) are increasingly deployed to search, rank, and describe cultural collections. However, their judgments of "artistic value" (e.g., labeling an image a "masterpiece" or "influential") are not direct observations of merit but are responses to specific image-text associations learned from web-scale data. A critical measurement problem exists: when applying these models to museum archives, it is unclear whether observed disparities in scores (e.g., between male- and female-attributed works) reflect genuine model bias, learned associations from training data, or artifacts of the archive itself. Museum records are not neutral sampling frames; they contain historical imbalances in attribution, digitization, and cataloging. Existing audits often rely on benchmark images or broad demographic prompts, failing to account for how these effects behave within historically situated, query-defined archival cohorts where metadata is often incomplete or unresolved.

Methodology: Archive-Conditioned Audit Protocol
The study introduces an archive-conditioned audit design that treats the model, the prompt, and the metadata provenance as a single measurement procedure. The protocol adheres to four strict rules:

Archive Freezing: The dataset and image selection rules are fixed before inference to ensure a reproducible object cohort.
Metadata Resolution: Creator categories are derived strictly from documented fields and name-resolution procedures (using a "gender-guesser" dictionary for first names), while explicitly retaining "unknown or unattributed" records rather than discarding them.
Prompt Specification: Evaluative prompt pairs are fixed prior to testing. The study uses three value-oriented contrasts ("masterpiece," "quality," "influence") and one neutral control ("artwork"), comparing a high-value phrase against a lower-value alternative.
Instrument Separation: The study evaluates four distinct VLMs (OpenAI CLIP, OpenCLIP ViT-B/32, OpenCLIP ViT-L/14, and SigLIP) without fine-tuning. Recognizing that CLIP outputs softmax probabilities while SigLIP outputs logits, the authors avoid pooling raw scores. Instead, they perform within-model standardization for cross-model diagnostics, treating model disagreement as a feature of the measurement instrument rather than noise.

Experimental Setup

Data: 445 publicly accessible museum object images from the Metropolitan Museum of Art Open Access collection.
Cohort: A binary comparison cohort of 61 works (39 male-inferred, 22 female-inferred) derived from resolved metadata. The remaining 384 works (unknown/unattributed) are retained for descriptive context but excluded from the binary contrast.
Analysis: The study calculates prompt-relative score differences ( $\Delta_m$ ) between creator categories. It employs bootstrap 95% confidence intervals, Mann-Whitney U tests, and 10,000-label permutation tests. Sensitivity analyses include leave-one-department-out tests and cross-model instability diagnostics.

Key Results

Prompt and Model Dependence: There is no universal bias across models. The "Influence" prompt produced the clearest positive differences (male-inferred > female-inferred) for OpenAI CLIP ( $\Delta = 0.2892$ ) and OpenCLIP L/14 ( $\Delta = 0.3400$ ). However, OpenCLIP B/32 and SigLIP showed inconsistent or opposite directional trends for the same prompts.
Prompt Sensitivity: The choice of prompt significantly alters the observed association. For instance, OpenAI CLIP showed nearly zero difference for "masterpiece" and "quality" prompts but a strong positive difference for "influence." This indicates that the models are reacting to specific semantic associations (e.g., historical impact vs. canonical status) rather than a single latent construct of "artistic value."
Aggregate Instability: An aggregate index of value scores was reported only as a secondary summary because prompt correlations were weak or mixed. Cross-model diagnostics revealed high instability; for example, SigLIP produced negative aggregate differences while OpenCLIP L/14 produced positive ones.
Department Sensitivity: Leave-one-department-out analyses showed that directional trends (e.g., positive differences for OpenAI CLIP) remained stable across most department omissions, suggesting the association is consistent within this specific query-defined cohort, though not necessarily portable to other institutions.

Significance and Claims
The paper's primary contribution is methodological, not a definitive statement on artistic merit or a universal estimate of gender bias in art.

Measurement Protocol: The study establishes a protocol for auditing VLMs in cultural heritage that explicitly accounts for archival selection bias, unresolved metadata, and model-specific scoring scales.
Interpretive Boundaries: The authors assert that their findings describe associations between metadata-derived creator categories and model scores under specific prompts. They explicitly state that these results do not establish causal effects, artistic quality, or general model behavior.
Role of Disagreement: The paper argues that disagreement between models should be treated as meaningful evidence of evaluation uncertainty rather than statistical noise to be averaged away.
Responsible Use: The authors caution that model scores should not be used to rank artistic worth, replace curatorial judgment, or guide acquisition policies. Instead, these diagnostics serve to reveal how multimodal models handle cultural artifacts under specific prompt and metadata conditions, highlighting the need for transparency in how archives and models interact.

In summary, the study demonstrates that "whose art counts" in a VLM audit is contingent on the specific model, the phrasing of the prompt, and the archival survival of the metadata, rather than reflecting an objective or stable measure of cultural value.

Technical Summary: Whose Art Counts? Model- and Prompt-Dependent Associations in Vision-Language Judgments of Museum Art

More like this