💻 computer science

Human-AI Perceptual Alignment by Playing Hues and Cues

This paper introduces a novel evaluation framework using the board game Hues and Cues to assess human-AI perceptual alignment in color, revealing that while Contrastive Vision-Language Models replicate human biases for concrete objects, they systematically fail in abstract domains due to semantic misclassification and uncertainty collapse, a problem mitigated more effectively by curated pre-training datasets than by massive uncurated corpora.

Original authors: Nuria Alabau-Bosque, Jorge Vila-Tomás, Paula Daudén-Oliver, Pablo Hernández-Cámara, Valero Laparra, Jesús Malo

Published 2026-08-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nuria Alabau-Bosque, Jorge Vila-Tomás, Paula Daudén-Oliver, Pablo Hernández-Cámara, Valero Laparra, Jesús Malo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to see the world, not just by recognizing shapes, but by understanding the feeling of things. In the world of artificial intelligence, there is a special type of brain called a "Vision-Language Model." Think of these models as super-smart students who have read billions of books and looked at billions of photos from the internet. They are great at saying, "That is a dog," or "That is a car." But can they understand the color of a dog? Can they know that a banana is usually yellow, not because it's physically yellow in every single photo, but because our brains have a "memory" that bananas are yellow? This is the tricky part: human color isn't just about light hitting our eyes; it's about our culture, our memories, and our shared stories. Scientists want to know if these AI students are learning to see like humans, or if they are just guessing based on the weird patterns they found in their massive data piles.

To find out, a team of researchers decided to play a game. They didn't use boring charts or complex math tests. Instead, they used a board game called "Hues and Cues." Imagine a giant board with 480 tiny squares, each a slightly different color, arranged like a rainbow map. The game works like this: someone says a word (like "Banana" or "Hope"), and you have to point to the square on the board that best matches that word. It sounds simple, but it reveals how our brains connect words to colors. The researchers used this game to test 162 different AI models, asking them to play along with 325 real humans. They wanted to see if the AI's "guesses" matched the crowd's "memory," or if the AI was getting lost in its own head.

Here is what happened when the robots sat down to play.

The Setup: A Digital Board Game
The researchers built a digital version of the "Hues and Cues" board. This board had 480 distinct color squares, mapped out so they could be measured scientifically. They gathered a massive group of 325 people to play the game on their own phones and computers. Each person was asked to match 100 different words to the best color on the board. Some words were easy and physical, like "Banana" or "Blood." Others were tricky and abstract, like "Hope," "Feminism," or "Pop Culture."

To make sure the AI wasn't deviating from the protocol, the researchers gave the models the exact same digital board. The AI's job was to look at a word and pick the top 5 color squares it thought matched best. The researchers then compared the AI's choices to the "Human Consensus"—the average answer given by the real people. They even calculated a "Human Error Rate," which is basically how much humans disagree with each other. This gave them a perfect baseline: if an AI is as good as a human, it should make about the same amount of "mistakes" as a person does.

The Big Discovery: The "Blue" Crash
The results were a mix of impressive wins and a very strange, systematic failure.

First, the good news: When the words were about real, physical objects, the AI was surprisingly good. For concrete things like "Banana," "Pumpkin," or "Pig," the AI models often picked the exact same colors as the humans. They understood that bananas are yellow and pigs are pink. In fact, for some of these physical objects, the AI was so precise that it even matched the "idealized" memory color humans have, rather than just the literal color of a specific photo. This suggests that when there is a clear, physical reality, the AI can learn the human "vibe."

However, when the game moved to abstract or cultural concepts, the AI hit a wall. For words like "Hope," "Rage," "Feminism," or even "Tractor" (which has a specific cultural meaning in Spain), the AI stopped making sense. Instead of picking a color that matched the feeling or the culture, the models started doing something bizarre: they all collapsed into a single, default color.

The researchers found that when the AI didn't know the answer, it didn't guess randomly. Instead, it almost always picked a specific shade of blue (a coordinate they call "I18" on the board). It was as if the AI's brain short-circuited and said, "I don't know what this means, so I'll just pick blue." This happened over and over again, regardless of whether the word was "Winter," "Stitch" (from the movie), or "Axolotl."

Why Did This Happen?
The team dug deeper to find out why the AI was so obsessed with blue. They tested the models with "nonsense" words—things that have no meaning at all, like random letters or stop words like "the" and "and." They found that about 42% of the models had a built-in bias. When they saw nonsense, they didn't guess randomly; they defaulted to specific colors, often that same blue or sometimes yellow.

This revealed a critical flaw: the AI wasn't really "understanding" the word "Hope." It was just following a pattern it learned from its training data. Because the internet has so many pictures of the sky and water (which are blue), the AI learned that when it is confused, "blue" is a safe bet. It's like a student who, when they don't know the answer to a history question, just writes "Blue" because they saw that word a lot in their textbook.

The Role of Data Quality
The researchers also looked at how the AI was trained. They compared models trained on massive, messy piles of internet data (like the LAION dataset) with models trained on smaller, carefully cleaned and curated datasets.

They found that the messy, huge datasets didn't actually make the AI smarter at colors. In fact, models trained on the carefully curated, high-quality data were much better at matching human colors. The sheer size of the data didn't matter as much as the quality. It's like studying for a test: reading a million random, unedited blog posts might give you a lot of facts, but reading a well-edited textbook helps you understand the concepts better. The "clean" models made fewer mistakes and were less likely to crash into that default blue color.

The Takeaway
So, what does this all mean? The paper shows that while AI is getting really good at recognizing physical objects and their colors, it still struggles with the messy, cultural, and emotional side of human perception. When the AI gets confused, it doesn't admit uncertainty; it just defaults to a "safe" color, usually blue, based on what it saw most often in its training data.

The researchers suggest that to make AI truly understand us, we can't just feed it more data. We need to teach it to handle uncertainty without crashing, and we need to use better, cleaner data to help it learn the subtle, cultural rules of color that humans follow naturally. The game of "Hues and Cues" proved that even the smartest AI models still have a long way to go before they can truly see the world the way we do.

Technical Summary: Human-AI Perceptual Alignment by Playing Hues and Cues

Problem Statement
Current Contrastive Vision-Language Models (CVLMs), such as CLIP and its variants, have demonstrated robust zero-shot capabilities and strong semantic links between visual features and natural language. However, fundamental gaps remain in their fine-grained perceptual grounding, particularly regarding color. Traditional benchmarks often overlook the nuanced, culturally anchored, and cognitively biased nature of human color perception (e.g., "memory colors"). Furthermore, while CVLMs are known to suffer from semantic hallucinations or distribution collapses when facing abstract concepts, literature has not yet bridged these phenomena to explore how semantic ambiguity triggers systematic chromatic collapses within a psychophysically grounded space. Existing evaluation methods, relying on rigid bounding boxes or continuous color pickers, fail to capture the subjective, human-centric nuances of color association.

Methodology
The authors propose a novel evaluation framework leveraging the gamified, discrete color space of the board game Hues and Cues to assess the perceptual alignment of 162 CVLMs against a robust human baseline.

Experimental Environment: The study utilizes the Hues and Cues board, a $16 \times 30$ grid containing 480 unique discrete colors. These cells are mapped to the CIE $xy$ chromaticity diagram, transforming subjective grid coordinates into an objective, measurable colorimetric space. This discrete space overcomes the infinite search space issues of continuous color pickers.
Human Baseline Acquisition: A custom digital interface was used to collect color associations for a curated 100-word vocabulary (spanning seven semantic categories: Animal, Food, Mineral, Plant, Environment, Subjective, and Pop Culture) from 325 human observers. The dataset was filtered to exclude participants with Color Vision Deficiency (CVD), resulting in a validated cohort of 316 normal trichromats. To establish an empirical lower bound for error, the authors calculated a "Human Consistency" baseline using Leave-One-Out (LOO) cross-validation, quantifying the inherent noise and semantic disagreement within the human population itself.
Model Evaluation: The study evaluated 162 models across various architectural families (OpenAI, SigLIP, EVA, MobileCLIP, CoCa, etc.) and pre-training datasets (ranging from massive uncurated web scrapes like LAION to highly curated datasets like DataComp and DFN). For each word, models were tasked with retrieving the top-5 most semantically similar color patches from the 480-cell board.
Metrics:
- Mahalanobis Distance: Used to detect outliers by measuring the distance of a model's Top-1 prediction from the human consensus distribution (modeled as a bivariate normal distribution).
- Hotelling's $T^2$ Test: Used to assess distributional alignment between the model's Top-5 "confidence cloud" and the human response set.
- Nonsense Baseline: To distinguish between semantic misclassification and uncertainty collapse, models were tested on 16 semantically void inputs (numbers, stop words, pseudowords, random strings). This protocol identifies if models default to a specific chromatic prior (e.g., a "blue bias") when faced with ambiguity.
Categorization: The 480-cell board was tessellated into eight fundamental color categories (Brown, Red, Orange, Yellow, Green, Blue, Purple, Pink) to enable qualitative analysis of semantic severity (e.g., distinguishing a minor coordinate error from a categorical failure).

Key Results

Concrete vs. Abstract Divergence: CVLMs successfully replicate human cognitive biases for concrete physical referents (e.g., Banana, Pumpkin, Pig), often matching or exceeding human consistency in Top-1 predictions. However, they systematically diverge in abstract, subjective, and pop-culture domains (e.g., Hope, Feminism, Tractor).
Failure Modes: Two distinct failure modes were identified in severely misaligned concepts:
1. Semantic Misclassification: The model associates a concept with an entirely incorrect color category (e.g., predicting Green for Wheat or Pink for Ruby).
2. Uncertainty Collapse: When facing semantic ambiguity, models do not sample randomly but systematically collapse into a default blue coordinate (specifically I18, an achromatic gray-blue), disregarding the concept's actual visual properties.
Data Curation vs. Scale: The quality of pre-training data proved more critical than sheer volume. Models trained on highly curated or proprietary datasets (e.g., DataComp, DFN, WebLI) demonstrated significantly tighter error distributions and lower failure rates compared to those trained on massive, uncurated web scrapes (e.g., LAION, YFCC).
Uncertainty and Bias: Approximately 42% of the evaluated models exhibited biased behaviors under the "Nonsense Baseline," either collapsing into a specific focal color or a neutral achromatic coordinate. This indicates a risk of false positives where a model's apparent semantic alignment may be an artifact of its intrinsic chromatic prior rather than genuine understanding.
Architectural Differences: While newer architectures like CoCa and EVA showed long-tail error distributions, compact architectures like MobileCLIP exhibited narrower distributions, often approaching human baselines in concrete categories.

Significance and Claims
The paper claims that despite broad categorization capabilities, current CVLMs fail to capture the nuanced, localized consensus of human color memory, particularly in abstract domains. The study highlights that:

Gamified tasks are effective tools for exposing underlying model biases and abstraction failures that traditional testing strategies miss.
Data curation is a more effective mechanism for mitigating severe perceptual misalignments than increasing model complexity or training data volume.
Uncertainty collapse into default chromatic coordinates is a systemic issue in CVLMs, analogous to hallucinations in other domains, which can lead to misleading evaluations of semantic understanding.
The proposed framework provides a rigorous, reproducible method to establish an empirical lower bound for model performance, distinguishing between genuine semantic grounding and statistical coincidence.

The authors emphasize that their findings underscore the necessity of deploying human-centric tasks to reveal latent cognitive deficiencies in modern CVLMs, rather than relying solely on broad categorization benchmarks. All data and code are made publicly available to facilitate further metric testing.

Technical Summary: Human-AI Perceptual Alignment by Playing Hues and Cues

More like this