💬 NLP

The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping

This paper introduces the Visual Iconicity Challenge, a novel video-based benchmark that evaluates 13 state-of-the-art vision-language models on their ability to map dynamic sign language forms to meanings, revealing that while current models show some sensitivity to visually grounded structures, they still significantly lag behind human performance in predicting phonological forms and inferring meaning from iconicity.

Original authors: Onur Keleş, Aslı Özyürek, Gerardo Ortega, Kadir Gökgöz, Esam Ghaleb

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Onur Keleş, Aslı Özyürek, Gerardo Ortega, Kadir Gökgöz, Esam Ghaleb

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to understand human language, but instead of just listening to words, you want it to understand sign language. Sign language is unique because it's not just about moving hands; it's about how the shape and movement of your hands can actually look like the thing you are talking about. This is called iconicity.

For example, the sign for "cut" might look like you are holding scissors and snipping something. The sign for "butterfly" might look like wings flapping. This paper is a report card for the smartest AI vision models (robots that can see and read) to see if they can understand these visual clues.

Here is a breakdown of what the researchers did and what they found, using simple analogies:

The Challenge: The "Visual Riddle" Test

The researchers created a game called the Visual Iconicity Challenge. They took 96 short videos of signs from Dutch Sign Language and asked 17 different AI models to play three specific games:

The "Describe the Dance" Game (Phonology):
- The Task: The AI had to look at a video and describe the "moves." Did the signer use one hand or two? Was the hand shaped like a fist or an open palm? Was the movement a straight line or a circle?
- The Analogy: Imagine asking a robot to watch a dancer and tell you exactly which foot they stepped with and how they held their arms.
- The Result: The AI was surprisingly good at this. It found it easier to spot where the hands were (like near the head) than what shape the hands were making. Interestingly, this mirrors how human children learn sign language: they figure out location before they master complex hand shapes.
The "Guess the Word" Game (Transparency):
- The Task: The AI watched a sign video and had to guess the meaning without being told what it was. Could it look at a hand moving like wings and guess "butterfly"?
- The Analogy: This is like showing a stranger a mime act and asking them to guess the story.
- The Result: This was very hard for the AI. Even the smartest models only guessed correctly about 29% of the time. They struggled to connect the visual movement to the actual word. They were better at guessing signs that looked like static objects (like a telephone) but failed at signs that represented actions.
The "How Much Does It Look Like?" Game (Iconicity Rating):
- The Task: The AI had to rate, on a scale of 1 to 7, how much a sign "looked like" its meaning.
- The Analogy: Imagine a human rating a drawing of a cat. A realistic drawing gets a 7; a stick figure gets a 2. The AI had to do this for signs.
- The Result: The top AI models were actually quite good at this. Their ratings matched human ratings very closely. If humans thought a sign was very "iconic" (looked like the meaning), the AI agreed.

The Big Surprise: The "Object vs. Action" Bias

Here is the most interesting twist. Humans have a natural preference: we find signs that look like actions (like "brushing teeth") easier to understand and more iconic than signs that look like objects (like a "telephone").

However, the AI models had the exact opposite preference.

Humans: "I love the action signs; they make sense!"
AI: "I prefer the object signs; they look like static pictures!"

The researchers explain this by saying AI models are like tourists who only look at postcards (static images) and miss the movie (the movement). Because these models are trained heavily on still photos, they get confused when a sign is about doing something rather than being something.

The Secret Sauce: Thinking Helps

The researchers found that when they told the AI to "think out loud" (a feature called "thinking mode") before giving an answer, the models got much better at rating how iconic a sign was. It was like telling a student, "Don't just guess; explain your reasoning first." This simple step helped the open-source models catch up to the expensive, closed-source models.

The Bottom Line

What they did: They tested if AI can understand the visual "logic" of sign language.
What they found:
- AI is getting good at spotting the physical details of signs (hand shapes, locations).
- AI is decent at rating how "visual" a sign is.
- But, AI is terrible at guessing the meaning of a sign just by watching it, and it has a weird bias where it likes "object" signs more than "action" signs, which is the opposite of how humans think.
The Takeaway: To make AI truly understand sign language, we need to teach it to pay attention to movement and action, not just static shapes. The paper proves that if an AI can understand the physical "grammar" of the sign (the phonology), it is better at understanding the meaning, but it still needs to learn how humans connect movement to ideas.

Technical Summary: The Visual Iconicity Challenge

Problem Statement
Vision–Language Models (VLMs) are increasingly capable of processing multimodal data, yet their ability to ground visual forms in semantic meaning remains a critical test of their alignment with human-centric signals. Sign languages offer a unique natural testbed for this problem due to the pervasive nature of iconicity—the non-arbitrary resemblance between a sign's visual form (handshape, movement, location) and its meaning. While previous research has explored sound symbolism in text-only models and static image-captioning in VLMs, there is a lack of systematic evaluation regarding how well VLMs perceive the dynamic, visuospatial structure of sign languages and map it to conceptual meaning. Specifically, it is unclear whether models can distinguish between arbitrary and iconic signs, infer meaning from visual form alone (transparency), or replicate human sensitivity to graded iconicity.

Methodology
The authors introduce the Visual Iconicity Challenge, a video-based benchmark derived from the Sign Language of the Netherlands (NGT). The dataset consists of 96 isolated sign videos (64 iconic, 32 arbitrary), manually annotated with ground-truth phonological features, iconicity types, and human-rated iconicity scores (1–7 scale).

The evaluation framework assesses 17 state-of-the-art VLMs (including 11 open-source and 6 proprietary models) across three complementary tasks:

Phonological Form Prediction: A multi-class classification task where models predict five discrete phonological parameters from the video: handshape, location, path shape, path repetition, and handedness.
Transparency: An open-set and closed-set (10-choice) task requiring models to infer the lexical gloss (meaning) of a sign based solely on its visual form, minimizing reliance on textual priors.
Graded Iconicity Ratings: A regression task where models assign a 1–7 score representing the degree of resemblance between the sign's form and its meaning.

The study evaluates models in both zero-shot and few-shot (4-shot) settings. Human baselines were established using a deaf signer (native to Turkish Sign Language, non-native to NGT) and hearing sign-naïve participants. The authors also analyze the correlation between phonological prediction accuracy and iconicity/transparency performance to test the hypothesis that form competence underpins meaning mapping.

Key Contributions

Benchmark Creation: The introduction of the Visual Iconicity Challenge, the first large-scale diagnostic benchmark for sign language iconicity in VLMs, featuring dense phonological annotations and human baselines.
Human Baselines: Collection of performance data from sign-naïve humans and a deaf signer for phonology and transparency tasks, providing upper and lower bounds for model evaluation.
Systematic Evaluation: The first large-scale zero- and few-shot assessment of diverse VLMs on sign language form–meaning mapping, revealing specific biases and failure modes.
Resource Release: Public availability of evaluation code, annotations, and human baselines to ensure reproducibility.

Results

Phonological Prediction: Models generally mirror human acquisition patterns, finding location and handedness easier to predict than handshape and path shape. While large models (e.g., Gemini 3.1 Pro, GPT-5.4) achieve moderate accuracy (up to ~~0.73 mean), a significant gap remains compared to human baselines (~~0.79). Few-shot prompting yields modest gains, primarily for smaller models.
Transparency: Performance is low across the board. In open-set identification, the best model (Gemini 3.1 Pro) achieves only ~~29% accuracy, far below the human deaf signer baseline (~~59%). Even in 10-choice settings, performance remains limited, suggesting a fundamental difficulty in grounding visual form to lexical meaning without strong textual priors.
Iconicity Ratings: Models show moderate-to-strong correlation with human iconicity ratings (Spearman's $\rho$ up to 0.774 for Gemini 3.1 Pro). However, models exhibit a systematic bias: while humans prefer action-based iconic signs, most open-source models display a reverse preference for object-based signs. Closed-source models show less of this inversion.
Correlations: There is a strong positive correlation ( $\rho = .769$ ) between a model's phonological form prediction accuracy and its ability to rate iconicity, suggesting that sensitivity to structured bodily properties is a prerequisite for form–meaning mapping.
Reasoning Impact: Enabling "thinking mode" (chain-of-thought) in Qwen3-VL-32B significantly improved its scale calibration ( $\kappa_w$ ), bringing open-model performance closer to closed-model levels.

Significance and Claims
The paper claims that the Visual Iconicity Challenge successfully validates diagnostic tasks for probing VLMs on visually grounded structure. The findings indicate that while current VLMs can distinguish iconic from arbitrary signs and align with human phonological difficulty patterns, they fail to infer lexical meaning from visual form alone and exhibit a static, object-based bias that inverts human preferences for action-based signs.

The authors conclude that explicit reasoning (thinking mode) can narrow the calibration gap between open and closed models. They argue that bridging the gap between model performance and human-centric signals requires richer pretraining on gesture/sign data and better dynamic pose encoding, rather than relying solely on static visual features. The study positions these diagnostic tasks as essential for understanding the limits of current multimodal grounding in complex, temporally extended visuospatial languages.

The Challenge: The "Visual Riddle" Test

The Big Surprise: The "Object vs. Action" Bias

The Secret Sauce: Thinking Helps

The Bottom Line

Technical Summary: The Visual Iconicity Challenge

More like this