Would you still call this Dax? Novel Visual References in VLMs and Humans

Technical Summary: Novel Visual References in VLMs and Humans

Problem Statement
Vision-language models (VLMs) are frequently exposed to novel visual concepts at inference time, yet it remains underexplored how these models map new visual references to language, particularly when such references contradict prior knowledge acquired during pre-training. While human learners utilize inductive biases (e.g., shape bias, mutual exclusivity) to rapidly acquire novel word-referent mappings, it is unclear whether VLMs exhibit similar behaviors or if they overgeneralize learned labels. Existing benchmarks often rely on simplified or known object classes, failing to capture the challenges of learning truly novel, open-ended stimuli.

Methodology
To address this gap, the authors introduce the Novel Visual References Dataset (NVRD), comprising 19,176 images across 90 visual concepts. The dataset is structured as follows:

• Object Categories: 90 base objects are divided into three categories: Known (common objects like chairs, assigned nonce names to test conflict with prior knowledge), Composed (hybrids of existing objects, e.g., boar-toaster), and Fully Novel (objects not resembling real-world entities).
• Perturbations: Each base image is subjected to up to 20 levels of controlled visual perturbations across 11 axes. These include low-level edits (Gaussian noise, scale, pixelation, JPEG compression, color shift) and higher-level structural edits (texture shift, background change, artistic style, shape deformation, part addition, and part removal). Perturbations are applied compoundingly to create a continuous axis of visual distance.
• Evaluation Paradigms: The study evaluates three open-source VLMs (Qwen-2 VL 7B, Idefics-3 8B, Molmo-2 8B) and two closed-source models (GPT-4o Mini, Gemini-2.5 Flash) using three probing methods:
  1. Name Generation: In-context learning with distractor images to assess nonce word adoption.
  2. Token Probabilities: Computing log-probabilities of nonce vs. "vanilla" (standard) labels to measure model confidence and bias.
  3. Dual-Image Likert-Scale Rating: Models rate agreement (1–7 scale) on whether a perturbed image shares the same nonce label as the original.
• Human Comparison: A crowd-sourced study with 30 participants collected 2,400 judgments on a subset of NVRD using the same Likert-scale setup to enable direct human–model comparison.

Key Contributions

1. NVRD Corpus: A new dataset of entirely novel, open-ended stimuli with controlled perturbations, designed to probe generalization in novel concept learning.
2. Human–Model Comparison: A direct evaluation of VLM generalization against human judgments, revealing both alignments and divergences in concept identity perception.
3. Analysis of Inductive Biases: An investigation into how VLMs handle conflicts between in-context learning and pre-trained knowledge, and how they prioritize visual features (shape vs. texture) in novel contexts.

Results

• Conflict with Prior Knowledge: Models struggle to acquire novel concepts in-context when they contradict prior knowledge. Known entities received the lowest nonce reference usage, with models (particularly GPT-4o Mini) preferring existing "vanilla" labels. This mirrors the human mutual exclusivity bias. Novel entities saw the highest nonce usage, indicating that the absence of competing labels lowers the threshold for adopting new mappings.
• Sensitivity to Perturbations: Both models and humans are most sensitive to structural perturbations (part removal, part addition, shape deformation), rating these as less acceptable than surface-level changes (color, texture, background). This suggests VLMs exhibit a shape bias similar to humans, even in novel referential contexts.
• Overgeneralization: While models and humans correlate in their sensitivity to perturbation types, models significantly overgeneralize. Human ratings drop sharply for structural changes (e.g., below 3/7 for part removal at level 10), whereas models remain higher (4–6/7). Models extend learned labels to stimuli that humans reject, showing less discrimination in concept boundaries.
• Category Sensitivity: Humans are substantially more influenced by object novelty, rating novel entities lower than known ones as perturbations increase. Model ratings remain relatively flat across categories, suggesting models rely more on visual similarity than the psychological essentialism humans apply to established categories.
• Ablation: An adversarial ablation confirmed that model overgeneralization is not merely a tendency to "agree with everything," as models still diverge from humans when presented with entirely different objects, though they remain more permissive than human judges.

Significance
The paper argues that while VLMs can acquire novel visual references, their generalization patterns diverge from humans in critical ways, particularly regarding overgeneralization and the handling of prior knowledge conflicts. This misalignment has implications for reliable human-agent interaction, where shared referents must be maintained in evolving environments. The authors contribute NVRD as a tool for cognitively grounded evaluations, aiming to bridge the gap between machine and human concept learning and communication.