💻 computer science

Do Vision-Language Models Agree on the Affective Qualities of Shape? A Cross-Model Audit for Generative Design Interfaces

This paper audits six vision-language models to evaluate their consistency in ranking 3D objects by affective Kansei descriptors, revealing that while models show partial agreement above chance levels, their convergence varies significantly by object category and semantic alignment, prompting a UI prototype to guide the selection of reliable semantic controls for generative design interfaces.

Original authors: Luca Bux, Thiago Rios, Ingo Scholtes, Stefan Menzel

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luca Bux, Thiago Rios, Ingo Scholtes, Stefan Menzel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the early stages of design, whether sketching a new chair or modeling a car, creators often rely on words to guide their tools. They might ask for something "more elegant," "less bulky," or "more minimalist." Today, artificial intelligence systems are beginning to understand these requests, using powerful computer programs that can see images and read text to generate new shapes based on such descriptions. These systems, known as vision-language models, act as a bridge between a designer's vague idea and a concrete 3D object. However, a critical question remains: do these different computer programs actually agree on what those words mean? If one program thinks a tall, thin bottle is "elegant" while another thinks a short, wide one is, the tool becomes unreliable. Designers need to know if the digital assistant is consistent before they trust it to shape their work.

A team of researchers from Honda Research Institute Europe and Julius-Maximilians-Universität Würzburg set out to answer this question by auditing six of the most advanced vision-language models available. They wanted to see if these independent programs, built by different companies and trained on different data, would rank the same objects in the same way when asked to judge their emotional or "affective" qualities. To do this, they stripped away everything that could distract the computers. They took thousands of 3D models of everyday items like chairs, lamps, and jars and rendered them as plain, untextured grey shapes. By removing color, material, and texture, the researchers ensured the models were judging the objects based on form alone, much like a sculptor looking at a clay model before painting it.

The researchers then asked each of the six computer models to rank these grey shapes along various scales. Some scales were straightforward physical descriptions, such as "tall versus short" or "boxy versus curvy," which served as a control to see if the models could even agree on basic geometry. Others were the more complex, human-like descriptors the designers actually use, such as "modern versus traditional" or "elegant versus messy." Finally, they included a set of nonsense pairings, like "loud versus quiet," to establish a baseline for random agreement. If the models agreed on these irrelevant pairs, it would mean their rankings were just noise.

The results revealed a nuanced picture of agreement. The models were remarkably consistent when judging simple physical traits; they largely agreed on which objects were tall or short. When it came to the more abstract, emotional qualities, the models showed a moderate level of agreement that was significantly better than random chance, but not perfect. On average, the models agreed on the ranking of objects for emotional terms about 36 percent of the time, compared to a 14 percent agreement rate for the nonsense terms. This suggests that while the computers have developed a shared sense of how shape relates to feeling, they are not yet perfectly aligned.

Crucially, the study found that this agreement is not uniform across all types of objects. For some categories, like jars, the models agreed strongly on what looked "elegant" or "luxurious," with agreement rates reaching over 50 percent. For other categories, like bookshelves, the models struggled to find a common ground, with agreement rates dropping as low as 21 percent. The researchers discovered that this variation depends less on how different the objects look from one another and more on whether the specific shape variations in that category actually match the direction the word is pointing. For instance, if a group of chairs varies mostly in height, a model might easily agree on which is "tall," but if the group varies in subtle curves that don't align with the concept of "elegance," the models will disagree.

The researchers also tested whether these agreements were specific to the object type or just a general quirk of the text. They found that a descriptor like "plush versus rigid," which makes sense for sofas, did not work well when applied to bottles. This confirmed that the models are not just applying a generic rule but are actually interpreting the relationship between the specific shape of an object and the word used to describe it.

To make these findings useful for real-world design, the team built a prototype interface that visualizes this agreement. In this tool, a designer can select an object and see a list of potential controls, such as "make it more modern." Next to each control, the system displays a score indicating how much the different computer models agree on that instruction. If the score is high, the designer can be confident that the tool will behave predictably. If the score is low, the interface suggests that the control might be unreliable for that specific object and could be withheld or replaced with a better-supported alternative.

This work does not claim that the computers have learned human feelings or that their agreement proves they are right. Instead, it offers a practical method to check if a digital tool is stable before it is handed over to a user. By treating the agreement between different computer models as a form of quality control, designers can identify which emotional descriptors are safe to use and which might lead to confusion. The study suggests that while artificial intelligence is becoming a powerful partner in design, we must still carefully audit its understanding of the world, ensuring that when a designer asks for "elegance," the machine is looking at the same thing they are.

Technical Summary: Cross-Model Audit for Affective Qualities of Shape in Vision-Language Models

Problem Statement

Generative design interfaces increasingly utilize semantic controls (e.g., "more elegant," "more minimalist") to steer output, typically encoded by Vision-Language Models (VLMs). A critical practical question remains: do state-of-the-art VLMs represent objects consistently regarding these affective concepts? While human validation is the ultimate goal, large-scale annotated datasets of affective 3D shapes are scarce and expensive to produce. Furthermore, existing aesthetic evaluation research often focuses on photographs where texture, lighting, and context confound geometric form. The authors posit that before exposing semantic controls to users, it is necessary to determine if independently developed VLMs produce consistent rankings of affective attributes, independent of human labels.

Methodology

The authors conducted a "cross-model consistency audit" using six state-of-the-art VLMs: CLIP, OpenCLIP, SigLIP2, ALIGN, FLAVA, and Qwen3-VL-Embedding. These models were selected for their structural diversity in training objectives, architectures (ViT vs. EfficientNet), and data provenance.

Experimental Setup

Stimuli: 4,950 untextured, uniform grey 3D objects were sampled from 10 categories in the ShapeNetCore database (chair, table, lamp, sofa, cabinet, bookshelf, bottle, jar, clock, car). Objects were rendered from 8 fixed viewpoints to isolate geometric form while removing surface cues like color and texture.
Representation Extraction: Each view was encoded by the 6 VLMs. View-level embeddings were averaged and unit-normalized to create a single object-level representation.
Probe Design: The study utilized bipolar adjective pairs organized into three tiers:
- Geometric (Positive Controls): 7 pairs (e.g., tall–short, boxy–curvy) representing verifiable physical attributes.
- Kansei (Affective): 32 pairs describing stylistic and affective impressions (e.g., modern–traditional, elegant–messy). These included 3 "shared-core" pairs evaluated across all categories and category-specific pairs derived from Kansei engineering literature and LLM expansion.
- Irrelevant (Empirical Null): 20 pairs with no physical connection to static 3D shapes (e.g., loud–quiet, sweet–sour).
Directional Projection: For each model and word pair, a semantic direction vector was constructed by subtracting the text embedding of the antonym from the adjective. Object representations were projected onto these vectors to generate continuous scores.
Evaluation Metric: Cross-Model Convergence was defined as the mean pairwise Spearman rank correlation ( $\bar{\rho}$ ) of object scores across all 15 unique pairs of the 6 encoders. This measures whether different models agree on the ranking of objects along a semantic axis, rather than absolute scores.
Analysis: The study analyzed convergence across tiers, categories, and axes. It also investigated the relationship between convergence and "variation-subspace alignment" (whether the semantic direction aligns with the principal components of shape variation within a category).

Key Results

1. Convergence Across Tiers

The audit revealed a clear hierarchy in cross-model agreement:

Geometric axes showed the highest convergence ( $\bar{\rho} = 0.44$ ).
Affective (Kansei) axes showed moderate but significant convergence ( $\bar{\rho} = 0.36$ ), which is substantially higher than the empirical null ( $\bar{\rho} = 0.14$ ) but lower than geometric controls.
Irrelevant axes clustered near the null, establishing a baseline for spurious agreement.
A randomly chosen affective axis out-converged a randomly chosen irrelevant axis in 90.6% of cases (Common Language effect size = 0.906).

2. Category and Axis Dependence

Convergence is not uniform; it is highly dependent on the specific (category, axis) pair:

Category Variation: On the three shared-core axes, convergence ranged from 0.21 for bookshelves to 0.51 for jars.
Axis Specificity: Some affective descriptors (e.g., minimalist–ornate, $\bar{\rho}=0.53$ ) showed strong agreement, while others (e.g., natural–artificial, $\bar{\rho}=0.14$ ) fell to the level of the null.
Alignment with Variation: The study found a strong positive correlation ( $\rho \approx 0.72$ ) between convergence and the alignment of the semantic direction with the dominant subspace of shape variation within a category. Convergence is high when the semantic direction captures actual geometric variation present in the objects; it is low when the direction points into low-variance subspaces, even if the objects vary significantly in other ways.

3. Robustness and Specificity

Encoder Independence: Agreement was not driven by model redundancy (e.g., CLIP and OpenCLIP showed lower agreement than pairs with different architectures like SigLIP2 and Qwen3-VL).
Object Specificity: Category-specific affective axes (e.g., plush–rigid for sofas) converged significantly better when applied to their target category than to foreign categories, suggesting the signal is tied to the object class rather than being a generic text-space artifact.
Prompt Sensitivity: Using single-pole embeddings reduced convergence and increased the null baseline, confirming the necessity of bipolar subtraction to cancel shared anisotropic components.

Key Contributions

Offline Cross-Model Audit: The authors introduce a method to audit affective semantic controls in AI-assisted design without requiring human labels or model retraining. The audit is calibrated using geometric positive controls and an empirical null.
Characterization of Reproducibility: The study demonstrates that reproducible signals for affective controls exist but are conditional. Convergence depends primarily on the alignment between the semantic direction and the specific variation structure of the object category, rather than just the total amount of shape variation.
Interface Design Support: The authors provide a UI prototype demonstrating how audit results can inform interface design. The system exposes convergence scores alongside semantic controls, allowing users to see which descriptors are supported by consistent model behavior and which are uncertain.

Significance and Claims

The paper argues that cross-model convergence serves as a valuable first-stage consistency audit for semantic controls in generative design interfaces.

Practical Utility: It offers an inexpensive, scalable method to identify which affective descriptors are sufficiently stable across diverse models to warrant exposure as interactive controls.
Transparency: By visualizing model agreement, the proposed interface helps users distinguish between patterns shared across VLMs and behaviors specific to a single model, potentially reducing over-reliance on system outputs.
Limitations and Modesty: The authors explicitly state that cross-model convergence does not imply agreement with human perception or independent validation of human Kansei judgments. It is a model-based consistency criterion, not a substitute for human validation. Low-agreement controls should be flagged as uncertain rather than discarded, as they may still be valid for specific cultural or design contexts not captured by the models.

The work concludes that while VLMs do not perfectly agree on affective qualities, they exhibit a measurable, non-random convergence that can be leveraged to build more transparent and uncertainty-aware intelligent design interfaces.