TouchAI: Exploring human-AI perceptual alignment in touch through language model representations

1. Problem Statement

The paper addresses a critical gap in Human–AI alignment: specifically, perceptual alignment. While significant progress has been made in aligning AI with human values, goals, and visual perception, the alignment of AI with human tactile perception remains largely unexplored.

• The Challenge: Unlike vision (which can be quantified via RGB values) or hearing (frequency scales), touch is multidimensional, context-dependent, and lacks universal standards. Tactile descriptors (e.g., "soft," "rough") are often metaphorical and subjective.
• The Core Question: Do Large Language Models (LLMs), which are trained on vast text corpora, possess an internal representation of tactile experiences that aligns with human subjective judgments? Can an LLM understand that "silk satin is softer than cotton denim" based solely on textual descriptions of touch?
• The Risk: As AI systems increasingly mediate physical interactions (e.g., robotic manipulation, personalized product recommendations), a lack of perceptual alignment could lead to unreliable or frustrating user experiences.

2. Methodology

The authors designed a novel interactive task called "Guess What Textile?" to evaluate the perceptual alignment between humans and LLMs.

A. System Design & Task Flow

• Setup: Participants interact with a custom AI system in a controlled lab setting. They handle two textile samples inside a black box (to eliminate visual cues): a Target (unknown to the AI) and a Reference (known to the AI).
• Interaction: Participants verbally describe the tactile differences between the two fabrics.
• AI Mechanism:
  1. Embedding: The system uses the OpenAI text-embedding-3-small model to convert the 20 possible textile descriptions into pre-computed vector embeddings ($E_{textiles}$).
  2. Vector Search: The participant's comparative description is encoded into a difference vector ($v_{diff}$). The system combines this with the reference textile's vector ($v_{ref}$) using a directional offset formula:
     $$\tilde{v}_{predict} = v_{ref} + \alpha \frac{v_{diff}}{\|v_{diff}\|}$$
     (where $\alpha=2$ is a scaling factor).
  3. Prediction: The system calculates the cosine similarity between the predicted vector and all 20 textile embeddings, selecting the ID with the highest similarity as its guess.
  4. Iteration: If the guess is incorrect, the guessed textile becomes the new reference, and the process repeats (up to 5 attempts).

B. Experimental Setup

• Participants: 40 participants (non-experts, diverse backgrounds) recruited for an in-person study.
• Stimuli: 20 distinct textile samples covering four meta-categories (Natural, Animal, Regenerated, Synthetic), including materials like silk satin, cotton denim, wool, and faux leather.
• Metrics:
  1. Success Rate: Did the AI guess the correct textile within 5 attempts?
  2. Validity: (Subjective) How well did the AI's guess match the participant's verbal description (1–10 scale)?
  3. Similarity: (Subjective) How perceptually close was the guessed textile to the actual target (1–10 scale)?
• Analysis: The study employed linguistic analysis (TF-IDF, Sentence-BERT, OpenAI embeddings) to check if description consistency influenced AI performance. It also used Multidimensional Scaling (MDS) and Hierarchical Clustering to compare the geometry of the human perceptual space vs. the AI semantic space.

3. Key Contributions

1. First Exploration of Touch Alignment: This is the first study to systematically investigate the level of perceptual alignment between human touch experiences and LLM representations using an interactive, comparative task.
2. Mapping Sensory to Embedding Space: The authors successfully mapped tactile descriptions into LLM embedding spaces and measured representation similarities, revealing that LLMs exhibit systematic perceptual biases dependent on the material.
3. Interactive Task Design: The paper introduces a "Guess What Textile" framework that mirrors real-life scenarios (e.g., selecting clothing or fabrics) and can be adapted for other sensory domains (food, perfumes).
4. Empirical Evidence of Bias: The study provides quantitative evidence that LLMs are not uniformly "blind" to touch but are highly inconsistent, performing well on some materials (e.g., silk) and failing on others (e.g., denim).

4. Key Results

A. Overall Performance

• Low Success Rate: The AI achieved a 22.5% success rate (18/80 tasks), which is statistically indistinguishable from random guessing (23.7% baseline).
• Subjective Ratings:
  • Validity: Mean score of 5.27/10 (not significantly different from neutral).
  • Similarity: Mean score of 4.77/10 (significantly below neutral).
  • Correlation: Validity and Similarity were strongly correlated ($r=0.758$), suggesting that when the AI misunderstood the description, the resulting guess was also perceptually distant.

B. Material-Specific Bias

• High Alignment: Silk satin achieved a 100% success rate. The LLM's embedding space appears to have strong, well-defined representations for this material, likely due to rich textual associations in training data.
• Low Alignment: Cotton denim, faux leather, and faux fur had 0% success rates. The model failed to distinguish these materials despite participants providing clear descriptions.
• Confusion Matrix: The AI showed a tendency to predict specific "popular" textures (like linen plain or silk crepon) regardless of the actual target, indicating a bias toward certain clusters in the embedding space.

C. Human vs. AI Space Geometry

• Weak Structural Alignment: While the AI captured broad categorical knowledge (e.g., grouping wools together), it failed to replicate the fine-grained perceptual relationships humans perceive.
• Local Neighbors: Only 30% of the "nearest neighbors" (most similar textiles) in the AI's space matched those in the human perceptual space.
• Correlation: The correlation between human similarity ratings and AI semantic similarity was moderate ($r=0.441$), explaining only ~19.5% of the variance.

D. Causes of Misalignment

• Training Data Scarcity: A corpus analysis of WikiText-103 and BookCorpus revealed that textile/tactile terms appear 0.0033%–0.0018% of the time, whereas color terms appear 0.15%–0.04%. The lack of tactile data in training corpora is a primary driver of misalignment.
• Linguistic vs. Perceptual Clustering: LLMs cluster textiles based on linguistic/categorical similarity (e.g., "cotton" and "linen" are both natural fibers) rather than perceptual similarity (how they actually feel).

5. Significance and Implications

• Limitations of Text-Only Models: The study demonstrates that current LLMs cannot fully encode the "touch space." They rely on linguistic associations rather than embodied physical understanding.
• Modality-Specific Alignment: Perceptual alignment cannot be assumed to generalize across modalities. A model aligned for vision may remain "blind" to touch.
• Need for Multimodal Grounding: To achieve robust alignment, future AI systems must integrate multimodal sensory data (visual, haptic, auditory) rather than relying solely on text.
• Ethical and Safety Constraints: The authors noted that safety filters in models (e.g., GPT-4) sometimes blocked tactile descriptions due to associations with sexual content, hindering research. This highlights the need for nuanced safeguards that allow legitimate sensory research.
• Future Directions: The paper calls for the creation of large-scale, diverse datasets of tactile descriptions and the development of Multimodal Large Language Models (MLLMs) that can bridge the gap between language and physical sensation.

In conclusion, TouchAI reveals that while LLMs possess some latent knowledge of touch, their perceptual alignment with humans is systematically biased and incomplete. Achieving true human-AI alignment in the physical world requires moving beyond text-based representations to include embodied, multimodal learning.