💻 computer science

Can LLMs interpret figurative language as humans do?: surface-level vs representational similarity

While large language models align with humans on the surface-level interpretation of dialogue, they significantly diverge at the representational level, particularly when processing context-dependent and socio-pragmatic figurative language such as idioms, slang, and sarcasm, with GPT-4 demonstrating the closest approximation to human patterns among the tested models.

Original authors: Samhita Bollepally, Aurora Sloman-Moll, Takashi Yamauchi

Published 2026-01-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Samhita Bollepally, Aurora Sloman-Moll, Takashi Yamauchi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand human jokes, sarcasm, and slang. You show it a sentence like, "This food is gas," and ask, "Is this funny?" or "Is this positive?"

A new study by researchers at Texas A&M University asks a tricky question: When the robot gives the same answer as a human, does it actually think the same way a human does?

Here is the breakdown of their findings, using some everyday analogies.

The Experiment: The "Taste Test"

The researchers set up a massive taste test, but instead of ice cream, they used 240 different sentences. These sentences covered six "flavors" of language:

Conventional: Normal sentences (e.g., "The dog is outside").
Idiomatic: Phrases where the words don't mean what they say (e.g., "Bite the bullet").
Emotional: Sentences full of feelings.
Funny: Jokes and puns.
Sarcastic: Saying the opposite of what you mean.
Gen Z Slang: Modern internet speak (e.g., "This food is gas").

They asked 211 humans and four different AI models (including GPT-4, Llama, and Mistral) to rate every sentence on 40 different questions, like "Is this positive?" or "Is this concerning?" on a scale of 1 to 10.

The Two Ways to Compare: The "Score" vs. The "Map"

The researchers looked at the results in two very different ways:

1. Surface-Level Similarity (The Scoreboard)
This is like looking at the final score of a soccer game. If the AI says "8/10" for a joke and a human says "8/10," they look like they agree.

The Finding: On this level, the AI models, especially the smartest one (GPT-4), looked very human-like. They gave similar scores to humans for most sentences. It looked like they were on the same team.

2. Representational Similarity (The Map)
This is the deeper part. Imagine you and a friend are both drawing a map of a city.

Human Map: You know that the "Pizza Place" is right next to the "Park," and the "School" is far away from the "Cemetery." Your map has a specific structure based on how you understand the world.
AI Map: The AI might also put the "Pizza Place" and "Park" close together, but for the wrong reasons. Maybe it thinks they are close because they both have the word "place" in them, not because of how they function in real life.

The researchers used a mathematical tool called Representational Similarity Analysis (RSA) to compare the structure of these maps. They didn't just check if the scores matched; they checked if the relationships between the sentences were the same.

The Big Reveal: The "Imposter" Effect

Here is what they found:

The Surface is Deceptive: The AI models were great at mimicking human scores. If you asked, "Is this sarcastic?", the AI often said "Yes," just like a human would.
The Map is Different: When the researchers looked at the underlying "maps," the AI and the humans were actually quite far apart. The AI organized the meaning of sentences differently than humans do.

The Analogy:
Think of the AI as a very talented parrot.

If you ask the parrot, "Is this sentence funny?" it can say "Yes" perfectly, just like a human.
But the parrot doesn't actually get the joke. It just knows that when humans hear this specific pattern of words, they usually say "Yes."
The human, however, understands the joke because they understand the social context, the tone, and the hidden meaning.

The Results by "Flavor"

The study found that the AI's "parrot skills" worked better for some types of language than others:

Emotional & Conventional Sentences: The AI's map was somewhat similar to the human map. It's easier to guess that "I am sad" is negative than to guess that "I'm so excited I wet my plants" is funny.
Idioms, Sarcasm, and Slang: This is where the AI failed the "Map Test."
- Idioms: When humans hear "Break a leg," they know it means "Good luck." The AI often struggled to map this correctly to the concept of "good luck" without getting confused by the literal words "break" and "leg."
- Sarcasm & Slang: These rely heavily on social context and shared human experiences (like Gen Z slang). The AI's internal map for these was very different from the human map. It was like the AI was speaking a different language, even though the words looked the same.

The "Big Brain" vs. The "Small Brain"

The study compared different AI models:

GPT-4: This was the "big brain." It had the most similar map to humans, but it still wasn't a perfect match. It was the closest thing to a human, but it still had gaps.
Smaller Models (Llama, Mistral): These were further away. Their maps were very different from humans, especially when it came to tricky stuff like sarcasm and slang.

The Bottom Line

The paper concludes that just because an AI gives the same answer as a human, it doesn't mean it understands the language the same way.

The AI is excellent at pattern matching (mimicking the scoreboard), but it lacks the deep, social, and contextual "grounding" that humans have. It organizes meaning in its own unique way, which works well for simple tasks but falls apart when the language gets tricky, emotional, or culturally specific.

In short: The AI is a very good actor that can say the right lines, but it doesn't necessarily feel the emotions behind them.

Technical Summary: Can LLMs Interpret Figurative Language as Humans Do?

Problem Statement

While Large Language Models (LLMs) generate text that resembles human output, the extent to which they align with human judgments in interpreting figurative and socially grounded language remains uncertain. Existing literature suggests LLMs may match humans on surface-level judgments (e.g., grammatical precision or basic sentiment) but diverge in deeper interpretive processes involving pragmatics, context, and subjective experience. Specifically, it is unclear whether LLMs possess the cognitive grounding necessary to organize meaning in a shared semantic space with humans, particularly for complex linguistic traits such as idioms, sarcasm, humor, emotional tone, and Gen Z slang. Current research often relies on aggregate agreement (surface-level similarity), which may mask failures to capture the fine-grained distinctions required for nuanced pragmatic phenomena.

Methodology

To address this, the authors conducted a study comparing human participants and four instruction-tuned LLMs (GPT-4, Gemma-2-9B, Llama-3.2, and Mistral-7B) using Representational Similarity Analysis (RSA).

Data and Stimuli

Stimuli: A dataset of 240 dialogue-based sentences was compiled, divided into 120 "sensible" (containing a specific linguistic trait) and 120 "non-sensible" (structurally similar but omitting the key trait) sentences.
Categories: Sentences were classified into six linguistic traits: Conventional, Idiomatic, Emotional, Funny, Sarcastic, and Gen Z Slang.
Evaluation: Each sentence was paired with 40 interpretive questions (e.g., "Is this sarcastic?", "Is this relevant to you?") designed to assess meaning, tone, and expressive quality.
Scale: Both humans and models rated the sentences on a 10-point Likert scale, resulting in 9,600 sentence–question pairs.

Participants

Humans: 211 undergraduate participants from Texas A&M University.
Models: Four decoder-only, instruction-tuned LLMs accessed via API or Hugging Face.
Conditions: Two studies were conducted. Study 1 used a standard zero-shot prompt. Study 2 introduced refined constraints (strict 1–10 rating bounds and single-sentence justifications) to assess prompt sensitivity.

Analytical Framework

The study distinguished between two levels of alignment:

Surface-Level Similarity (SLS): Computed via Pearson correlation coefficients between human and model ratings for specific sentence categories. This measures output-level agreement.
Representational Similarity (RSA): This method examines the internal structure of interpretation.
- Pairwise Euclidean distances were calculated between all sentence pairs based on their 40-dimensional rating vectors, creating a Representational Dissimilarity Matrix (RDM) for both humans and each model.
- RSA quantified alignment by correlating the upper-triangular entries of the human RDM with the model RDMs.
- Significance: High RSA correlation indicates that humans and models organize sentences in structurally similar ways within their semantic spaces, even if absolute rating values differ. Low RSA correlation suggests divergent internal representational geometries.

Key Results

Surface-Level Similarity (SLS)

General Alignment: Models, particularly GPT-4, showed strong surface-level alignment with humans (correlations $r \approx 0.60–0.90$ for sensible sentences).
Model Hierarchy: GPT-4 consistently outperformed other models. Gemma and Mistral showed moderate alignment, while Llama-3.2 exhibited the weakest alignment, especially for figurative categories.
Category Differences: Conventional and emotional sentences yielded higher correlations than humor, sarcasm, and slang.
Prompt Sensitivity: Study 2 (refined prompts) showed modest improvements in SLS across models, but non-sensible sentences remained challenging.

Representational Similarity (RSA)

Divergence: Despite high surface-level agreement, RSA revealed systematic divergence between model and human representational spaces.
Figurative Language: Alignment was significantly weaker for pragmatically rich categories (idioms, sarcasm, Gen Z slang) compared to conventional or emotional sentences.
- GPT-4 showed the strongest representational alignment ( $r \approx 0.50–0.85$ for sensible sentences) but still fell short of human–human consistency ( $r \approx 0.75–0.85$ ).
- Smaller models (Llama, Mistral) showed very low representational alignment ( $r \approx 0.10–0.35$ ) for figurative categories.
Non-Sensible Sentences: Models struggled to distinguish sensible from non-sensible variants in their internal representations, with correlations dropping significantly compared to human baselines.
Prompt Impact: While refined prompts improved surface-level ratings, they did not substantially restructure the internal semantic organization of the models, particularly for Llama.

Key Contributions

Methodological Distinction: The study explicitly separates surface-level behavioral agreement from representational structural alignment, demonstrating that high correlation in ratings does not guarantee shared internal semantic organization.
Empirical Evidence of Divergence: It provides evidence that LLMs, even state-of-the-art models like GPT-4, fail to fully replicate human-like representational geometries for figurative language, particularly for context-dependent traits like sarcasm and slang.
Model Hierarchy: It establishes a consistent hierarchy where GPT-4 approximates human representational patterns more closely than smaller models, yet no model reaches human–human reliability.
Insight into Mechanisms: The results suggest that current training objectives (next-token prediction) are insufficient to constrain the internal organization of meaning for pragmatic inference, leading models to rely on surface heuristics rather than stable semantic grounding.

Significance and Claims

The paper claims that while LLMs can approximate human judgments at a surface level, they do not yet possess the representational structure required for genuine human-like interpretation of figurative and socially grounded language.

Limitations of Current Models: The findings support the view that LLMs rely on distributional pattern matching rather than stable semantic grounding. The inability to align on representational geometry for idioms and slang suggests a lack of context-sensitive supervision and social grounding in current architectures.
Implications for Evaluation: The authors argue that evaluating LLMs solely on output accuracy or surface-level correlations is insufficient. Future frameworks must probe how meaning is organized internally (via methods like RSA) rather than just what output is generated.
Modest Scope: The authors acknowledge limitations, including the relatively small dataset size, the lack of rich contextual cues in the stimuli (which are crucial for sarcasm and humor), and the assumption that humans and LLMs share comparable internal frameworks. Consequently, the claims are delimited to context-independent pragmatic interpretation, suggesting that the observed gaps may be exacerbated by the absence of real-world discourse context.

In conclusion, the study posits that current LLMs encode figurative language in representational spaces that are only partially aligned with human semantic organization, highlighting a fundamental gap between generating human-like text and possessing human-like cognitive grounding for pragmatic language.