💻 computer science

Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing

This paper employs diagnostic analyses of semantic alignment, lexical invariance, and syntactic sensitivity to reveal that while large language models achieve strong behavioral performance in metaphor tasks, their underlying mechanisms exhibit semantic drift and contextual biases, suggesting that benchmark success does not necessarily equate to robust, integrated semantic understanding.

Original authors: Fengying Ye, Shanshan Wang, Lidia S. Chao, Derek F. Wong

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fengying Ye, Shanshan Wang, Lidia S. Chao, Derek F. Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of very smart, well-read robots (Large Language Models, or LLMs) that are excellent at spotting and explaining metaphors. If you ask them, "The computer is a turtle," they might confidently say, "It means the computer is slow." They get high scores on tests, so we assume they truly understand the joke.

But this paper asks a tricky question: Are they actually understanding the meaning, or are they just really good at guessing based on patterns?

To find out, the researchers didn't just give the robots a multiple-choice test. Instead, they acted like detectives, poking and prodding the robots in three specific ways to see how their "brains" worked under the hood.

Here is a breakdown of their investigation using simple analogies:

1. The "Target Practice" Test (Semantic Alignment)

The Idea: When a human hears "The computer is a turtle," they might think of "slow speed." But a turtle also has a "long lifespan" or a "hard shell." A human knows to pick the "slow" part because of the context. Did the robot pick the right part, or did it just grab the first turtle fact it knew?

The Experiment:
The researchers set up a "target zone" in a digital map. This zone was defined by two human experts and a literal sentence. They then asked the robots to explain the metaphor and plotted their answers on this map.

The Result: The robots' answers often drifted away from the center of the target zone. It's like an archer who hits the target board but keeps landing on the outer rim instead of the bullseye.
The Takeaway: The robots can produce answers that look correct, but they often miss the specific, intended meaning, focusing on random details instead of the core idea.

2. The "Memory vs. Context" Test (Lexical Invariance)

The Idea: Imagine you see the word "arm." Your brain immediately thinks of "war" or "fighting" because those words often go together. But if I say, "The tree has an arm," you should ignore the war thoughts and think of a branch. Do the robots ignore their "automatic thoughts" when the context changes?

The Experiment:
The researchers asked the robots to swap words in two different scenarios:

With Context: "The council appealed..." (Here, "appealed" is metaphorical).
Without Context: Just the word "appealed" alone.
They checked if the robots suggested the same replacement words in both situations.

The Result: The robots were stubborn. Even when the sentence context changed, they kept suggesting the same words they usually associate with that term. It's like a person who, when asked to describe a "bank," always says "money" even if you are clearly talking about a "river bank."
The Takeaway: The robots rely heavily on fixed, pre-programmed associations (like "arm = war") rather than truly reading the whole sentence to figure out the meaning. This helps them with common metaphors but confuses them with new, tricky ones.

3. The "Scrambled Sentence" Test (Syntactic Influence)

The Idea: Humans understand metaphors by looking at how words fit together in a sentence structure. If you scramble the words, the meaning usually breaks. Do robots need the sentence structure to work, or do they just look for specific "magic words"?

The Experiment:
The researchers took metaphorical sentences and messed them up in three ways:

Random Scramble: Shuffling all words like a deck of cards.
Part-of-Speech Swap: Changing a noun to a verb (e.g., "The council complainant" instead of "The council complained").
Moving the Word: Taking the metaphorical word and moving it to the start or end of the sentence.

The Result: When the sentence structure was broken (especially by swapping word types), the robots' ability to spot the metaphor changed drastically. Some robots actually got better at spotting metaphors when the grammar was weird, suggesting they were reacting to the "weirdness" of the sentence rather than understanding the meaning.
The Takeaway: The robots are very sensitive to surface-level patterns. If a sentence looks "broken" or unusual, they might flag it as a metaphor, not because they understand the metaphor, but because they recognize the pattern of "weird grammar."

The Big Conclusion

The paper concludes that while these robots are great at performing metaphor tasks (getting high scores), they might not be understanding metaphors the way humans do.

Think of it like a parrot that has memorized thousands of phrases. If you ask it a question, it can repeat the perfect answer it heard before. But if you change the context slightly or scramble the words, the parrot might start making things up or sticking to its old habits.

In short: The robots are using a mix of "memory tricks" (remembering word pairs) and "pattern spotting" (noticing weird grammar) to pass the test. They aren't necessarily building a deep, flexible understanding of what the metaphor means. The authors warn us not to be fooled by high test scores; just because a robot gets the right answer doesn't mean it truly "gets" the joke.

Technical Summary: Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing

Problem Statement

While Large Language Models (LLMs) demonstrate strong behavioral performance on metaphor detection and interpretation tasks, it remains unclear whether this success reflects deep, integrated semantic understanding or reliance on superficial heuristics. Existing linguistic theories, such as Selection Preference Violation (SPV) and Conceptual Metaphor Theory (CMT), emphasize that metaphor processing involves identifying implicit cross-domain mappings rather than simple lexical substitutions. However, prior computational work has largely focused on prediction outcomes (e.g., accuracy scores) rather than the underlying mechanisms driving these behaviors. This paper addresses the gap by investigating whether observed LLM performance stems from a unified semantic mechanism or a heterogeneous combination of lexical associations, trigger word effects, and syntactic cues.

Methodology

The authors propose a diagnostic framework comprising three complementary probing dimensions to analyze LLM behavior under controlled conditions. The experiments utilize the Fig-QA dataset for interpretation analysis and the Metaphor Understanding Challenge Dataset (MUNCH) for generation and detection tasks.

1. Spatial Analysis (Semantic Attribute Alignment)

To assess whether model-generated interpretations align with intended semantic attributes, the authors employ a geometric probing framework.

Reference Plane Construction: For a target metaphor $m_i$ , a reference plane $\gamma_i$ is constructed in a shared embedding space using three points: two human-annotated interpretations ( $R_i, R'_i$ ) capturing the shared semantic attribute, and a model-generated literal paraphrase ( $S_i$ ) of the target sentence.
Geometric Measures: The model's generated interpretation ( $M_i$ $M_{i}$ ) is evaluated against this plane using two metrics:
- $d_p$ (Perpendicular Distance): Measures the magnitude of geometric deviation of $M_i$ from the reference plane.
- $\cos \theta$ (Cosine Similarity): Measures the orientation difference between the interpretation plane (spanned by $R_i, R'_i, M_i$ ) and the reference plane.
Validation: These geometric signals are correlated with a similarity-based sum score ( $A_d$ ) and validated via human evaluation to ensure they reflect meaningful semantic deviations rather than artifacts.

2. Metaphorical Imagination (Lexical Invariance)

This dimension probes the stability of lexical associations across contextual conditions, investigating "trigger word effects" where models may rely on stable lexical anchors rather than context.

Experimental Design: The study compares Contextualized Generation (producing words within a sentence) against Decontextualized Generation (producing words in isolation).
Tasks:
- Literal-to-Metaphor (LM): Generating metaphorical counterparts for literal inputs.
- Metaphor-to-Literal (ML): Generating literal counterparts for metaphorical inputs.
Metric: An Anchor Score quantifies the overlap between contextualized and decontextualized outputs. High scores indicate the persistence of context-invariant lexical associations.

3. Syntactic Shuffle (Syntactic Influence)

This dimension examines the sensitivity of metaphor detection to structural cues by perturbing sentence syntax while largely preserving lexical content.

Perturbations:
- Random Shuffle: Disrupts both syntax and semantic coherence.
- POS Shuffle: Replaces the metaphorical word with a synonym of a different Part-of-Speech (POS), introducing syntactic irregularity while maintaining lexical meaning.
- Metaphorical Word Reposition: Moves the metaphorical word to the beginning, middle, or end of the sentence.
Goal: To determine if detection relies on integrated sentence structure or shallow heuristic cues like POS anomalies.

Key Contributions

Geometric Probing Framework: The authors introduce a novel behavioral proxy using geometric measures ( $d_p$ and $\cos \theta$ ) to quantify semantic attribute alignment in metaphor interpretation, offering a finer-grained view than discrete accuracy metrics.
Lexical Invariance Analysis: Through Metaphorical Imagination tasks, the study demonstrates that LLMs exhibit persistent, context-invariant lexical associations, suggesting a reliance on stable lexical anchors that may support conventional metaphors but bias novel ones.
Syntactic Sensitivity Assessment: The paper analyzes the impact of controlled syntactic perturbations on detection, revealing that models are highly sensitive to syntactic irregularities (e.g., POS shuffles) which can artificially inflate detection accuracy.
Diagnostic Perspective: The work shifts the focus from performance comparison to a diagnostic analysis of how models process metaphors, identifying consistent behavioral patterns and limitations across diverse LLMs.

Results & Analysis

Semantic Drift: Spatial analysis reveals that model-generated interpretations consistently exhibit semantic drift relative to reference attributes. While geometric measures correlate with human judgments (low $d_p$ corresponds to higher human alignment scores), models often fail to capture the intended mapping, focusing instead on salient but irrelevant attributes (e.g., interpreting "The computer is a turtle" as having a "long lifespan" rather than "low speed").
Lexical Anchoring: Anchor Scores are consistently high (65%–80%) across models and genres, indicating that lexical associations often persist regardless of context. This effect is observed even in novel metaphors (novelty score > 0.3), though less uniformly. The authors interpret this as evidence that models may rely on lexical priors that can bias interpretation in contexts where integration is required.
Syntactic Sensitivity: Metaphor detection accuracy varies significantly with syntactic perturbations. Notably, models often achieve higher accuracy under POS shuffle conditions compared to original sentences, suggesting that syntactic irregularity itself serves as a heuristic cue for detection rather than evidence of robust semantic integration. Random shuffling acts as a stress test, revealing unstable behavior in some models (e.g., DeepSeek-V3) while others (e.g., LLaMA-3.1-8B) remain near chance levels.

Significance and Claims

The paper concludes that strong behavioral performance on metaphor benchmarks does not necessarily imply robust, integrated semantic understanding. Instead, LLM success appears to arise from a heterogeneous combination of:

Semantic Drift: A tendency to deviate from intended attribute mappings.
Lexical Priors: Reliance on stable, context-invariant lexical associations that may support conventional metaphors but hinder the processing of novel ones.
Syntactic Heuristics: Sensitivity to surface structural disruptions (like POS anomalies) that function as detection cues independent of deep semantic composition.

The authors emphasize the need for caution when interpreting benchmark results as evidence of genuine metaphor understanding. They argue for evaluation approaches that probe attribute-level alignment, contextual reasoning, and syntactic integration beyond pattern-based cues. The study is modest in its scope, acknowledging limitations such as the reliance on constructed reference regions, the focus on single-word metaphors, and the use of English datasets, noting that generalizability to other languages and discourse-level phenomena requires further investigation.