Beyond Memorization: Assessing Semantic Generalization in Large Language Models Using Phrasal Constructions

Technical Summary: Beyond Memorization: Assessing Semantic Generalization in Large Language Models Using Phrasal Constructions

Problem Statement
The pretraining of Large Language Models (LLMs) on web-scale data presents a critical evaluation challenge: distinguishing between genuine linguistic competence and the mere memorization of frequent patterns. While models often perform well on tasks involving data well-represented in their training sets, their ability to generalize to out-of-domain, creative, or less common language instances remains unclear. Specifically, it is difficult to determine if models truly understand the abstract semantics of syntactic structures or if they rely on statistical correlations derived from high-frequency lexical items. Existing evaluation methods often struggle to ensure test independence from pretraining data or fail to isolate specific linguistic phenomena like constructional semantics.

Methodology
To address these challenges, the authors construct a diagnostic evaluation framework grounded in Construction Grammar (CxG). CxG posits that language is learned through usage, where speakers abstract over syntactic slots to interpret and produce novel instantiations of constructions, even with unfamiliar lexical items. The authors leverage this framework to design two experiments using Natural Language Inference (NLI) as a proxy for understanding:

1. Experiment 1 (CxNLI): This experiment tests the model's ability to generalize constructional semantics to creative, infrequent instantiations of common phrasal constructions. The dataset consists of 435 NLI triples derived from 8 high-frequency English constructions (e.g., Resultative, Caused-Motion, Let-Alone). Premises utilize "entrenched" constructional templates but are instantiated with atypical verbs or lexical items unlikely to appear in that specific structure within pretraining data. The task requires the model to determine if a hypothesis (derived from the construction's abstract meaning) entails, contradicts, or is neutral to the premise.
2. Experiment 2 (CxNLI-Distinction): This experiment tests the model's ability to distinguish between syntactically identical constructions that possess divergent semantic meanings. The dataset contains 99 NLI triples featuring constructions (e.g., Depictive vs. Resultative) that share the same surface syntax (e.g., NP V NP ADJ) but differ in meaning (e.g., state resulting from an action vs. state existing during an action). The goal is to assess whether models can avoid over-generalizing the semantics of highly entrenched constructions to less entrenched, syntactically identical distractors.

The evaluation involves testing state-of-the-art models, including GPT-4o, GPT-o1, GPT-3.5, and Llama 3 (8B and 70B), across zero-shot and in-context learning (ICL) settings. The ICL settings utilize examples from the authors' own dataset (CxNLI) and the standard Stanford NLI (SNLI) dataset to probe the models' reliance on specific constructional knowledge versus general NLI capabilities.

Key Contributions

1. Novel Diagnostic Dataset: The authors introduce a manually validated dataset of 534 NLI triples specifically designed to test constructional generalization and distinction, moving beyond metalinguistic identification tasks to evaluate underlying semantic understanding.
2. CxG-Based Evaluation Framework: Unlike prior works that focus on single constructions or metalinguistic tasks, this study utilizes the well-established NLI task to probe the functional understanding of constructional meaning, testing both creative instantiation and syntactic ambiguity.
3. Broad Model Evaluation: The study evaluates a range of models, including the largest currently available (GPT-o1, Llama 3 70B), providing a comprehensive view of the current state of constructional generalization in LLMs.
4. Open Resources: The dataset, prompts, and model responses are made publicly available to facilitate further research into LLM linguistic generalization.

Results
The experiments reveal a significant discrepancy between human and model performance, particularly in the ability to distinguish syntactically identical constructions:

• Experiment 1 (Generalization): Models, particularly GPT-4o and GPT-o1, demonstrated high accuracy (up to 90%) in generalizing constructional semantics to creative instantiations. Performance improved with in-context learning using construction-specific examples, suggesting that models can learn the abstract mapping when provided with relevant cues.
• Experiment 2 (Distinction): A sharp performance drop occurred when models were tasked with distinguishing syntactically identical constructions. Even the most advanced models (GPT-o1, GPT-4o) saw their accuracy fall by over 40% compared to Experiment 1. For instance, GPT-o1 dropped from 90% accuracy in Exp. 1 to 46% in Exp. 2.
• Statistical Significance: Bayesian A/B tests confirmed that the performance drop in Experiment 2 is statistically significant and far more pronounced for LLMs than for humans. While human performance remained robust (IAA of 83%), models struggled to reject the semantics of highly entrenched constructions when presented with less entrenched, syntactically identical distractors.
• Error Analysis: Models frequently failed to distinguish between constructions like the Resultative (action causes state) and the Depictive (state exists during action), often misinterpreting the causal relationship. This suggests a failure in causal reasoning and an over-reliance on the most frequent lexical-semantic associations.

Significance and Claims
The authors claim that their findings highlight a fundamental limitation in current LLM architectures: while models can process constructional semantics to a degree, they fail to generalize this understanding to novel instantiations when syntactic ambiguity is introduced. The results challenge the "more data is better" paradigm, suggesting that scaling web-text data may reinforce biases toward frequent constructional meanings rather than fostering true abstract reasoning.

The paper argues that the inability to distinguish between syntactically identical constructions represents a failure of causal reasoning with potential safety implications, as misunderstanding the difference between a co-occurring state and a caused outcome could lead to catastrophic errors in AI agents. Furthermore, the study emphasizes the need for diagnostic, contrastive benchmarks to accurately assess linguistic competence, as broad-coverage benchmarks may overestimate a model's true capabilities. The authors conclude that while current models show robustness in handling entrenched constructions, their ability to generalize constructional meaning to distinct, syntactically identical forms lags substantially behind human linguistic knowledge.