💬 NLP

Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions

This paper demonstrates that modestly sized open-source language models can effectively grasp the semantics of rare paired-focus constructions, with this understanding emerging later in training than syntactic knowledge and correlating with gains in specific domains of world knowledge.

Original authors: Wesley Scivetti, Ethan Wilcox, Nathan Schneider, Kanishka Misra, Leonie Weissweiler

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wesley Scivetti, Ethan Wilcox, Nathan Schneider, Kanishka Misra, Leonie Weissweiler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine language as a giant, complex recipe book. Most of the recipes are standard: "Mix flour and water." But then there are the rare, fancy, specific instructions like "Let alone the fancy garnish, you can't even handle the basic dough." In linguistics, these are called PAIRED-FOCUS constructions (phrases like "let alone," "much less," "not to mention," and "never mind").

This paper asks a simple question: Do computer language models (AI) actually understand these fancy recipes, or are they just memorizing the words?

Here is the breakdown of what the researchers found, using some everyday analogies.

1. The "Form vs. Meaning" Test

Think of a construction like a specific handshake.

The Form (Syntax): Knowing how to do the handshake (the specific hand movement, the order of fingers).
The Meaning (Semantics): Knowing what the handshake implies (e.g., "If I can't do the easy thing, I definitely can't do the hard thing").

The researchers built a new test to see if AI models could do both. They didn't just ask, "Is this sentence grammatically correct?" They asked, "Does the AI understand the logic inside the sentence?"

The Analogy: Imagine a robot that knows exactly how to say, "I can't lift a pebble, let alone a boulder."

The Syntax Test: Can the robot say the sentence without stumbling?
The Meaning Test: If you ask the robot, "So, can you lift the boulder?" does it correctly say "No"? Or does it get confused because it only memorized the words but didn't understand the logic?

2. The Results: Size Matters (But Not Just a Little)

The researchers tested 36 different AI models, ranging from tiny ones (like a pocket calculator) to massive ones (like a supercomputer).

The Tiny Models (Under 400 Million parameters): These are like toddlers. They are great at the handshake. They know the words "let alone" go together and can say the sentence perfectly. But when it comes to the meaning, they fail. They don't understand the logic; they are just guessing.
The Medium Models (Around 400 Million to 1 Billion parameters): These are like smart teenagers. They start to get it. They can do the handshake and they start to understand the logic behind it.
The Big Models: These are like adults. They are very good at both the form and the meaning.

The Key Finding: You don't need a "god-tier" AI to understand these rare phrases. A "modest" AI (one that isn't the biggest or most expensive) can actually grasp the meaning, provided it's big enough.

3. The Learning Timeline: Learning the Dance Steps Before the Music

The researchers watched how these models learned over time, like watching a student practice for a dance competition.

Step 1: The Steps (Syntax) come first. The models learned the correct word order and grammar rules very early in their training. They knew how to say the phrase quickly.
Step 2: The Music (Meaning) comes later. It took much longer for the models to understand why the phrase works. They had to learn a lot more about the world before they could connect the dots.

The Analogy: Imagine learning to drive. First, you learn where the pedals are and how to turn the wheel (Syntax). You can do this quickly. But understanding when to brake, how to judge the speed of oncoming traffic, and the "feel" of the road (Semantics) takes much more experience and time. The AI learned the "pedals" long before it learned the "feel."

4. The "World Knowledge" Connection

The study found a strong link between understanding these phrases and knowing how the real world works.

The Scenario: The AI was tested on sentences like, "I couldn't lift a tiny rock, let alone a huge one."
The Result: The AI understood this because it knows rocks have weight.
The Twist: When the researchers tried to trick the AI with sentences that broke the laws of physics (e.g., "I couldn't lift a huge rock, let alone a tiny one"), the smaller AIs got confused. They relied too much on their "common sense" about the world rather than the strict logic of the phrase. The bigger, smarter AIs, however, could ignore the weird world facts and stick to the logic of the phrase.

5. The "Human-Scale" Problem

The researchers also tested models trained on "human-scale" data (amounts of text a human could actually read in a lifetime).

The Verdict: These models failed completely at understanding the meaning. They could say the words, but they didn't get the logic.
The Takeaway: To truly understand the deep logic of rare language phrases, an AI needs to "read" way more than a human ever could. It needs that massive exposure to see the patterns.

Summary

This paper is like a report card for AI on a very specific, tricky subject.

Can small AIs do it? Yes, but only the "medium-sized" ones. The tiny ones just memorize the words.
Do they learn it all at once? No. They learn the grammar (the form) first, and the meaning (the logic) much later.
Do they need to know the world? Yes. Understanding these phrases is tied to understanding how the world works (like knowing that big things are heavier than small things).

The researchers conclude that while AI is getting better at these rare phrases, it still learns the "shape" of the language before it learns the "soul" of the language.

Title: Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of PAIRED-FOCUS Constructions

Problem Statement
While large language models (LLMs) have demonstrated success on many linguistic tasks, their ability to grasp the semantics of rare linguistic constructions—specifically form-meaning pairings central to Construction Grammar—remains an open question. Previous research indicates that while models often learn the formal properties (syntax) of rare constructions, they frequently fail to acquire their nuanced semantic meanings, particularly when trained on "human-scale" data (e.g., BabyLMs). Furthermore, existing evaluation datasets for constructional semantics often rely on arbitrary contrasts or unnatural test items that may not effectively isolate scalar semantics or world knowledge interactions. It is unclear whether open-source models of modest size can robustly understand these constructions, and if so, how this learning unfolds relative to syntactic acquisition and other domains of knowledge.

Methodology
The authors introduce a novel dataset and evaluation framework to test the understanding of four specific PAIRED-FOCUS constructions in English: let alone, much less, not to mention, and never mind. These constructions conjoin two focused elements, implying a scalar relationship where the second element represents a higher value on a scale than the first.

Dataset Construction: Unlike prior work using arbitrary examples, the authors generated 3.5k sentence pairs per construction using scalar adjectives and common nouns derived from established scales (Wilkinson and Tim, 2016). This ensures a coherent scalar relationship grounded in general world knowledge.
Semantic Evaluation: The evaluation measures a model's sensitivity to the construction's meaning by comparing the probability of a follow-up sentence that is entailed by the construction's scalar logic versus one that contradicts it. Crucially, the authors control for world knowledge plausibility by comparing these results against a control condition using the simple conjunction "or." The metric ( $\Delta P$ ) isolates the probability shift attributable specifically to the construction's semantics.
Syntactic Evaluation: The authors adapted existing syntactic tests (e.g., movement constraints, negative polarity item behavior) to determine if models recognize the ungrammaticality of certain manipulations specific to PAIRED-FOCUS constructions.
Model Scope: The study evaluates 36 models varying in parameter count (from 68M to 13B), architecture (autoregressive vs. masked language models), and pretraining data size.
Learning Dynamics: Using intermediate checkpoints of three high-performing models (Pythia-12b, Ettin-encoder-400m, Ettin-decoder-1b), the authors track the acquisition of form and meaning over training steps, comparing these trajectories against benchmarks for syntax (BLiMP), conceptual knowledge (COMPS), and world knowledge (EWoK).

Key Results

Model Size and Data: Models with fewer than ~400M parameters generally perform at or below chance on semantic evaluations, regardless of training data size. In contrast, several modestly sized open-source models (e.g., Ettin-400m, Pythia-12b) achieve high accuracy (>90%) on semantic tasks. However, models trained on "human-scale" data (BabyLMs) fail to demonstrate above-chance semantic understanding.
Syntax vs. Semantics Dissociation: There is a clear dissociation between syntactic and semantic learning. Even the smallest models tested demonstrate robust knowledge of the form (syntax) of PAIRED-FOCUS constructions, often achieving near-perfect scores. Semantic understanding, however, emerges significantly later in training.
Learning Trajectories:
- Syntactic knowledge plateaus early in pretraining, mirroring the learning curve of the BLiMP benchmark.
- Semantic understanding follows a later, more gradual trajectory, closely mirroring the learning curve of the EWoK (world knowledge) benchmark.
- Performance on PAIRED-FOCUS semantics is moderately correlated with performance on the physical relations domain of EWoK, suggesting a link between constructional meaning and general world knowledge.
Plausibility Sensitivity: Larger models (e.g., Pythia-12b) show robustness to implausible contexts, correctly inferring the scalar relationship even when it conflicts with world knowledge. Smaller models are more reliant on world knowledge cues and struggle when the construction's scalar implication contradicts typical plausibility.
Construction Correlation: The four PAIRED-FOCUS constructions tested are highly correlated in performance, though let alone (the most frequent) is learned slightly earlier than the others.

Significance and Claims
The paper claims that relatively modest-sized open-source models (under 400M parameters) are capable of acquiring nontrivial semantic knowledge of rare constructions, challenging the notion that only frontier-scale models possess such capabilities. However, the study highlights a critical limitation: models trained on human-scale data fail to acquire this semantic knowledge, casting doubt on their utility as "model systems" for constructionist theories that posit the joint acquisition of form and meaning.

The authors argue that the observed dissociation—where syntax is learned early and robustly, but semantics emerges later and correlates with world knowledge—suggests that text-only language models do not naturally model the joint acquisition of form and meaning as posited by Construction Grammar. Instead, semantic understanding appears to be a downstream effect of accumulating world knowledge and scalar reasoning capabilities. The paper concludes that while LMs can learn these constructions, the learning dynamics differ significantly from human acquisition, and future work should explore multimodal or situated learning environments to better model the integration of form and meaning.