CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models

1. Problem Statement

While recent research has increasingly analyzed Large Language Models (LLMs) from a linguistic perspective, existing benchmarks primarily focus on grammatical acceptability (distinguishing grammatically correct sentences from incorrect ones based on form-level constraints). These benchmarks often treat syntax and semantics as independent, failing to evaluate a model's ability to interpret the meanings conveyed by specific grammatical forms.

The authors argue that true language understanding requires grasping form–meaning pairings (constructions), where the syntactic structure itself encodes specific semantic relations (e.g., causality, directionality, or transfer of possession). Current evaluation methods, such as Natural Language Inference (NLI), often rely on single-sentence pairs with implicit labels, making it difficult to isolate specific linguistic cues or compare models of varying sizes (from small, developmentally plausible models to massive LLMs) systematically.

2. Methodology: The CxMP Benchmark

The authors introduce CxMP (Constructional Minimal-Pair), a benchmark grounded in Construction Grammar. It evaluates whether models can map specific syntactic constructions to their implied semantic relations.

Key Design Features

• Constructional Focus: The benchmark covers nine distinct construction types ranging from partially fixed patterns to fully schematic ones:
  1. Let-alone (Scalar comparison/Negation)
  2. Causative-with (Supply/Recipient relation)
  3. Way-manner (Self-propelled movement)
  4. Comparative-correlative (Dependency between clauses)
  5. Conative (Attempted action vs. success)
  6. Ditransitive (Transfer of possession)
  7. Caused-motion (Agent causing theme movement)
  8. Resultative (Action causing state change)
  9. Intransitive-motion (Autonomous movement)
• Minimal-Pair Design: Each evaluation item consists of a Constructional Example (a sentence containing the target construction) and a Diagnostic Sentence (probing the meaning).
  • For each construction, there are Plausible and Implausible versions of the diagnostic sentence.
  • Controlled Variants: The dataset includes "switched" versions where entity roles (Subject/Object) are swapped to test if models rely on shallow heuristics (e.g., always picking the first noun) rather than understanding the construction.
  • Lexical Control: The dataset uses four types of entities (Common nouns, Male/Female names, Name+Alphabet) to mitigate lexical biases.
• Data Generation:
  • Generated automatically using gpt-5-chat-latest based on predefined templates.
  • Filtered for semantic coherence and manually inspected.
  • Human Validation: Conducted via Prolific with native speakers, achieving 96.65% majority-vote accuracy, confirming the reliability of the generated items.
  • Total Size: 43,000 Evaluation Pairs.

Experimental Setup

• Models Evaluated: A diverse range of models including:
  • Developmentally plausible small models: BabyLM (10M/100M tokens), BabyBERTa.
  • Mid-sized models: Pythia suite (14M to 12B), Open-sci-ref (1.3B/1.7B).
  • Large Open LLMs: Llama-3.1 (8B/70B, base and instruction-tuned).
  • Closed-source Upper Bound: GPT-5.
• Evaluation Metric: Models predict the probability of the Plausible vs. Implausible diagnostic sentence. Metrics include length-normalized log-probability (CLM) and pseudo log-likelihood (MLM).
• Comparative Analysis: The study compares CxMP performance against BLiMP (a grammatical acceptability benchmark) to trace learning trajectories.

3. Key Results

A. Performance Gap: Syntax vs. Semantics

• Small Models: Perform at near-chance levels on CxMP, whereas they often perform well on formal grammatical benchmarks like BLiMP. This indicates that constructional understanding is significantly harder than syntactic acceptability.
• Large Models: Even 70B parameter open models struggle with specific constructions (e.g., Let-alone, Comparative-correlative). Only the closed-source GPT-5 approaches high accuracy across most constructions.
• Learning Trajectories:
  • BLiMP (Syntax): Models reach 80% accuracy early in training (50B tokens) and plateau.
  • CxMP (Semantics): Performance improves gradually throughout training, suggesting that mastering the semantic implications of form requires longer-term learning and more data.

B. Factors Influencing Performance

• Model Size: Performance generally scales with model size, particularly for complex, non-standard constructions.
• Data Scale: Increasing training data from 300B to 1T tokens yields significant improvements, whereas gains from 50B to 300B are limited.
• Instruction Tuning: Instruction-tuned models generally outperform base models on argument-structure constructions but show decreased performance on non-standard constructions (e.g., Let-alone), suggesting tuning does not uniformly enhance constructional understanding.
• Training Objectives: Masked Language Models (MLMs like RoBERTa) generally outperform Causal Language Models (CLMs like GPT-2) of similar size, likely due to their ability to leverage bidirectional context to integrate form and meaning.

C. Bias and Heuristics

• Shallow Heuristics: Mid-sized models exhibit large score gaps between "switched" and "non-switched" sentences, indicating reliance on surface cues (e.g., selecting the subject noun regardless of the construction's logic).
• Bias Evolution: Bias is low in early training (random guessing), peaks in mid-training (reliance on heuristics), and declines in later stages as models converge toward accurate judgments.
• Lexical Bias: Common nouns trigger stronger biases than controlled name variants, as models rely on co-occurrence statistics (e.g., "teacher" + "teach").

D. Pseudoword Generalization

• When content words are replaced with pseudowords (e.g., "The rom gorped the blickit"), model performance drops significantly compared to real words.
• This indicates that current LLMs rely heavily on lexical semantics rather than purely abstract constructional forms to infer meaning, unlike humans who can interpret meaning from construction alone.

4. Key Contributions

1. New Benchmark (CxMP): A rigorous, controlled benchmark for evaluating constructional understanding (form-meaning mapping) rather than just grammaticality.
2. Systematic Comparison: Enables fair comparison across a wide spectrum of models (from small developmental models to massive LLMs) and construction types.
3. Insight into Learning Trajectories: Reveals that while syntactic competence emerges early, constructional semantic understanding develops gradually and lags behind, even in state-of-the-art models.
4. Identification of Limitations: Demonstrates that current LLMs struggle with abstract constructional meaning without lexical support and are prone to shallow heuristics when data is insufficient.

5. Significance

The paper highlights a critical gap in current LLM evaluation: high grammatical accuracy does not equate to deep linguistic understanding. By shifting focus from "is this sentence grammatical?" to "does the model understand what this grammatical structure means?", CxMP provides a framework to study how models learn the interplay between syntax and semantics.

The findings suggest that:

• Current models have not fully mastered the abstract rules of Construction Grammar.
• Instruction tuning and scaling alone are insufficient to solve these semantic gaps.
• Future models need to be evaluated on their ability to generalize meaning from form, not just from lexical co-occurrence, to achieve human-like language acquisition.

This work serves as a foundational step for developing models that truly understand the semantic contributions of linguistic forms, moving beyond surface-level pattern matching.