💬 NLP

Light or Full Verb? A Minimal-Pair Dataset for Probing Phraseological Competence in Language Models

This paper introduces a minimal-pair dataset to demonstrate that language models can distinguish between light-verb and full-verb uses of words like 'have' and 'make' within identical contexts, providing a reusable resource for probing phraseological competence.

Original authors: Francesca Franzon, Nicolas Rosàs Gómez, Leo Wanner

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Francesca Franzon, Nicolas Rosàs Gómez, Leo Wanner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a Swiss Army knife. Sometimes, you use the main blade to cut a rope (a full, powerful action). Other times, you use the tiny screwdriver just to turn a screw (a light, supportive action). The tool is the same, but its job changes completely depending on what you are doing with it.

In the English language, verbs like "make," "take," and "have" work exactly like that Swiss Army knife.

Full Verb: When you say "make a cake," the word "make" is doing heavy lifting. It means you are physically creating something.
Light Verb: When you say "make a decision," the word "make" is just a helper. The real meaning comes from the word "decision." You aren't physically building a decision; you are just performing the action of deciding.

The Big Question

The researchers at Universitat Pompeu Fabra wanted to know: Do AI language models (like the one you are talking to right now) actually understand this difference?

Or, do they just memorize that "make" and "decision" often appear together, without realizing that "make" is acting differently in that sentence compared to "make a cake"?

The Experiment: A "Twin" Test

To find out, the team built a special dataset. Think of it as a minimal pair test, like a "spot the difference" game for sentences.

They created thousands of sentences where everything was identical—the context, the grammar, the length—except for the object at the end.

Sentence A (Light): "During the afternoon, they made a decision."
Sentence B (Full): "During the afternoon, they made a cake."

By keeping the rest of the sentence exactly the same, they could see if the AI treated the word "decision" differently than "cake," even though the verb "made" was right there in both.

What They Found

They ran two main tests on a specific AI model (Gemma-3-270m):

1. The "Surprise" Test (Surprisal)
Imagine the AI is reading a sentence and trying to guess the next word.

When the AI saw "During the afternoon, they made...", it was less surprised to see the word "decision" than "cake."
This means the AI "knew" that "make a decision" is a common, standard phrase (a Light Verb Construction), so it expected it more strongly.
In reverse, when the sentence was passive ("A decision was made..."), the AI was less surprised to see the verb "made" than if it had been "A cake was made."
The takeaway: The AI isn't just memorizing words; it has a sense of which verbs act as "helpers" and which act as "creators."

2. The "Brain Scan" Test (Embeddings)
The researchers looked inside the AI's "brain" (its internal mathematical representations) to see how it visualized the objects ("decision" vs. "cake").

They found that the AI's internal map for "decision" in a light-verb sentence was clearly different from its map for "decision" in a full-verb sentence.
It's as if the AI has two different folders for the same word, depending on whether the verb next to it is doing heavy lifting or just helping out.

Why This Matters

Before this study, most tests just asked: "Is this sentence grammatically correct?" (e.g., "The decision was made" vs. "The decision was made by the cake").

This paper is special because it asks a harder question: "Can the AI tell the difference between two perfectly correct sentences where the verb changes its meaning?"

The answer is yes. The AI can distinguish between a verb acting as a "support worker" and a verb acting as a "main character," even when the sentence structure is identical.

The Bottom Line

The researchers have released this "twin sentence" dataset as a free tool for other scientists. It's like giving linguists a new microscope to check if future AI models are truly understanding how language works, or just guessing based on patterns. They plan to expand this to more verbs and other languages, but for now, they've proven that current models have a surprisingly sharp ear for the subtle difference between "making a cake" and "making a decision."

Technical Summary: Light or Full Verb? A Minimal-Pair Dataset for Probing Phraseological Competence in Language Models

Problem Statement
Frequent English verbs (e.g., make, have, take) exhibit a dual lexical-functional status: they function either as full lexical predicates (e.g., make a cake) or as light verbs (support verbs) in light-verb constructions (LVCs) where the semantic weight is carried by the nominal base (e.g., make a decision). While existing computational linguistics work has extensively treated LVCs as an identification or extraction problem—determining whether a given verb-noun pair constitutes an LVC—there is a gap in understanding whether language models (LMs) internally represent the distinction between these two uses. Current models may recognize LVCs through distributional association or noun-category cues without encoding the specific lexical-functional contrast between a support-verb use and a full lexical use. Existing minimal-pair benchmarks (e.g., BLiMP, SyntaxGym) typically contrast grammatical sentences with ungrammatical ones; however, probing LVC competence requires contrasting two grammatical, natural sentences that differ only in the lexical-functional status of the same surface verb.

Methodology
The authors introduce a large-scale, controlled dataset designed to probe this distinction using minimal sentence series.

Dataset Construction: The dataset covers five high-frequency English verbs (make, take, give, have, receive) known to occur in LVCs. For each verb, the authors curated 20–40 LVC candidates (validated against criteria such as predicative nominal bases and canonical collocational relations) and paired them with full-verb counterparts using concrete object nouns.
Experimental Design: The dataset generates sentence series where the same context is paired with different objects, yielding half LVC readings and half full-verb readings. These are constructed in both active and passive configurations (e.g., they made a decision vs. they made a cake; a decision was made vs. a cake was made).
Probing Experiments: The authors evaluated the Gemma 3 270m model using two complementary approaches:
1. Surprisal-Based Probing: Calculating token-level surprisal (negative log-probability) for critical regions. In active sentences, the critical region is the object noun; in passive sentences, it is the verb following the nominal base. This tests whether the model assigns higher probability to the expected construction type given the context.
2. Contrastive Embedding Analysis: Extracting hidden states for the object noun phrases, averaging token embeddings, and applying dimensionality reduction (PCA) and clustering (KMeans) to determine if the model's internal representations separate LVC objects from full-verb objects under matched syntactic and verbal contexts.

Key Contributions

A Minimal-Pair Dataset: The release of a reusable, extensible dataset of minimally varying English sentence series that isolates the lexical-functional contrast of light vs. full verbs while holding syntactic frames and contexts constant.
Methodological Framework: A shift from acceptable/unacceptable sentence contrasts to grammatical/grammatical contrasts specifically targeting the lexical-functional status of verbs.
Open Resources: The authors release the dataset, generation code, and materials to support extensions to broader contexts, additional verbs, and other languages.

Results

Surprisal Findings: The model demonstrates sensitivity to the LVC distinction. In active configurations, nominal bases in LVCs (e.g., decision in make a decision) generally receive lower surprisal than their full-verb counterparts, indicating the model assigns higher probability to conventional light-verb pairings. In passive configurations, predicative nominal bases increase the predictability of their associated light verbs, though variation exists across specific verbs (e.g., make shows the strongest effect).
Embedding Findings: Clustering analysis of object-level hidden states reveals that the model's internal representations separate LVC objects from full-verb objects. This separation is particularly distinct in later layers and when analyzing the change in representation between layers ( $\Delta_l$ ), suggesting the model updates object representations differently based on the construction type.

Significance and Claims
The paper claims that language models differentiate between light-verb and full-verb uses even in minimal contexts and exhibit separable patterns across object types. The findings suggest that models capture both verb-to-noun and noun-to-verb expectations in LVCs, reflecting this distinction in both token-level probabilities and the geometry of contextual object representations. The authors position this work as a controlled probing instrument to investigate phraseological competence, moving beyond simple identification tasks to assess whether models represent the underlying lexical-functional contrasts. The framework is designed to support future extensions, including additional verbs, broader contexts, and cross-linguistic applications, though the current study is limited to English and a specific set of verbs.