'Layer su Layer': Identifying and Disambiguating the Italian NPN Construction in BERT's family

1. Problem Statement

Pretrained Language Models (PLMs) like BERT achieve high predictive accuracy but remain largely "opaque" regarding the specific linguistic information they encode. While previous interpretability research has focused on English, there is a need to evaluate whether PLMs capture Construction Grammar (CxG) principles—specifically, the mapping of form to meaning in semi-schematic constructions—in other languages.

The paper focuses on the Italian noun-preposition-noun (npn) construction family (e.g., layer su layer, "layer upon layer"). This construction is theoretically complex because:

• It involves nominal reduplication interrupted by a preposition.
• It forms a network of horizontally related constructions with overlapping lexemes and subtle semantic distinctions (e.g., succession vs. juxtaposition).
• It sits on the continuum between the lexicon and syntax, making it a rigorous test case for whether models learn abstract constructional schemas or merely surface-level distributional regularities.

The core research questions are:

1. RQ1: Can BERT distinguish Italian npn construction instances from superficially similar distractors?
2. RQ2: Can BERT distinguish between the specific semantic meanings associated with different npn instantiations within the same family?

2. Methodology

Dataset Construction

The authors created a dataset based on Masini (2024a), expanded with full sentential contexts from the CORIS corpus.

• Target Constructions: Four specific Italian npn semi-schematic constructions using prepositions a ("at/to") and su ("on").
• Labels: Instances were annotated with three semantic categories:
  1. Succession/Iteration/Distributivity.
  2. Greater Plurality/Accumulation.
  3. Juxtaposition/Contact.
• Distractors: 1,751 instances of non-target patterns, including:
  • Surface-isomorphic patterns (e.g., N-prep-N where the noun is part of a larger phrase).
  • Structurally distinct constructions (e.g., pnpn, verbal constructions).
• Split Strategy: Unlike previous English studies that enforced strict lemma-level disjointness, this study used a modified split: lemma-label pairs were disjoint between train/test, but the same lemma could appear across different labels. This tests if the model learns constructional distinctions beyond lexical identity.

Models

The study evaluated four models from the BERT family:

1. BERT-base-italian-cased: Monolingual Italian.
2. mBERT: Multilingual BERT.
3. UmBERTo: RoBERTa-based, monolingual Italian.
4. XLM-RoBERTa: Multilingual RoBERTa.

Experimental Design

Two probing tasks were implemented using linear logistic regression classifiers trained on contextual embeddings extracted from each layer of the models:

1. Construction Identification (Binary Classification): Distinguishing npn constructions from distractors.
   • Variations: Three training configurations were tested to analyze the impact of distractor types: SIMPLE (all distractors), OTHER (training on structurally distinct distractors only), and PSEUDO (training on surface-isomorphic distractors only).
   • Inputs: Embeddings of the preposition token (PREP) and the preposition replaced by [UNK] (UNK) to test reliance on lexical semantics vs. contextual structure.
2. Semantic Disambiguation (Multi-class Classification): Distinguishing between the three semantic labels.
   • Generalization Test: A specific experiment trained on prepositions a and su and tested on unseen prepositions per ("by") and dopo ("after") to test abstract semantic generalization.

Baselines

• Static Embeddings: FastText (chosen over GloVe for better handling of Italian morphology).
• Morphological Baseline: FastText vectors built from the inflected noun form (to account for number agreement, a key cue for distinguishing su singular vs. plural).
• Control Classifier: Trained on random labels to ensure performance isn't due to spurious correlations.

3. Key Contributions

1. Cross-Linguistic Extension: Extends the npn construction probing framework from English to Italian, validating the applicability of Construction Grammar analysis in non-English PLMs.
2. Refined Distractor Design: Moves beyond simple syntactic discrimination by introducing heterogeneous distractors (structurally distinct vs. surface-isomorphic), offering a more nuanced view of what "constructional knowledge" entails.
3. UNK vs. PREP Analysis: Systematically compares representations of the preposition itself against a masked token to isolate the contribution of lexical semantics versus contextual/structural cues.
4. Generalization to Unseen Prepositions: Demonstrates that models can generalize semantic meanings to prepositions not seen during training, suggesting the encoding of abstract constructional schemas rather than just preposition-specific associations.

4. Results

Construction Identification

• Performance: All BERT models achieved high accuracy in distinguishing npn constructions from distractors, significantly outperforming static FastText baselines in middle-to-late layers.
• UNK vs. PREP: Both [UNK] and PREP embeddings yielded similar high performance. This indicates that successful discrimination relies heavily on the surrounding nouns and sentential context, not just the lexical semantics of the preposition.
• Distractor Sensitivity: Performance dropped slightly when training on only surface-isomorphic distractors (PSEUDO) or structurally distinct ones (OTHER), highlighting that the definition of "minimal pairs" significantly impacts probing results.
• Error Patterns: Most errors were False Positives (distractors classified as constructions), suggesting the model robustly encodes constructional features but sometimes over-generalizes.

Semantic Disambiguation

• Baseline Strength: Static baselines performed surprisingly well, particularly when using inflected noun forms (morphological cues). This suggests that surface morphology (e.g., singular vs. plural) is a strong predictor for certain meanings (e.g., greater plurality).
• Layer Dynamics: BERT models outperformed static baselines only in the highest layers, indicating that abstract constructional distinctions are encoded in deeper layers, while lower layers rely on surface cues.
• Semantic Confusion: The model struggled to distinguish between Juxtaposition/Contact and Succession/Iteration when both used the preposition a. This suggests that for high-frequency, lexicalized types (e.g., gomito a gomito), the boundary between spatial contact and temporal succession is gradient and difficult for the model to categorize strictly.
• Generalization: In the generalization test (training on a/su, testing on per/dopo), the model achieved high accuracy in late layers. This confirms that the model learns an abstract "succession/distribution" schema that applies across different prepositions, rather than memorizing specific preposition-label pairs.

Model Comparisons

• Monolingual vs. Multilingual: Monolingual Italian models (BERT-ita, UmBERTo) generally outperformed multilingual models (mBERT, XLM-R) on the Italian disambiguation task, though they were comparable on the identification task. This highlights the importance of language-specific pretraining for capturing fine-grained constructional nuances.

5. Significance

• Theoretical Validation: The results provide empirical support for Construction Grammar, showing that PLMs encode structured form-meaning pairings that go beyond simple distributional statistics.
• Abstraction Path: The study demonstrates an "abstraction path" in PLMs: lower layers capture lexical/morphological cues, while higher layers encode abstract constructional regularities capable of generalizing to unseen instantiations.
• Methodological Caution: The paper emphasizes that probing results are highly sensitive to task design (specifically distractor selection). It warns against uncritical expansion of the "Constructicon" (network of constructions) based solely on model behavior without considering how task definitions shape performance.
• Future Directions: The authors note that while models encode this information, it remains to be seen if they use it causally for prediction. Future work involves cross-validating these findings with human behavioral experiments to assess cognitive plausibility.

In conclusion, the paper successfully argues that BERT-family models, particularly in their deeper layers, encode the Italian npn constructional family in a way that aligns with constructionist theories, distinguishing between form and meaning and generalizing across prepositions, though with limitations regarding the gradient nature of certain semantic boundaries.