A Hierarchical and Attentional Analysis of Argument Structure Constructions in BERT Using Naturalistic Corpora

Technical Summary: A Hierarchical and Attentional Analysis of Argument Structure Constructions in BERT Using Naturalistic Corpora

Problem Statement
The central question addressed by this study is whether large language models (LLMs), specifically the Bidirectional Encoder Representations from Transformers (BERT), develop internal representations of abstract grammatical knowledge that mirror human linguistic theories, particularly Construction Grammar. While previous research has demonstrated that models like BERT can capture syntactic dependencies, much of this work relies on synthetically generated or templated sentences, which may lack the variability and ecological validity of natural language. Furthermore, existing studies often aggregate data across diverse genres, potentially confounding constructional knowledge with stylistic variations. There is a specific gap in understanding how BERT processes four fundamental Argument Structure Constructions (ASCs)—Resultative, Caused-Motion, Ditransitive, and Way—within a controlled, naturalistic context, and how these representations evolve dynamically across the model's processing layers.

Methodology
To address these limitations, the authors employed a multi-dimensional analytical framework using a strictly controlled dataset.

• Corpus Construction: The study extracted 800 sentences (200 per construction type) exclusively from the fiction genre of the British National Corpus (BNC) and the Corpus of Contemporary American English (COCA). This genre control aimed to isolate constructional syntax and semantics from stylistic variation.
• Data Processing: Sentences were annotated for core syntactic roles (Subject, Verb, Direct Object, Indirect Object, Preposition, and the keyword "way") using SpaCy. These roles were mapped to BERT's sub-word tokens, with the first sub-word token serving as the representative vector.
• Model Analysis: The pre-trained bert-base-uncased model was used without fine-tuning. The study extracted contextualized embeddings and attention weight matrices from all 12 Transformer encoder layers.
• Analytical Techniques:
  1. Dimensionality Reduction: Multidimensional Scaling (MDS) and t-Distributed Stochastic Neighbor Embedding (t-SNE) were applied to visualize the geometric clustering of construction types.
  2. Cluster Separation Metrics: The Generalized Discrimination Value (GDV) was calculated to quantify the separation of construction clusters across layers.
  3. Linear Probing: Linear Support Vector Machine (SVM) classifiers were trained on token embeddings (CLS, Subject, Verb, Object) to test the linear accessibility of construction category information.
  4. Attention Analysis: The Fisher Discriminant Ratio (FDR) was computed on attention weights to identify which syntactic tokens and attention heads provide the most diagnostic information for distinguishing between constructions.

Key Results
The analysis revealed a hierarchical and dynamic process of constructional representation within BERT:

• Layer-wise Evolution:
  • Early Layers (1-2): Representations appeared as an "undifferentiated soup" with extensive overlap between construction types, dominated by shallow lexical and local syntactic features.
  • Middle Layers (3-6): Construction-specific information emerged, forming maximally separable clusters. The GDV analysis identified Layer 4 as the point of optimal linear separability. The "Way" construction consistently formed the most compact and isolated cluster, highlighting its unique status as a highly schematic form-meaning pairing.
  • Later Layers (7-12): Clusters remained distinct but underwent subtle refinement, suggesting a shift toward integrating information for downstream task-specific predictions.
• Linear Accessibility: By Layer 2, construction category information became linearly decodable from the embeddings of all major syntactic tokens (Subject, Verb, Object, CLS), with classification accuracy exceeding 90% and remaining stable through the final layers.
• Attentional Specialization: A significant dissociation was observed between linear accessibility and attentional focus. While construction information was present in all token embeddings, the attention mechanism specialized in diagnosing constructions primarily through the Verb-Object (VERB-OBJ) relationship. The Object token exhibited the highest FDR scores, followed by the Verb, while Subject and CLS tokens showed minimal discriminative power in attention patterns.
• Substructure: t-SNE visualizations revealed internal substructures within construction clusters (e.g., within the Ditransitive construction), suggesting the model captures granular, usage-based variations beyond abstract schemas.

Key Contributions

1. Ecological Validity: The study provides evidence based on a naturalistic corpus from a single genre, mitigating the confounds of synthetic data and cross-genre stylistic variation.
2. Hierarchical Mapping: It delineates a clear trajectory of how abstract grammatical schemas emerge, peak in separability, and are maintained within the Transformer architecture, identifying the middle layers as the locus of constructional abstraction.
3. Mechanistic Insight: The research uncovers a dissociation where constructional identity is globally encoded in embeddings but locally diagnosed by the attention mechanism via the core predicate-argument nexus (Verb-Object).
4. Validation of Construction Grammar: The findings offer computational evidence for Construction Grammar theories, demonstrating that models trained solely on next-token prediction spontaneously organize internal representations to mirror theoretically defined ASC categories, including the distinct status of the "Way" construction.

Significance
The paper claims that its findings establish a methodological bridge between computational linguistics and cognitive science. By demonstrating that BERT systematically encodes constructional abstractions in a structured, hierarchical manner, the study suggests that these models can serve as computational testbeds for theoretical linguistic concepts. The results support the "pipeline hypothesis" of contextualized representations, where syntactic and semantic abstractions crystallize in middle layers before pragmatic details are resolved. Furthermore, the study posits that the ability of LLMs to learn these patterns from natural data alone strengthens the argument for the psychological reality of constructions as fundamental units of linguistic cognition. The work concludes that modern LLMs are not merely black boxes but develop internal representations that resonate with long-standing theories of how humans store and process linguistic knowledge.