Technical Summary: The Grammar of Transformers

Problem Statement

Since the observation by Gulordava et al. (2018) that language models (LMs) acquire non-trivial syntactic knowledge despite being trained solely on token prediction, a growing body of research has sought to characterize the nature and extent of this knowledge. However, the field has lacked a general, quantitative picture of syntactic knowledge in Transformer-based Language Models (TLMs). Existing literature consists largely of narrative reviews (e.g., Limisiewicz and Mareček, 2020; Linzen and Baroni, 2021; Kulmizev and Nivre, 2022) which, while valuable, are interpretive and lack the transparent, quantitative synthesis necessary to map the landscape of findings across the rapidly expanding scale of recent research. Furthermore, the field is characterized by methodological heterogeneity and a heavy concentration on English and specific model architectures (notably BERT), limiting the robustness and generalizability of current conclusions.

Methodology

The authors conducted a systematic review following PRISMA guidelines (Page et al., 2021), a methodology not commonly applied in Computational Linguistics (CL) and NLP. The process involved:

1. Data Collection (Summer 2025): A dual-strategy approach was used to identify candidates: a keyword search in Google Scholar combining "syntactic structure/knowledge" with "LLMs/LMs/transformer," and a snowballing strategy leveraging six prior reviews. This yielded 636 initial articles.
2. Eligibility Assessment: Articles were filtered to include only those focusing on Transformer architectures (excluding RNNs, LSTMs, and encoder-decoder MT models), empirically assessing syntactic knowledge, and published in peer-reviewed venues or preprints. This resulted in a final corpus of 337 articles.
3. Annotation: The authors defined 33 annotation variables across five categories:
   • Meta-data: Publication details and entry/exit reasons.
   • Model-related: Model names, types (bidirectional vs. causal), fine-tuning status, and language coverage.
   • Experimental Design: Syntactic phenomena (categorized into 11 labels via a bottom-up approach), languages tested, and materials.
   • Interpretability Techniques: Categorized as behavioral, probing, or mechanistic (following Milli`ere, 2024), with further annotation on causal nature (observational vs. interventional), training phase, and locality.
   • Findings: Prose summaries of syntactic competence and model comparisons.
   • Note: 28 categories were annotated manually; 5 were semi-automated using an AI-assisted workflow validated against a manual gold standard.

Key Contributions

1. A Quantitative Synthesis: The paper provides the first systematic, quantitative review of syntactic knowledge in TLMs, aggregating 3,074 individual datapoints across languages, phenomena, and methods.
2. Open Resource: The authors contribute a publicly available, annotated database of the reviewed studies and the associated analysis code, designed to be extensible.
3. Taxonomy of Phenomena: The work establishes a bottom-up typology of 11 syntactic phenomena labels recognizable to both linguistics and NLP communities, facilitating cross-study comparison.
4. Methodological Audit: The review critically assesses the distribution of interpretability methods (behavioral, probing, mechanistic) and highlights the field's over-reliance on observational rather than causal approaches.

Key Results

1. Field Landscape and Bias

• Temporal Trends: Interest in the topic surged post-BERT (2019), peaking in 2022. While early work focused on bidirectional models, causal models have dominated recent years.
• Model Bias: The literature is heavily concentrated on BERT and its variants, which account for 58% of all datapoints. The four most studied models account for 62%.
• Language Bias: English is overwhelmingly dominant (69% of articles are English-only; 91% include English). Studies on non-English languages are scarce, and multilingual studies are underrepresented (20%).
• Phenomena Bias: Research focuses heavily on formal syntactic phenomena (e.g., agreement, word order, hierarchical structure, POS) which are easily captured by current NLP tools. Phenomena at the syntax-semantics interface (e.g., binding, coreference, negation, ellipsis) and typological aspects receive significantly less attention.

2. Behavioral Performance

• BLiMP Benchmark: On the English BLiMP benchmark, TLMs perform remarkably well, with the best models matching or exceeding human performance in 7 out of 11 categories. However, performance varies significantly across models, driven largely by overall model capability rather than phenomenon-specific strengths.
• Formal vs. Interface: TLMs consistently perform better on formal syntactic relations (medians >85% for agreement) than on syntax-semantics interface phenomena (medians <75% for binding, argument structure, NPIs).
• Cross-Linguistic Factors: Regression analysis of 1,557 non-benchmark datapoints across 113 languages reveals that causal models outperform bidirectional models by ~10%, and performance scales positively with digital language support (5% gain per support level). Lower-resourced languages show degraded performance.

3. Internal Mechanisms (Probing and Mechanistic Studies)

• Probing: 85% of probing studies report successful recovery of syntactic information from model representations. The middle layers are most frequently identified as the locus of syntactic knowledge, supporting a distributed representation view.
• Mechanistic Studies: 79% of mechanistic papers find clear evidence supporting the presence of syntactic knowledge. However, these studies are overwhelmingly observational (81%); only 19% employ interventional methods to establish causal links between internal representations and behavior.
• Methodological Gaps: There is a disconnect between probing success (high recovery rates) and behavioral performance (variable across phenomena), suggesting that successful probing does not necessarily imply functional use of representations for prediction.

Significance and Claims

The paper claims that the current literature robustly supports the view that TLMs encode a non-trivial amount of syntactic knowledge, particularly regarding formal syntactic relations. However, the authors emphasize that the field's conclusions are currently limited by:

1. Lack of Causal Evidence: The dominance of observational methods prevents a deep understanding of how syntactic representations are used computationally.
2. Limited Generalizability: The heavy focus on English and BERT-like models restricts the applicability of findings to the broader landscape of language and architecture.
3. Fragmented Phenomena: The syntax-semantics interface remains underexplored despite models showing weaker performance in these areas.

The authors conclude that while TLMs possess remarkable syntactic capabilities, understanding the acquisition, usage, and universality of this knowledge requires greater methodological harmonization, more explicit theoretical commitments, and a broader empirical landscape that moves beyond English and purely observational studies. They position this review as a foundation for future work, inviting the community to contribute to the open database and adopt the proposed annotation standards.