A large-scale pipeline for automatic corpus annotation using LLMs: variation and change in the English consider construction

Technical Summary: A Large-Scale Pipeline for Automatic Corpus Annotation Using LLMs

1. Problem Statement

The primary challenge addressed in this paper is the "annotation bottleneck" in corpus linguistics. While digital text corpora (like COHA) have grown to billions of words, the ability to perform nuanced syntactic and semantic classification remains limited by the speed and cost of human manual annotation.

Traditional automation methods—such as rule-based pattern matching or supervised machine learning—often fail to capture the complex, context-dependent nature of grammatical constructions (e.g., distinguishing between different semantic uses of the same verb). Consequently, researchers often settle for smaller, less statistically powerful samples, potentially missing rare phenomena or subtle diachronic (historical) changes.

2. Methodology

The authors propose a redesigned, scalable, four-phase pipeline that moves away from the "iterative/supervised" chat-based approach used in previous studies toward a fully automated batch-processing workflow.

The Four-Phase Pipeline:

1. Comprehensive Prompt Engineering: Creating a self-contained instruction set using XML tags, "Chain of Thought" reasoning (instructing the model to think step-by-step), and explicit handling of edge cases and prototypical examples.
2. Pre-hoc Evaluation: Validating the prompt on multiple independent random samples to ensure accuracy meets a strict threshold (recommended >95%) before full deployment.
3. Automated Batch Processing: Using an LLM API (specifically GPT-5) to process large volumes of data in batches (e.g., 100 concordance lines per call) without human intervention.
4. Post-hoc Validation: Using stratified random sampling (ensuring all decades and genres are represented) to verify the reliability of the entire annotated dataset.

Case Study Implementation:
The authors applied this pipeline to the Corpus of Historical American English (COHA) to study the evaluative consider construction (e.g., consider X as/to be/Ø Y). The task involved two sequential steps:

• Task 1: Distinguishing "evaluative" uses from "non-evaluative" (cognitive) uses.
• Task 2: Classifying the evaluative instances into three complement variants: zero, to be, and as.

3. Key Contributions

• Methodological Framework: A replicable, four-stage workflow that prioritizes "front-loading" intellectual labor (prompt design/validation) to allow for massive-scale execution.
• Scalability & Efficiency: Demonstrating that a task requiring ~400 hours of manual labor can be completed in under 60 hours of automated processing.
• Linguistic Theory Advancement: Using the automated data to provide empirical evidence for the relationship between register formality and morphosyntactic change (reduction vs. enhancement).
• Transparency & Ethics: Providing detailed documentation on API costs, data licensing compliance (ensuring data is not used for model training), and environmental impact.

4. Results

• High Accuracy: The pipeline achieved 98%+ accuracy across both classification tasks. Post-hoc validation confirmed that the error rate was negligible and did not bias the linguistic findings.
• Computational Efficiency: The researchers processed 143,933 concordance lines in approximately 58.6 hours at a total cost of roughly $104 USD.
• Linguistic Findings:
  • The as-variant has undergone a steady decline (obsolescence) across most genres.
  • The zero-variant became the dominant form through a process of reduction, primarily driven by informal/narrative genres (fiction, TV/movies).
  • The to be-variant shows signs of enhancement (increased explicitness), but this growth is largely restricted to academic prose, suggesting that formal registers prioritize precision over the economy found in informal registers.

5. Significance

This paper marks a shift in corpus linguistics from "manual curation" to "automated orchestration." It demonstrates that LLMs can act as highly reliable "copilots" for linguists, enabling the study of massive datasets that were previously beyond practical reach. By democratizing access to sophisticated annotation—requiring only basic Python skills rather than deep machine learning expertise—the authors provide a toolset that can be applied to diverse languages and complex grammatical phenomena, ultimately allowing for more robust and statistically significant linguistic theories.