Technical Summary: A Large-Scale Pipeline for Automatic Neologism Detection

Problem Statement

The automatic detection of neologisms (lexical innovations) faces a fundamental scalability challenge. While large-scale social media corpora offer rich sources of language change, the vast majority of tokens absent from standard dictionaries are not neologisms but rather typos, misspellings, concatenated strings, code fragments, or foreign-language material. In a corpus of 527 million English-language Reddit posts, 124.6 million unique tokens were extracted. Classifying each of these individually using Large Language Models (LLMs) is computationally infeasible. Furthermore, existing detection systems often rely on static, language-specific exclusion dictionaries that lack portability and require extensive manual verification of large candidate sets. The core problem is to design a pipeline that reduces this massive candidate set to a manageable size for expert verification while maintaining high recall for genuine lexical innovations.

Methodology

The authors present a scalable, modular pipeline that combines deterministic rule-based filtering with multi-model LLM classification. The system is grounded in two complementary word-formation frameworks:

1. Grammatical Word Formation: Based on Štekauer's onomasiological theory, covering rule-governed processes like affixation and compounding.
2. Extra-Grammatical Word Formation: Based on Mattiello's framework, covering non-rule-governed processes like blending, clipping, and expressive morphology (e.g., doggo).

The pipeline operates through a multi-stage filtering cascade:

1. Tokenization & Preprocessing: Using spaCy, the system removes non-lexical content (URLs, mentions, emojis) and normalizes text.
2. Vocabulary Filtering: Tokens present in reference vocabularies compiled exclusively from pre-2015 sources (16.3 million surface forms from sources like WordNet, Wiktionary, and Urban Dictionary) are excluded.
3. Pattern-Based Cleaning: Language-independent and language-specific rules remove noise, such as excessive character repetition, low entropy strings, and implausible phonotactic clusters.
4. Typo and Concatenation Detection: Using SymSpell, the system flags tokens that are likely misspellings or space-less concatenations of existing words based on edit distance and segmentation against a frequency dictionary.
5. Frequency Threshold & Reintegration: Tokens flagged as errors are reconsidered if they appear frequently (threshold ≥100 occurrences) in the corpus. This mechanism prevents the premature discarding of genuine high-frequency neologisms that might resemble typos (e.g., vtuber).
6. Foreign Language Detection: A language detector (Lingua) filters tokens identified as non-English, though a confidence threshold allows ambiguous cases to proceed to classification.
7. LLM Classification: The remaining candidates are classified into four categories: Neologism (new words/slang), Entity (proper nouns), Foreign (non-English words), and None (residual noise).
   • Ensemble Voting: Three open-source models (Qwen 2.5 72B, LLaMA 3.3 70B, Mistral Large 2 123B) independently classify tokens. A majority vote determines the label.
   • Verification: A fourth model (Claude 4.5 Haiku) acts as a strict verification filter, re-evaluating the majority-vote neologism candidates to reduce false positives.

Key Contributions

1. Scalable Pipeline Architecture: A modular system capable of processing hundreds of millions of tokens, reducing a 124.6 million token set to 1,021 candidates (a 122,031:1 compression ratio) suitable for manual expert verification.
2. Theoretical Grounding: The classification scheme is explicitly derived from grammatical and extra-grammatical word-formation theories, ensuring the "neologism" label captures both rule-based and analogical innovations.
3. Comparative LLM Evaluation: A systematic evaluation of multiple LLMs on a four-class neologism classification task, revealing significant cross-model disagreement and demonstrating the value of ensemble methods.
4. Annotated Dataset: The release of a manually annotated list of 1,021 pipeline output candidates, including gold labels, error analysis, and classification by word-formation process.

Results

Applied to the 527 million Reddit posts (2005–2024), the pipeline yielded the following results:

• Filtering Efficiency: The rule-based stages reduced the initial 124.6 million unique tokens by 99.86%, leaving 174,973 candidates for LLM classification.
• Model Disagreement: There was substantial disagreement among the three open-source models. LLaMA was the most aggressive in predicting neologisms (12.2%), while Mistral was the most conservative (assigning 59.9% to "none"). Unanimous agreement occurred in only 45.8% of cases.
• Verification Impact: The majority vote produced 10,499 neologism candidates. The verification step using Claude 4.5 Haiku rejected 90.3% of these, confirming only 897 as neologisms.
• Final Output: After verification, 1,021 candidates remained. Manual annotation confirmed that 599 (58.7%) were genuine lexical innovations (465 neologisms and 134 entities). The remaining 422 were false positives (361 "none" and 61 foreign language tokens).
• Word Formation Patterns: The 599 confirmed neologisms exhibited diverse formation processes, including standard grammatical processes (prefixation, suffixation, compounding) and extra-grammatical processes (blending, expressive morphology, and analogical formations like finfluencer and tradwife).
• Recall Estimate: Based on a reference list of 103 documented neologisms, the pipeline detected 20 true positives. However, 48 were correctly excluded as pre-2015 forms, and 33 were false negatives. The recall for genuinely post-2015 items was 37.7% (20/53), with the primary source of false negatives being vocabulary homograph conflicts (e.g., rizz as a character name vs. the slang term).

Significance and Claims

The paper positions the pipeline not as a perfect classifier but as a high-recall candidate generator. Its primary significance lies in making the task of neologism detection feasible for human experts by compressing an unmanageable dataset into a small, verifiable set.

The authors claim that:

• The pipeline successfully operationalizes complex linguistic theories (Štekauer and Mattiello) into a computational workflow.
• The combination of rule-based pre-filtering and multi-model LLM classification is necessary to handle the noise inherent in social media data.
• The resulting dataset of 599 verified neologisms provides a valuable resource for studying contemporary word-formation processes, capturing patterns from standard compounding to analogical "secreted" combining forms (e.g., -fluencer, -pilled).
• The error analysis highlights that future improvements should focus on the earliest filtering stages (vocabulary and tokenization) rather than downstream classification, particularly regarding semantic shifts and homograph conflicts.

The authors explicitly note limitations, including the inability to detect multi-word expressions, semantic shifts (e.g., Karen), or non-alphabetic innovations (algospeak), and acknowledge that the current instantiation is English-specific, though the architecture is designed to be transferable.