💬 NLP

Operationalizing Linguistic Methods through Prompt-Engineering Skills: An Automatic Chinese Web Neologism Detection Pipeline

This paper presents an automatic pipeline for detecting Chinese web neologisms that translates traditional linguistic principles into prompt-engineering skills, achieving high coverage on a large corpus while identifying candidate generation and semantic classification as key bottlenecks through a novel conditional recall decomposition analysis.

Original authors: Yufeng Wu, Meichun Liu

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yufeng Wu, Meichun Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find new, trendy slang words that people are inventing on the Chinese internet. It's like trying to catch fish in a massive, murky ocean where the water is full of old, known fish, random debris, and a few brand-new, shiny creatures you've never seen before.

This paper describes a four-stage fishing net designed specifically to catch these new "internet fish" (neologisms) automatically, without needing a human to look at every single piece of debris.

Here is how the process works, using simple analogies:

The Big Picture: Turning Rules into "Skills"

Traditionally, linguists have written books describing how new words are made (like how you can combine two words to make a new one). But those books are just descriptions; they don't tell a computer how to do it.

The authors took those linguistic rules and turned them into "skills" for an AI (a Large Language Model). Think of it like giving a very smart but literal-minded intern a specific checklist and a set of examples, rather than just saying, "Find me new words."

The Four-Stage Pipeline

Stage 1: The Big Net (Candidate Generation)
The team started with a massive library of 267 million Chinese web documents. Instead of using a standard dictionary to chop up the text, they just grabbed every possible combination of 2, 3, or 4 characters that appeared frequently.

The Result: They pulled up a bucket containing 1.2 million potential word candidates. This is the "raw catch."

Stage 2: The Dictionary Check & The Glue Test (Pre-filtering)

The Dictionary Check: They checked if these words were already in the standard 2005 Chinese dictionary. If they were, they threw them out (because they aren't new).
The Glue Test (PMI): They used a math test to see if the characters in a candidate "stick" together tightly. For example, "social" and "death" stick together well to make "social death" (a new concept), but random characters like "apple" and "cloud" might not. If the glue is weak, the candidate is discarded.
The Result: The bucket shrinks to 783,000 candidates.

Stage 3: The Grammar Architect (Well-formedness Skill)
This is where the AI gets its first "skill." The AI looks at the remaining candidates and asks: "Does this look like a real Chinese word structure?"

It checks if the word follows rules like "Adjective + Noun" or "Verb + Object."
It ignores whether the AI has heard the word before; it only cares if the structure makes sense.
The Result: The bucket shrinks to 226,000 candidates that look structurally sound.

Stage 4: The Final Judge (Three-Way Classification)
This stage splits into two steps to decide what the word actually is:

4A (The Rule Filter): A simple rule-based filter quickly removes obvious junk (like words starting with "the" or ending with a preposition that makes them sound like sentence fragments).
4B (The AI Judge): The AI uses a second "skill" to make the final call. It has to choose between three options:
1. Neologism: A brand-new, valid slang word.
2. Entity: A name of a person, place, or thing (not a new word).
3. None: Just random noise or an existing phrase.
The Result: The final bucket contains 226,959 classified items, including 4,853 confirmed new words.

The "X-Ray" Evaluation: Finding the Leaks

The authors didn't just say, "We found 4,853 words!" They wanted to know where they lost the words they should have found. They used a special "X-ray" method called conditional recall decomposition.

Imagine you have a leaky bucket with five holes. Instead of just measuring how much water is left at the end, they measured how much water leaked out at each specific hole.

The Findings:

Two Big Leaks: They found that most of the "new words" were lost at the very beginning (Stage 1, because the initial net wasn't wide enough) and at the very end (Stage 4B, because the AI struggled to decide if a word was truly "new" or just a known entity).
The Length Problem: They discovered a funny pattern based on word length:
- The AI was great at checking if 2, 3, or 4-character words were structurally correct (it passed almost all of them).
- However, the AI got worse at deciding if they were new as the words got longer. It was good at spotting new 2-character words, but its accuracy dropped significantly for 4-character words.

The Bottom Line

The paper proves that you can turn old-school linguistic rules into modern AI "skills" to automatically find new words. They released their list of 4,853 new words and their "X-ray" testing method as a public resource.

However, they also found a boundary: while AI is excellent at checking if a word looks like a word, it still struggles a bit with the harder job of deciding if that word is truly new, especially when the word is long and complex.

Technical Summary: Operationalizing Linguistic Methods through Prompt-Engineering Skills

Problem Statement
Chinese web neologisms—new words and expressions emerging from online language use—present a persistent challenge for computational lexicography. While traditional Chinese linguistics offers well-developed descriptive accounts of word-formation principles (e.g., modification, verb-object structures, affixation) and the boundaries between novel coinages and existing lexical inventory, these accounts have largely remained descriptive. They are used by linguists for manual identification rather than for automatic detection. Bridging the gap between descriptive linguistic knowledge and automatic detection procedures has been a long-standing goal in Chinese NLP. Furthermore, while Large Language Models (LLMs) offer new possibilities for operationalizing linguistic knowledge via prompts, simply asking an LLM to "judge if this is a new word" does not equate to encoding linguistic principles in a reproducible, structured form.

Methodology
The authors propose a four-stage, fully automatic pipeline for detecting Chinese web neologisms from the BAAI CCI 3.0 corpus (267 million documents). The core innovation is the operationalization of traditional linguistic identification principles as prompt-engineering skills. A "skill" is defined here as a structured LLM-task interface combining explicit rule specifications, representative examples, and format constraints.

The pipeline proceeds as follows:

Stage 1: Tokenizer-Independent Candidate Generation. To avoid lexical bias inherent in tokenizer-dependent pipelines, the method extracts character n-grams (lengths 2, 3, and 4) directly via frequency counting, retaining the top 400,000 candidates for each length (1.2 million total).
Stage 2: Dictionary Anchoring and Statistical Pre-filtering. Candidates are filtered against the Modern Chinese Dictionary (5th edition, 2005) to remove pre-existing words. A Pointwise Mutual Information (PMI) pre-filter is applied (PMI > 2 for 2/3-char; > 4 for 4-char) to control downstream costs, though the authors acknowledge this may discard some low-cohesion neologisms.
Stage 3: Structural Well-formedness Skill (LLM). Using the Qwen3.5-27B model, candidates are classified as WELL_FORMED or FRAGMENT. The prompt encodes Chinese word-formation principles (modification, verb-object, parallel, quadrisyllabic templates, affixation). This stage ensures structural validity rather than memory-based recognition.
Stage 4: Three-way Classification.
- 4A (Rule-based): A chain of five character-position rules filters out obvious non-words (e.g., function-word prefixes/suffixes).
- 4B (LLM Skill): The remaining candidates undergo a three-way classification (NEOLOGISM, ENTITY, NONE) via an LLM skill. The skill includes class definitions, examples, and decision rules (e.g., uncertainty defaults to NONE).

Evaluation Protocol
Instead of reporting a single aggregate recall score, the authors develop a per-stage conditional recall decomposition. By tracing the survival of entries from the Hou (2023) New Chinese Words dictionary (4,199 entries) through each stage, they compute the conditional recall ( $R_k$ ) for each stage. The overall strict recall is mathematically the product of these stage recalls ( $R_1 \times R_2 \times R_3 \times R_{4A} \times R_{4B}$ ). This allows for diagnostic comparison at the sub-pipeline level.

Key Results

Pipeline Output: Applied to the CCI 3.0 corpus, the pipeline produced 226,959 classified candidates, including 4,853 labeled neologisms, 21,119 entities, and 200,987 non-words.
Bottleneck Analysis: The per-stage decomposition reveals a bimodal bottleneck structure.
- Stage 1 (Candidate Coverage): $R = 41.5\%$
- Stage 4B (Semantic Classification): $R = 60.0\%$
- Intermediate stages (2, 3, 4A) are near-lossless ( $R > 97\%$ ).
- Together, Stage 1 and Stage 4B account for 83.7% of false negatives.
Length-Stratified Diagnosis: A critical finding is the dissociation between the two LLM skills regarding candidate length:
- Structural Well-formedness (Stage 3): Length-invariant, retaining ~97–100% of 2, 3, and 4-character entries.
- Semantic Novelty Classification (Stage 4B): Length-dependent, with recall dropping significantly as length increases (65.6% for 2-char, 59.0% for 3-char, 44.1% for 4-char).
Cross-Lingual Comparison: The Stage 4B recall (60.0%) closely matches the 58.7% true-novelty rate reported by Rossini and van der Plas (2026) for English neologism detection, suggesting a potential cross-lingual ceiling for current LLM-based classification.

Significance and Claims
The paper makes four methodological contributions:

Operationalization: It demonstrates a method to convert descriptive linguistic knowledge into reproducible, automatic detection procedures via prompt-engineering skills.
Diagnostic Evaluation: It introduces a per-stage conditional recall evaluation protocol that decomposes pipeline performance into multiplicative factors, enabling granular error attribution.
Resource Release: It releases a public resource of 226,959 classified candidates from a massive Chinese web corpus.
Boundary Mapping: It identifies a specific boundary in skill-based linguistic operationalization: while LLMs can reliably judge structural word-formation (length-invariant), their ability to judge semantic novelty degrades significantly as candidate length increases.

The authors modestly claim that this work does not introduce the concept of "skills" (which is established practice) but rather investigates the reliability and boundaries of operationalizing linguistic identification knowledge through this paradigm. The results suggest that while the structural aspect of neologism detection is well-handled by current LLMs, semantic novelty judgment remains a significant challenge, particularly for longer candidates.

The Big Picture: Turning Rules into "Skills"

The Four-Stage Pipeline

The "X-Ray" Evaluation: Finding the Leaks

The Bottom Line

Technical Summary: Operationalizing Linguistic Methods through Prompt-Engineering Skills

More like this