💬 NLP

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

This paper systematically evaluates open-source large language models for generative named entity recognition, demonstrating that with parameter-efficient fine-tuning and structured output formats, they achieve performance competitive with traditional models while relying on instruction-following capabilities rather than memorization, all without compromising general model abilities.

Original authors: Qi Zhan, Yile Wang, Hui Huang

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qi Zhan, Yile Wang, Hui Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart librarian (a Large Language Model, or LLM) who has read almost every book in the world. For years, if you wanted to find specific names in a text—like people, places, or organizations—you had to hire a specialized, rigid robot (traditional NER models) trained only for that one job.

This paper asks a big question: Can we just ask our giant librarian to find those names for us, and will they do a better job than the specialized robot?

Here is the breakdown of their findings, using simple analogies:

1. The New Way of Working: "The Rewrite" vs. "The Highlighter"

The Old Way (Traditional Models): Imagine a highlighter pen that only knows how to color specific words. It goes through a sentence and slaps a "Person" sticker on "John" and a "Place" sticker on "New York." It's fast and accurate, but it's a bit robotic.
The New Way (Generative NER): Imagine asking the librarian to rewrite the sentence for you. You say, "Here is a sentence; please rewrite it, but put the names inside brackets like this: [John | Person]."
The Result: The paper found that if you give the librarian clear instructions and a specific format (like using brackets or XML tags), they can rewrite the sentence just as accurately as the specialized highlighter robot. In fact, they often beat the older "zero-shot" versions of the librarian (where you just ask without training) and even rival the best specialized robots.

2. The Shape of the Answer Matters

The paper tested five different ways to ask the librarian to format the answer.

The Winners: Formats that look like natural sentences with tags embedded in them (like [Name | Type] or <Name>Type</Name>) worked best. It's like asking the librarian to write a story where the names are naturally highlighted.
The Losers: Formats that required the librarian to do complex math, like counting exactly which character number a word starts at (e.g., "Start at character 45, end at 50"), were a disaster. The librarian got confused by the rigid math and made many mistakes.
The Lesson: If you want the AI to work well, ask it to write in a natural, structured way, not in a spreadsheet format.

3. Is the Librarian Cheating? (Memorization vs. Learning)

Since the librarian has read so much, maybe they are just cheating by remembering the answers from their training data?

The Test: The researchers tried to trick the librarian. They replaced the standard names (like "Person" or "Location") with random symbols (like "A" or "B") and gave them new instructions.
The Result: The librarian didn't crash. They still figured out the task. This proves the librarian isn't just reciting a memorized list of "John = Person." They are actually learning the concept of what a name is and applying it to new situations, just like a human would.

4. The "Side Effect" of Training

Usually, when you train a specialized robot to do one thing, it forgets how to do everything else. If you teach a calculator to do math, it might forget how to tell a joke.

The Surprise: When the researchers taught the giant librarian to find names, it didn't forget how to do other things. In fact, on a reading comprehension test (where you have to find answers in a text), the librarian actually got better.
Why? Because finding names is a huge part of understanding a story. By practicing finding names, the librarian got sharper at understanding the whole text.

5. Where the Librarian Still Struggles

The librarian is amazing at general topics (like news or history), but they stumble a bit in very specialized fields, like biology or medicine.

The Gap: In the "Biomedical" dataset, the specialized robot (trained specifically on medical terms) still beat the librarian. The librarian knows general knowledge well but lacks the deep, specific "medical school" training that the specialized robot has.

The Bottom Line

This paper shows that we don't necessarily need to build a new, specialized robot for every single task anymore. We can take a powerful, general-purpose AI, give it a clear set of instructions and a good format, and it can do the job of finding names just as well as the old methods. Plus, it keeps its ability to do other smart tasks, making it a very friendly and flexible tool for everyone.

Technical Summary: Assessment of Generative Named Entity Recognition in the Era of Large Language Models

Problem Statement

Named Entity Recognition (NER) has traditionally been framed as a sequence labeling task, predominantly solved by auto-encoding pre-trained models (e.g., BERT) that assign tags to tokens. With the advent of Large Language Models (LLMs), NER is shifting toward a generative paradigm where models sequentially generate text containing both the original sentence and the identified entities. While preliminary studies have explored zero-shot or few-shot NER using closed-source models (e.g., ChatGPT), there is a lack of systematic investigation into:

The performance gap between fine-tuned open-source LLMs and traditional state-of-the-art (SOTA) NER models on both flat and nested tasks.
The impact of diverse output formats on generative NER performance.
Whether LLMs solve NER through genuine entity learning or merely by memorizing entity-label correlations from training data.
Whether instruction tuning for NER degrades the general capabilities of LLMs.

Methodology

The authors conducted a systematic evaluation across eight open-source LLMs (families: LLaMA, Qwen, Granite, Gemma) with parameter scales ranging from 3B to 9B. The experiments utilized four standard datasets:

Flat NER: CoNLL2003 and OntoNotes5.0.
Nested NER: ACE2005 and GENIA (biomedical domain).

Experimental Design:

Fine-tuning Strategy: All models were fine-tuned using Low-Rank Adaptation (LoRA) with a rank of 256, compared against full fine-tuning and traditional baselines (e.g., BERT-Tagger, W2NER, BERT-MRC).
Output Formats: Five distinct formats were evaluated to determine their influence on performance:
1. Inline Bracketed: [Entity | Label]
2. Inline XML: <Label>Entity</Label>
3. Category-grouped JSON: Grouped by label keys.
4. Occurrence-based JSON: Includes occurrence indices.
5. Offset-based JSON: Requires precise character start/end positions.
Memorization Analysis: To distinguish between learning and memorization, the authors replaced standard text labels (e.g., "PER", "LOC") with arbitrary symbols (e.g., "A", "B") in the CoNLL2003 dataset. They tested models under two conditions: with explanations for the symbols (CoNLL2003-SE) and without (CoNLL2003-SO).
Capability Preservation: The authors evaluated the models on general benchmarks (HellaSwag, DROP, GSM8K, MMLU, TruthfulQA) before and after NER instruction tuning to assess catastrophic forgetting or capability transfer.

Key Results

1. Performance and Output Formats

Format Sensitivity: The output format has a decisive impact. Inline Bracketed and Inline XML formats achieved the best performance (Average F1: 91.91 and 91.71 on flat NER, respectively). These formats, which embed entities directly into the sentence structure, align well with the generative nature of LLMs.
JSON Limitations: JSON-based formats, particularly those requiring precise character offsets, resulted in significant performance drops (e.g., Offset-based JSON averaged 53.91 F1 on flat NER), indicating that LLMs struggle with fine-grained positional alignment during generation.
Comparison with Baselines: With parameter-efficient fine-tuning and optimal formats (Inline Bracketed/XML), open-source LLMs (e.g., LLaMA3.1-8B, Gemma2-9B) achieved performance competitive with or superior to traditional encoder-based models (e.g., BERT-MRC, GNN-SL) on flat NER tasks. They also surpassed closed-source GPT-3 models using in-context learning.
Domain Gap: While LLMs perform well in general domains, they lag behind specialized traditional models in low-resource domains like biomedicine (GENIA), where the best LLM result was 4.02 F1 points lower than the best specialized model.

2. Memorization vs. Learning

Few-shot vs. Fine-tuning: Without fine-tuning, 5-shot prompting yielded moderate F1 scores significantly lower than fine-tuned models, suggesting that pre-training exposure alone is insufficient for high-performance NER.
Symbol Replacement: Replacing text labels with arbitrary symbols resulted in negligible F1 changes (average $\Delta$ F1 of -0.18 with explanations and +0.01 without). This indicates that LLMs do not rely on memorizing specific entity-label pairs but rather learn the underlying task of mapping text spans to abstract label concepts through instruction tuning.

3. Error Analysis

Traditional Models: Predominantly suffer from Omitted Mentions (45.5%), reflecting a conservative prediction strategy.
Generative LLMs: Exhibit a lower omission rate but a higher rate of Wrong Types (38.2%) and Completely-O errors (hallucinating entities not in the text). This suggests LLMs are more aggressive in extraction but prone to misclassifying types due to conflicts between their broad pre-trained knowledge and specific dataset annotation guidelines (e.g., misclassifying "Permian" as a location instead of miscellaneous).

4. Preservation of General Capabilities

Instruction tuning for NER had minimal negative impact on general capabilities. Performance on reasoning and knowledge benchmarks remained stable or slightly fluctuated.
Notably, performance on the DROP benchmark (reading comprehension requiring discrete reasoning over paragraphs) improved significantly by 25.50 to 45.32 F1 points. The authors attribute this to the enhanced ability of the model to detect and extract entity spans, which are often the answers in DROP.

Significance and Claims

The paper claims that generative NER using open-source LLMs is a promising and user-friendly alternative to traditional sequence labeling methods. The key contributions are:

Viability: Open-source LLMs, when fine-tuned with parameter-efficient methods and structured output formats, can match or exceed traditional SOTA models in general domains.
Mechanism: The success of generative NER stems from the models' instruction-following and generative capabilities rather than data memorization.
Safety: Fine-tuning for NER does not degrade general intelligence; in some cases (like DROP), it enhances performance by improving entity understanding.
Practicality: The natural language interface of generative NER lowers the barrier for non-experts, suggesting a potential shift in how NER systems are deployed and interacted with.

The authors conclude that while challenges remain in low-resource and nested domains, the generative paradigm represents a robust evolution in NER technology.