💬 NLP

Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition

This paper demonstrates that scaling in-context learning to hundreds of examples enables large language models to match or surpass fully supervised BERT performance in Named Entity Recognition, while also serving as an effective data annotation framework that significantly improves low-resource NER models.

Original authors: Qi Zhang, Fangping Lan, Cornelia Caragea, Longin Jan Latecki, Eduard Dragut

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qi Zhang, Fangping Lan, Cornelia Caragea, Longin Jan Latecki, Eduard Dragut

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to Read by Showing It Examples

Imagine you have a very smart robot (a Large Language Model, or LLM) that knows how to read, but it's never been taught specifically how to find names of people, places, or organizations in a sentence. Usually, to teach a robot this skill, you have to hire a team of human teachers to write thousands of examples, label them perfectly, and then spend weeks "training" the robot on those examples. This is expensive and slow.

This paper proposes a different way: Show, Don't Just Tell.

Instead of retraining the robot's brain, the researchers simply show the robot hundreds of examples right before it reads a new sentence. This is called In-Context Learning (ICL). The paper asks: What happens if we show the robot not just 5 examples, but 100, 200, or even 500 examples all at once?

The Main Discovery: More Examples = Smarter Robot

The researchers found a surprising trend: The more examples they showed the robot, the better it got.

The Old Way (Few-Shot): Showing the robot 5 examples was like giving a student a quick cheat sheet. It helped, but the student still made mistakes.
The New Way (Many-Shot): Showing the robot 100+ examples was like giving the student a whole textbook chapter to study right before the test. The robot didn't just "guess" better; it learned the patterns so well that it performed as well as, or even better than, robots that had been trained for weeks on thousands of human-labeled examples.

The Analogy: Think of the robot as a new employee.

Fine-Tuning (The Old Way): You hire a trainer to spend a month teaching the employee every rule of the job.
Many-Shot ICL (The New Way): You sit the employee down, hand them a stack of 100 completed forms, and say, "Look at how these were done, then fill out this new one." The employee figures it out instantly without needing a month of training.

The Problem: The Robot is Slow and Expensive to Use

While the "Many-Shot" method worked great for getting answers, it had a catch. Every time you wanted the robot to read a new sentence, you had to feed it those 100+ examples again.

The Analogy: It's like asking a librarian to find a book. If you have to hand the librarian a 100-page history book every single time you ask for a book, it takes a long time and costs a lot of money. You can't do this in real-time for thousands of documents.

The Solution: The "Annotation Factory" (ICA)

To solve the speed and cost problem, the researchers built a framework called In-Context Annotation (ICA).

Instead of using the robot to read documents in real-time, they used it as an offline factory worker.

The Setup: They gave the robot a small "seed" of about 100 human-labeled examples (the "seed" is like the blueprint).
The Factory: The robot used those 100 seeds to read through thousands of unlabeled documents and label them itself.
The Quality Control: Sometimes the robot makes mistakes. The researchers added a "Quality Control" step where the robot checks its own work, fixes errors, and votes on the best answer (like a committee of robots reviewing each other).
The Result: They now have a massive pile of high-quality, labeled data.
The Deployment: They take this new data and train a small, fast, cheap robot (like BERT) on it.

The Analogy:

Step 1: You hire a highly paid, slow expert (the big LLM) to label 2,000 documents. You give them a few examples to start.
Step 2: The expert does the work, double-checks their own mistakes, and hands you a perfect dataset.
Step 3: You fire the expensive expert. You take that perfect dataset and teach a fast, cheap intern (the small BERT model) how to do the job.
Step 4: Now, when you need to read a new document, you use the fast, cheap intern. It's instant and costs almost nothing.

Key Findings in Plain English

Scaling Works: The more examples you show the big robot, the smarter it gets. Around 100 examples is the "sweet spot" where it becomes incredibly accurate.
Human Effort is Low: You only need about 100 human-labeled examples to get the whole system working. This is a tiny amount of work compared to the thousands usually required.
Better than the Best: The small robot trained on the "factory-made" data performed about 10% better than the current best methods for low-resource tasks.
Refinement Matters: The "Quality Control" steps (where the robot fixes its own errors) were crucial. Specifically, a method called "Error-Aware Refinement" (where the robot is taught to look for specific types of mistakes like "missing names" or "wrong names") made the biggest difference.

Summary

The paper shows that we don't need to spend months training robots on massive datasets. Instead, we can use a big, smart robot as a temporary teacher. By showing it hundreds of examples, it can generate high-quality training data for a small, fast robot. This small robot then becomes the one we actually use in the real world, giving us high performance with very little human effort and low cost.

Technical Summary: Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition

Problem Statement

Named Entity Recognition (NER) is a fundamental information extraction task traditionally reliant on fully supervised models (e.g., fine-tuned BERT) that require large volumes of human-labeled data. While Large Language Models (LLMs) offer a promising alternative via In-Context Learning (ICL) without parameter updates, prior research indicates that LLMs in standard few-shot settings often underperform compared to supervised baselines. Furthermore, existing ICL studies for NER have primarily focused on few-shot scenarios (typically <100 examples), leaving the potential of scaling to hundreds of demonstrations ("many-shot ICL") largely unexplored. Additionally, using LLMs for real-time inference is often computationally expensive and slow, limiting their practical deployment in resource-constrained environments.

Methodology

1. Many-Shot In-Context Learning (ICL) for NER

The authors formulate NER as a conditional generation problem where the model generates an XML-style annotated output (e.g., <entity type="Location">Canada</entity>). This format explicitly marks entity boundaries and types, improving boundary sensitivity.

Scaling Experiments: The study evaluates seven LLMs (including Qwen-2.5, LLaMA-3.1, GPT-4o, and DeepSeekV3) across four domains (MIT-Restaurant, MIT-Movie, CoNLL2003, WNUT 2017).
Demonstration Strategies: Two strategies for selecting in-context examples are compared:
- Random Sampling: A fixed set of $k$ examples is reused for all test inputs.
- Retrieval-Based: Top- $k$ similar examples are dynamically retrieved for each test input using BM25.
Observation: Performance consistently improves as the number of demonstrations increases from 0 to 500, with retrieval-based methods plateauing earlier (around 100 examples) and random sampling showing steady gains up to 500 examples.

2. The In-Context Annotation (ICA) Framework

To address the inference cost of LLMs, the authors propose using many-shot ICL not as a direct inference engine, but as an offline data annotation tool to train efficient small models (e.g., BERT).

Workflow:
1. Seed Data: A small set of human-labeled examples (e.g., 100) is used as fixed in-context demonstrations.
2. Annotation: An LLM annotates a large corpus of unlabeled data (e.g., 2,000 sentences) using these demonstrations.
3. Refinement: To enhance quality, three strategies are applied to low-confidence samples:
  - Self-Consistency: Multiple predictions with varied demonstration orders are aggregated via majority voting.
  - Self-Correction: The LLM is prompted to review and revise its own output.
  - Error-Aware Refinement (EAR): A modular approach using three distinct prompts to sequentially correct specific error types: spurious entities, missing entities, and incorrect entity types.
4. Downstream Training: The refined, high-quality labeled data is used to fine-tune a BERT-based NER model.

3. Experimental Setup

Datasets: The ICA framework is evaluated on the CrossNER benchmark (AI, Literature, Music, Politics, Science) under low-resource settings (100–200 human-labeled seeds).
Baselines: Comparisons include traditional data augmentation (DAGA, NERDA), LLM-based augmentation (ProgGen), and state-of-the-art low-resource/cross-domain NER methods (e.g., DTrans-MPrompt, GLiNER).

Key Results

Many-Shot ICL Performance

Scaling Effect: Increasing the number of in-context examples leads to monotonic performance gains. With approximately 100–500 examples, LLMs achieve performance comparable to or exceeding fully supervised BERT models trained on thousands of examples.
Efficiency: Random sampling with hundreds of examples yields competitive results, approaching the performance of more expensive retrieval-based methods while requiring less annotation effort.

ICA Framework Performance

Superiority over SOTA: On the CrossNER benchmark, the ICA framework (specifically with EAR) achieves an average absolute F1 improvement of approximately 10 points over previous state-of-the-art low-resource approaches.
Comparison to Direct ICL: Converting LLM predictions into training data for BERT outperforms direct many-shot ICL inference (ICL-MS) by over 5 F1 points, demonstrating the value of distilling LLM knowledge into efficient models.
Refinement Impact: The Error-Aware Refinement (EAR) strategy consistently outperforms other variants, adding an average of 1.79 F1 points over standard ICA by explicitly addressing common annotation errors.
Data Efficiency: The method achieves high performance using only ~100 human-labeled seed examples to generate ~2,000 annotated samples, significantly reducing human annotation costs.

Analysis of Requirements

Annotation Scale: Performance saturates after annotating approximately 1,500–2,500 sentences; additional data yields diminishing returns.
Seed Size: Approximately 75–100 human-labeled examples are sufficient to guide effective ICA; increasing beyond this yields marginal gains.
Model Choice: Larger models (e.g., GPT-4o, Qwen-72B) produce higher-quality annotations than smaller models (e.g., Qwen-7B), though open-source models scaled up can be competitive.

Significance and Contributions

The paper makes four primary contributions:

Empirical Study: It provides the first systematic investigation of many-shot ICL for NER, demonstrating consistent performance gains as demonstration counts increase across multiple datasets and models.
Framework Proposal: It introduces the In-Context Annotation (ICA) framework, which leverages a modest number of human labels to generate high-quality training data for efficient downstream models, decoupling the cost of LLM inference from deployment.
Experimental Validation: Experiments show that ICA significantly outperforms existing SOTA low-resource NER methods (by ~10 F1 points) and traditional data augmentation techniques.
Scalability Analysis: The study offers practical insights into the trade-offs between seed data size, annotation volume, and model selection, establishing that many-shot ICL is a viable, cost-effective paradigm for building NER datasets in resource-constrained domains.

The authors conclude that while limitations exist regarding the need for seed labels and computational overhead during the annotation phase, the ICA framework offers a practical solution for reducing annotation costs and improving data efficiency in NER tasks.