CodeNER: Code Prompting for Named Entity Recognition

1. Problem Statement

Named Entity Recognition (NER) is traditionally a sequential labeling task (e.g., using the BIO schema: Begin, Inside, Outside) where every token in a text must be assigned a specific label. While Large Language Models (LLMs) have shown success in zero-shot and few-shot NER via text-based prompting, they face a fundamental architectural mismatch:

• The Mismatch: LLMs typically operate on a "text-in, text-out" schema, generating natural language responses. However, NER requires a "text-in, span-out" or "token-level" output, demanding precise sequential alignment and boundary detection.
• Limitations of Text Prompts: Existing text-based prompting methods often struggle to capture the strict sequential nature of the BIO schema. They frequently fail to:
  • Correctly identify entity boundaries (e.g., treating a URL as multiple tokens instead of one).
  • Avoid duplicate labeling for the same word.
  • Handle repeated entities in a text consistently.
  • Capture long-range dependencies required for accurate span prediction without explicit token-level supervision.

2. Methodology: CodeNER

The authors propose CodeNER, a novel prompting framework that embeds NER instructions within structured programming code (specifically Python and C++) rather than natural language.

Core Mechanism:

• Structured Code Prompts: Instead of asking the LLM to "list entities," CodeNER provides a Python script skeleton. This script includes:
  • A dictionary mapping entity labels to numeric IDs.
  • A function find_ner_tags containing a for-loop that iterates through the input sentence token-by-token.
  • Logic that forces the model to process the input sequentially, mimicking the BIO tagging process.
• Leveraging LLM Capabilities: The method exploits the LLM's inherent ability to understand programming language structures. Code provides a rigid, unambiguous structure that naturally enforces sequential processing, bridging the gap between the LLM's generative nature and the sequential requirements of NER.
• Key Components of the Prompt:
  • sentence: The input text tokenized into a list.
  • ner_tag_labels: A dictionary defining the schema (e.g., {'PER': 1, 'LOC': 2}).
  • ner_word_dictionary: A dynamic structure for the model to define and populate with entity mappings.
  • Sequential Logic: The for loop ensures the model considers the context of the previous token when labeling the current one, effectively simulating the sequential dependency found in traditional CRF-based models.

Variations Explored:

• Languages: Tested with Python and C++ (Java was found ineffective due to lack of pre-training knowledge in some models).
• Enhancements: Combined with Chain-of-Thought (CoT) reasoning ("Let's think step by step") to further improve performance.
• Settings: Evaluated in Zero-shot, One-shot, and Few-shot scenarios.

3. Key Contributions

1. Novel Prompting Paradigm: Introduced CodeNER, the first method to systematically leverage code-based prompting to enforce the sequential BIO schema in LLMs for NER, addressing the "text-in-text-out" vs. "text-in-span-out" gap.
2. Explicit Sequential Guidance: Demonstrated that embedding a for-loop in the prompt forces the LLM to process tokens sequentially, significantly reducing errors related to overlapping labels and boundary misidentification.
3. Comprehensive Evaluation: Conducted extensive experiments across 10 benchmarks covering 5 languages (English, Arabic, Finnish, Danish, German) and both closed (GPT-4, GPT-4 Turbo) and open-source models (Llama-3, Phi-3).
4. Analysis of Limitations: Identified that while CodeNER excels at token-level precision, it may underperform on datasets requiring the labeling of long text spans containing function words (e.g., "Plot" or "Quote" in movie reviews), where natural language prompts focusing on semantic meaning are superior.

4. Experimental Results

The results demonstrate that CodeNER consistently outperforms vanilla text-based prompting (Vanilla) and other code-based baselines (GoLLIE, GNER).

• Performance Gains:
  • GPT-4: CodeNER achieved an average 2.41 point improvement in Macro-F1 over Vanilla. Notable gains included +18.3 on the Finnish dataset (FIN) and +9.48 on MIT Restaurant.
  • GPT-4 Turbo: Achieved an average 3.07 point improvement.
  • Open Models (Llama-3/Phi-3): The improvements were even more dramatic for open models, with CodeNER outperforming baselines by significant margins (e.g., +19.54 points on Phi-3 for the FIN dataset).
• Label-Specific Improvements: CodeNER showed particular strength in identifying MISC (Miscellaneous) entities and handling complex boundaries, though it occasionally struggled with ORG (Organization) labels in specific datasets like CoNLL03.
• Code vs. Natural Language: CodeNER significantly outperformed GoLLIE and GNER (which use partial code or annotation guidelines), proving that the structure of the code (the loop) is critical, not just the presence of code syntax.
• Chain-of-Thought: Combining CodeNER with CoT instructions yielded further performance boosts in zero-shot settings.
• Programming Language Choice: Python and C++ yielded comparable results on CoNLL03, suggesting dataset characteristics matter more than the specific language, provided the model understands the syntax.

5. Significance and Implications

• Bridging the Gap: CodeNER provides a robust solution for applying LLMs to structured tasks like NER without requiring fine-tuning. It effectively translates the rigid requirements of NER into a format LLMs process naturally (code execution).
• Zero-Shot Efficiency: The method is highly effective in low-resource and zero-shot scenarios, where explicit token-label supervision is absent. It allows open-source models to compete with or surpass closed models in specific NER tasks.
• Error Reduction: By enforcing a token-by-token processing loop, CodeNER eliminates common LLM hallucinations in NER, such as duplicate labeling or failure to split/merge entities correctly.
• Future Directions: The authors suggest that while code prompting is powerful, future work should address the handling of long spans with function words (where natural language excels) and potentially pre-train models on this specific code-structured NER task to further enhance capabilities.

In conclusion, CodeNER represents a significant shift in how LLMs are prompted for structured extraction tasks, demonstrating that leveraging the structural rigidity of programming languages can unlock superior reasoning capabilities for tasks requiring precise sequential alignment.