NERdME: a Named Entity Recognition Dataset for Indexing Research Artifacts in Code Repositories

The paper introduces NERdME, a new dataset of 200 manually annotated README files containing over 10,000 labeled spans across 10 entity types, designed to bridge the gap in scholarly information extraction by enabling the automatic indexing of implementation-level research artifacts in code repositories.

The Gist

Imagine the world of scientific research as a massive, bustling library. For years, librarians (and computer programs) have been very good at cataloging the books (the published scientific papers). They know how to find the title, the author, and the main topic of a book.

But there's a problem: the backpacks and toolkits that scientists use to actually do the research are often left in a messy pile in the corner. These toolkits are the code repositories (like GitHub) where the actual software, data, and instructions live.

The problem is that the instructions for these toolkits are written in README files. Think of a README file as a sticky note taped to a toolbox. It's written in free-form text (Markdown), full of jargon, links, and casual descriptions. It's messy, unstructured, and hard for a computer to read. If you ask a computer, "Where is the dataset for this project?" it might get confused because the answer is buried in a sentence like, "We used the 'COCO' dataset, which you can grab here."

Enter NERdME.

What is NERdME?

NERdME is like a super-smart training manual for computers, teaching them how to read those messy sticky notes (README files) and find the important ingredients inside.

The authors created a dataset of 200 of these README files and manually highlighted (annotated) over 10,000 specific pieces of information. They taught the computer to spot two very different types of "ingredients":

1. The "Paper" Ingredients: Things you'd find in a formal book, like the name of a Conference, a Publication, or a Workshop.
2. The "Code" Ingredients: Things you'd only find in the toolbox, like the Software used, the Programming Language (e.g., Python), the License (who owns the code), or the Dataset used.

Before this, computers were great at finding "Paper" ingredients but terrible at finding "Code" ingredients, and vice versa. NERdME is the first to teach them to find both at the same time.

How Did They Build It?

Imagine you are trying to teach a robot to identify fruits in a chaotic fruit basket.

• The Collection: They went to a giant online marketplace (GitHub) and picked 200 baskets that were known to have good fruit (projects linked to real scientific papers).
• The Annotation: They hired three human experts for every basket. These experts read the sticky notes and circled every mention of a "Dataset" or "Software."
• The Consensus: If two out of three experts circled the same word, the robot learned that it was definitely a "Dataset." If they disagreed, the robot ignored it to avoid confusion. This ensured the training data was high-quality.

What Did They Discover?

They tested the robot using two methods:

1. The "Guessing" Robot (Zero-shot LLMs): A powerful AI that tries to guess the answers without any specific training on these files. It's like asking a smart person who has never seen a toolbox to guess what's inside just by looking at a photo. It was okay, but it missed a lot of details.
2. The "Trained" Robot (Fine-tuned Transformers): An AI that studied the 200 annotated files specifically. It learned the patterns.

The Results:

• The Trained Robot was much better. It learned that "Python" usually means a programming language, while "COCO" usually means a dataset.
• The "Long Tail" Problem: The robot was great at finding common things (like "Software" or "Datasets") because they appeared often. But it struggled with rare things (like "Workshops" or "Ontologies") because they appeared very few times. This is like a chef who is a master at making pizza but has never tried to make a soufflé because they've never seen the recipe before.
• The Boundary Challenge: It was hard for the robot to know exactly where a name starts and ends. For example, is the name "TensorFlow" or just "Tensor"? The dataset showed that defining the exact edges of these names is tricky.

Why Does This Matter? (The Downstream Test)

To prove this wasn't just a game, they ran a treasure hunt.
They took the "Dataset" names the robot found in the READMEs and tried to match them to real records in a global database called Zenodo.

• The Result: The robot was surprisingly good at this! It could take a messy mention like "the COCO dataset" and correctly link it to the official COCO record in the database.
• The Analogy: It's like finding a handwritten note that says "The big red apple from the orchard down the road" and successfully finding the exact apple in a massive warehouse catalog.

The Big Picture

NERdME is a bridge. It connects the formal world of scientific papers with the messy, practical world of software code.

By teaching computers to understand the "sticky notes" on code repositories, we can:

• Automatically find the tools scientists used.
• Link papers to the actual data and code they rely on.
• Make scientific research more reproducible (easier for others to repeat the experiment) because the "how-to" instructions are finally being understood by machines.

In short, NERdME is the dictionary that finally helps computers understand the messy, real-world instructions scientists leave behind, turning a pile of chaotic sticky notes into a well-organized library of research tools.

Technical Summary

Here is a detailed technical summary of the paper "NERdME: a Named Entity Recognition Dataset for Indexing Research Artifacts in Code Repositories."

1. Problem Statement

Current Scholarly Information Extraction (SIE) research is heavily biased toward scientific papers (e.g., abstracts, full texts). While datasets like SciERC and SciREX exist, they fail to capture implementation-level metadata found in software repositories (e.g., GitHub).

• The Gap: Critical research artifacts—such as specific datasets, software dependencies, licenses, and programming languages—are often documented in README files.
• The Challenge: README files are written in free-form Markdown. Unlike structured academic papers, they lack explicit semantic cues, making automatic extraction difficult.
• The Limitation of Existing Work: Existing datasets either focus solely on paper-level entities (tasks, metrics) or implementation-level entities (software, languages) but not both. Furthermore, existing README-focused tools (like Hidden Entity) focus on URL classification rather than providing span-level annotations for named entities, preventing the linking of conceptual entities across scientific writing and code repositories.

2. Methodology: Dataset Construction (NERdME)

The authors introduced NERdME, a manually curated Named Entity Recognition (NER) dataset designed to bridge the gap between paper-level and implementation-level research artifacts.

• Data Source: 200 GitHub README files associated with data science papers from "Papers with Code" (PwC). These were selected from an initial pool of 2,600 repositories to ensure rich scholarly context.
• Entity Schema: The dataset defines 10 entity types based on the NFDI4DS ontology, covering two distinct categories:
  • Paper-Level: CONFERENCE, DATASET, EVALUATION METRIC, PUBLICATION, WORKSHOP, ONTOLOGY.
  • Implementation-Level: SOFTWARE, PROGRAMMING LANGUAGE, LICENSE, PROJECT.
• Annotation Process:
  • Crowdsourcing: Three independent annotators (CS background) labeled each file using the INCEpTION tool.
  • Curation Rules: To ensure high quality, only spans agreed upon by at least two annotators were retained. A strict sentence-level consistency rule was applied: if annotators disagreed on the type of an entity within a specific sentence, all spans of that type in that sentence were removed to prevent label noise.
  • Handling Complexity: The annotation accounts for nested entities (e.g., software names inside publication titles) and entities embedded in URLs or code snippets.
• Dataset Statistics:
  • Total Spans: 10,691 labeled spans (4,328 unique).
  • Distribution: Highly imbalanced (Long-tail distribution). Common types include SOFTWARE (4,332 spans) and DATASET (1,596 spans), while rare types include WORKSHOP (79 spans) and ONTOLOGY (112 spans).
  • Split: 70% Training, 10% Validation, 20% Testing.
• Quality Control: Inter-annotator agreement (Krippendorff's alpha) was 0.70 ± 0.19, indicating substantial agreement.

3. Key Contributions

1. First Joint Dataset: NERdME is the first resource to jointly annotate paper-level and implementation-level entities within the same document type (READMEs).
2. Benchmarking: It provides a new benchmark for evaluating NER models on free-form technical documentation, revealing performance gaps in current State-of-the-Art (SOTA) models.
3. Downstream Applicability: The authors demonstrated the dataset's utility in Entity Linking (EL), showing that extracted entities can be successfully linked to external scholarly records (Zenodo).
4. Linguistic Analysis: The paper provides a comparative analysis showing that implementation-level entities are shorter, more standardized, and have higher language-model perplexity compared to the more descriptive, longer paper-level entities.

4. Experimental Results

A. Named Entity Recognition (NER) Performance

The authors evaluated Zero-shot Large Language Models (LLMs) (e.g., LLaMA3.1, GPT-4o-mini) against Fine-tuned Transformers (SciBERT, RoBERTa).

• Supervision Impact: Fine-tuned models significantly outperformed zero-shot LLMs, particularly for common entities.
  • Example (Exact Match F1): SOFTWARE improved from 23.12% (Zero-shot) to 72.46% (Fine-tuned).
  • Example (Exact Match F1): DATASET improved from 45.30% to 63.94%.
• Long-Tail Challenge: Rare entities (e.g., WORKSHOP, ONTOLOGY) remained difficult for both approaches, with WORKSHOP dropping to 0.00% Exact Match for fine-tuned models due to extreme data scarcity.
• Boundary Alignment: There was a significant drop in performance from Partial Match to Exact Match F1 scores (e.g., CONFERENCE dropped from 77.38% to 47.63% for zero-shot). This indicates that while models can identify the type of entity, they struggle with precise span boundaries in free-form text.

B. Downstream Task: Entity Linking (EL)

An experiment linked DATASET mentions from NERdME to canonical records in Zenodo.

• Methods: Compared Fuzzy Matching (token-set ratio) vs. Semantic Similarity (using all-MiniLM-L6-v2).
• Results: Semantic similarity vastly outperformed fuzzy matching.
  • F1 Score: Semantic (47.75%) vs. Fuzzy (17.67%).
  • MRR (Mean Reciprocal Rank): Semantic (37.69%) vs. Fuzzy (15.27%).
• Conclusion: The annotated spans contain sufficient semantic and contextual cues to support reliable artifact discovery and metadata integration.

5. Significance and Future Directions

• Bridging the Gap: NERdME enables the automated extraction of critical metadata that is currently "invisible" to SIE systems, facilitating better reproducibility and resource discovery in open science.
• Realistic Benchmark: The dataset's long-tail distribution and free-form nature provide a realistic testbed for evaluating model robustness, moving beyond the clean, structured data of traditional academic papers.
• Future Work: The dataset opens avenues for research in:
  • Scholarly entity coreference resolution.
  • Information extraction from heterogeneous research artifacts.
  • Adapting LLMs for domain-specific, structure-aware extraction.

In summary, NERdME addresses a critical blind spot in scholarly information extraction by providing the first high-quality, span-level annotated dataset for GitHub READMEs, proving that implementation-level metadata can be effectively extracted and linked to support the broader research ecosystem.