💬 NLP

SciDef: Datasets and Tools for Automated Definition Extraction from Scientific Literature with LLMs

The paper introduces SciDef, a comprehensive resource suite comprising the DefExtra benchmark, DefSim similarity judgments, and an open LLM-based pipeline to advance automated scientific definition extraction, while identifying relevance-aware filtering as the primary challenge for fully automatic systems.

Original authors: Filip Kučera, Christoph Mandl, Isao Echizen, Radu Timofte, Timo Spinde

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Filip Kučera, Christoph Mandl, Isao Echizen, Radu Timofte, Timo Spinde

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a giant, universal dictionary for scientists. The problem is that every scientist writes their own rules. One researcher might define "media bias" as "slanted news," while another calls it "internal bias in articles." They are talking about the same thing, but using different words. This makes it incredibly hard to compare their work or build tools that understand science as a whole.

The authors of this paper, SciDef, are like a team of librarians and engineers who decided to fix this mess. They built a toolkit to help computers automatically find, read, and compare these scientific definitions.

Here is how their toolkit works, broken down into three main parts:

1. The "Gold Standard" Dictionary (DefExtra)

Think of DefExtra as a massive, carefully curated flashcard deck.

What it is: A collection of 268 real definitions that humans manually pulled out of 75 different academic papers.
The Catch: They didn't just grab any sentence that looked like a definition. They filtered out vague descriptions and only kept the "author-stated" definitions—the ones the scientists actually wrote down to explain their terms.
Why it matters: Before this, if you wanted to test a computer program to see if it could find definitions, you had to make your own test. Now, everyone can use this same "Gold Standard" deck to see who is doing the best job.

2. The "Taste Test" (DefSim)

Finding a definition is easy; knowing if two definitions mean the same thing is hard.

The Problem: If a computer says "Media bias is bad reporting" and the human answer is "Media bias is slanted coverage," are they the same? A simple computer check might say "No, the words are different."
The Solution: DefSim is a small set of 60 pairs of definitions where humans gave them a "similarity score" from 1 to 5 (like rating two songs on how similar they sound).
The Goal: This acts as a "taste test" to see if the computer's math (its similarity metric) agrees with human intuition. The paper found that a specific type of AI logic (called NLI) is very good at passing this taste test.

3. The Robot Librarian (SciDef Pipeline)

This is the actual machine that does the work.

How it works: Imagine a robot that takes a PDF of a scientific paper, chops it up into small chunks (like slicing a loaf of bread), and then asks a Large Language Model (LLM) to "Find the definitions in this slice."
The Experiment: The authors tested 16 different AI models using different ways of asking the question (prompts) and different ways of slicing the paper (chunking).
The Results:
- The "Catch-All" Strategy: Some settings made the robot find almost every definition (86.4% of them!). However, it also found a lot of junk. It was like a metal detector that beeps for every piece of metal, including bottle caps and nails, not just gold coins.
- The "Smart" Strategy: The best-performing setup used a special optimization technique (DSPy). It didn't find every single definition, but the ones it found were high-quality and relevant. It got a score of 0.397, which was the highest among the group.
- The Bottleneck: The main problem isn't finding the definitions; it's filtering. The robots are great at casting a wide net, but they struggle to throw away the irrelevant "noise" without human help.

The Big Takeaway

The authors aren't claiming that robots can now perfectly read and understand science on their own. Instead, they are saying:

We have the tools: They gave the world the data (DefExtra, DefSim) and the robot (SciDef) to keep improving.
Robots are good at "High Recall": They can find almost everything you're looking for, but they bring back a lot of clutter.
Humans are still needed: Because the robots over-generate (find too much junk), the next step for researchers is to build better filters to separate the "gold coins" from the "bottle caps."

In short, SciDef is a starter kit for anyone who wants to teach computers how to understand the language of science, complete with a test deck, a grading rubric, and a working prototype.

Technical Summary: SciDef – Datasets and Tools for Automated Definition Extraction from Scientific Literature with LLMs

Problem Statement

Scientific concepts are frequently defined inconsistently across academic papers, creating barriers to comparing findings, reusing terminology, and constructing reliable downstream resources. As publication volumes increase, manually identifying and consolidating definitions for specific keywords becomes infeasible. While Large Language Models (LLMs) show promise in recognizing definitional statements beyond surface-level keyword patterns, reproducible research in this domain is hindered by a lack of public benchmarks for extracted definitions, dedicated datasets for evaluating definition similarity, and reusable infrastructure for comparing extraction pipelines and prompting strategies.

Methodology

The authors introduce SciDef, a comprehensive resource suite comprising two datasets and an open-source LLM-based pipeline. The methodology is structured around three core components:

1. Dataset Construction

DefExtra (Extraction Benchmark): A dataset of 268 human-validated, author-stated definitions extracted from 75 curated academic papers (published 1987–2025), primarily focusing on the domain of media bias. The dataset includes source metadata, context spans, and labels distinguishing between explicit (direct quotes) and implicit (reworded but clear) definitions. It also flags out-of-domain papers to test model robustness. The construction involved a multi-stage process: initial retrieval and annotation based on the TaxoMatic effort, followed by strict manual filtering by two authors to retain only verifiable, author-stated definitions, and finally extension with additional papers annotated by a separate group of 11 annotators.
DefSim (Similarity Benchmark): A validation resource containing 60 definition pairs with human semantic-similarity labels (rated 1–5). The pairs include gold-prediction, gold-gold, and prediction-prediction combinations to cover exact, partial, and non-matching cases. This dataset is designed to validate semantic matching metrics rather than tune them.

2. The SciDef Pipeline

SciDef is a modular, reproducible pipeline for PDF preprocessing, chunking, definition extraction, prompt optimization, and evaluation.

Preprocessing: PDFs are parsed using GROBID.
Chunking: Four strategies are evaluated: section-level, paragraph-level, sentence-level, and a three-sentence sliding window.
Extraction Strategies: The pipeline compares several prompting approaches:
- OneStep: Directly extracts definitions from a chunk.
- MultiStep: First determines if a chunk contains a definition before extracting.
- Few-Shot Variants: Includes training examples (OneStep-FS, MultiStep-FS).
- DSPy Optimization: Implements and optimizes prompts using BootstrapFewShot, BootstrapFewShotWithRandomSearch, and MIPROv2. During optimization, models predict local context spans as an auxiliary grounding signal.
Evaluation Protocol: The pipeline evaluates 16 LLMs (open and proprietary) against DefExtra. Since exact string matching is insufficient for paraphrased definitions, the authors select a pairwise metric based on general semantic similarity benchmarks (STS3k, SICK, MSRP, QQP). They identify Natural Language Inference (NLI) as the superior metric and use DefSim to validate its alignment with human judgments.
Scoring: A bidirectional best-match score is computed. It measures the coverage of human annotations (recall) while penalizing over-generation (precision) by combining semantic similarity (via the NLI metric) with explicit/implicit type agreement.

Key Contributions

The paper makes four primary contributions:

DefExtra: A benchmark of 268 human-extracted definitions with rich metadata, including definition type labels and out-of-domain indicators.
DefSim: A task-specific validation resource of 60 definition-pair similarity judgments to evaluate semantic matching metrics.
SciDef: An open, documented, and versioned LLM pipeline supporting PDF preprocessing, chunking, extraction, prompt optimization, and evaluation.
Validation Experiments: A comprehensive benchmarking of 16 LLMs across multiple prompting strategies and chunking schemes, demonstrating the utility of the released resources for comparing extraction performance.

Results

The authors validated the resources using media bias as a challenging case study due to its inconsistent definitions across disciplines.

Metric Validation: NLI-based metrics outperformed embedding cosine similarity and LLM-as-a-Judge scoring on general semantic benchmarks. On DefSim, the NLI metric showed a strong Pearson correlation ( $\rho = 0.937$ ) with human similarity judgments, and human annotators showed high agreement ( $\alpha = 0.924$ ).
Extraction Performance:
- The strongest set-level configuration (a DSPy-optimized extractor using a Qwen-family model with section-level chunking) achieved a score of 0.397.
- The highest-coverage configuration (a one-step few-shot extractor) matched at least one prediction to 86.4% of gold definitions. However, this configuration significantly over-generated candidate definitions (average 39.13 predictions per paper), resulting in a lower overall score.
- The best-scoring extractors (mostly DSPy-based) demonstrated a better balance between coverage and specificity, with average prediction counts around 5–12 per paper.
Key Finding: While LLMs can recover a high percentage of scientific definitions, high recall often comes at the cost of relevance. The benchmark effectively distinguishes between models that find many plausible candidates and those that identify the specific, relevant definitions.

Significance and Claims

The paper positions SciDef as a reusable benchmark and tooling layer for definition-centric literature analysis. The authors claim that their resources enable researchers to:

Compare new definition extractors against a standardized human-validated benchmark.
Validate definition-matching metrics using task-specific human judgments.
Collect candidate definitions for downstream tasks such as literature reviews, taxonomy construction, ontology engineering, and domain mapping.

The authors modestly conclude that current LLMs serve effectively as high-recall definition discovery tools but should not yet be treated as fully automatic definition selectors. The primary bottleneck for fully automatic extraction remains relevance-aware filtering, as models frequently retrieve many plausible but irrelevant candidates. Consequently, human validation remains necessary for high-stakes applications. The work highlights the need for continued development in filtering mechanisms to bridge the gap between high recall and high precision in scientific definition extraction.