💬 NLP

Measuring research data reuse in scholarly publications using generative artificial intelligence: Open Science Indicator development and preliminary results

PLOS and DataSeer developed a new LLM-based indicator that reveals a 43% research data reuse rate in scholarly publications, demonstrating that generative AI can measure open science impacts at scale and suggesting that the positive effects of data sharing are currently underestimated.

Original authors: Lauren Cadwallader, Iain Hrynaszkiewicz, parth sarin, Tim Vines

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lauren Cadwallader, Iain Hrynaszkiewicz, parth sarin, Tim Vines

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of scientific research as a massive, bustling kitchen. For years, we've been trying to figure out how often chefs (scientists) are actually using ingredients (data) left on the counter by other chefs, rather than just chopping up entirely new vegetables from their own gardens.

This paper is about a new, high-tech "smart camera" that PLOS (a scientific publisher) and DataSeer (a tech company) built to take a snapshot of this kitchen and count exactly how much ingredient-sharing is happening.

Here is the breakdown of their project in simple terms:

The Problem: The "Silent" Reuse

Previously, we tried to count how often scientists reused data in two main ways:

Asking them directly: Surveys showed that about 50% of scientists say, "Yes, I use other people's data."
Looking for formal citations: We looked for official "thank you" notes (citations) in papers. But this method was like trying to count guests at a party by only looking at people wearing nametags. It missed everyone who was just chatting in the corner. This method suggested reuse was very low (around 9–16%).

There was a big gap between what scientists said they did and what the "nametag" method could see.

The Solution: The AI "Super-Reader"

To solve this, the team built a Generative AI (a type of smart computer brain) that acts like a super-attentive reader. Instead of just looking for formal citations, this AI reads the entire text of a scientific paper to understand the story.

Think of it like a detective who doesn't just look for a specific fingerprint, but reads the whole diary to figure out if the person borrowed a book from the library or bought a new one.

How they trained the AI:

They showed the AI thousands of scientific articles and taught it to spot three things:
1. Did the author create new data (like growing new vegetables)?
2. Did the author reuse existing data (like using someone else's leftover ingredients)?
3. Why did they do it? (The "reasoning").
They tested the AI against human experts to make sure it was accurate. The AI got very good at this, correctly identifying reuse about 85% of the time.

The Findings: The Kitchen is Busier Than We Thought

The AI analyzed 4,475 scientific papers published in the first three months of 2024. Here is what it found:

The Big Number: 43% of the papers reused data.
The Comparison: This is much higher than the old "formal citation" method (which saw only ~10%), but it lines up much better with what scientists told us in surveys (which said ~50%).
The Takeaway: The old methods were likely underestimating how helpful and collaborative scientists actually are. The "reuse" is happening, but it's often happening in the text of the paper without a formal "thank you" note attached.

Interesting Patterns

The AI also noticed some differences across the map and the subjects:

By Region: Scientists in Asia-Pacific were the most likely to reuse data (46%), while those in Africa were the least likely (31%). The authors suggest this might be related to how data is shared globally and concerns about who owns the data.
By Subject:
- Physical Sciences and Social Sciences were the most balanced, reusing data almost as often as they created new data.
- Life Sciences (like biology) were the most likely to create brand new data (76%) and the least likely to reuse existing data (37%).

Why This Matters

The authors argue that if we only look at formal citations, we are missing the full picture of how open science is working. By using this AI "smart camera," we can see that data sharing is having a bigger impact than we thought.

Important Note on Limits:
The authors are careful to say this is a "work in progress." They admit that:

It can be tricky to tell the difference between using your own old data versus someone else's.
They only looked at papers from one publisher (PLOS), so the results might look different in other journals.
The AI isn't perfect yet, but it's a huge step forward compared to the old ways of counting.

In short, this paper introduces a new tool that helps us see that scientists are sharing and reusing each other's work much more often than we realized, suggesting that the "open science" movement is working better than our old measuring sticks could show.

1. Problem Statement

The scientific community has long struggled to accurately measure the reuse of research data at scale.

Limitations of Current Methods:
- Surveys: While self-reported surveys indicate high reuse rates (52–54%), they are subjective and not scalable.
- Bibliometrics: Traditional citation analysis relies on formal dataset citations (e.g., DOIs). However, these are inconsistently implemented, and many reused datasets (e.g., genetic sequences) lack persistent identifiers. Bibliometric approaches currently suggest reuse rates of only 9–16%, creating a significant gap between reported behavior and detected metrics.
- Existing NLP: Previous automated attempts (e.g., using SciBERT) have shown promise but often lack the nuance to distinguish between data generation, reuse, and simple reference, or fail to capture the context of how and why data was reused.
The Gap: There is a critical need for an automated, scalable indicator that can detect data reuse in text, identify specific datasets, and understand the context of reuse, thereby providing a more accurate picture of the impact of Open Science policies.

2. Methodology

The authors (PLOS and DataSeer) developed a Generative AI (LLM) pipeline to analyze scholarly articles for data generation and reuse.

A. Indicator Definition

Through literature review and consultation with stakeholders (funders, institutions, researchers), they defined three core requirements for the indicator:

Identify if an article reuses research data.
Identify which specific datasets were reused.
Identify how or why the data was reused (contextual reasoning).

B. Model Development & Architecture

Base Model: The team fine-tuned Qwen3-32B (a large language model).
Training Data: A curated dataset of 1,732 PLOS articles annotated by human experts. Annotations included boolean labels for data generation/reuse and extraction of citations, accession numbers, URLs, and DOIs.
Training Strategy:
- Technique: Low-Rank Adaptation (LoRA) applied to all linear layers (except the final head).
- Optimization: A two-stage training process:
  1. Self-Distillation: Fine-tuning on high-reward completions (≥0.9) from a verification run.
  2. Reinforcement Learning (RL): Using Group Relative Policy Optimization (GRPO) to refine the model.
Reward Function: A multiplicative reward function $r(p, c, y) = f(c) \times e(p, c) \times v(c, y)$ $r (p, c, y) = f (c) \times e (p, c) \times v (c, y)$ was designed to enforce:
1. Format ( $f$ ): Valid JSON structure.
2. Grounding ( $e$ ): No hallucinations; outputs must be supported by the source text.
3. Accuracy ( $v$ ): Agreement with human annotations.
Inference Pipeline:
- Articles were converted to Markdown (using markitdown and Docling).
- Long articles (>25k tokens) were truncated while preserving abstracts and references.
- Pass@3 Strategy: For each article, the model generated three completions. A verifier model (base Qwen3-32B) ranked them, and the highest-ranked completion was selected.

C. Evaluation

The model was evaluated on a held-out test set of 296 articles.

Performance: The model achieved high accuracy on boolean fields (89.8% pass@1 for new data generation; 79.4% for reuse) and strong F1 scores for extracting specific identifiers (e.g., 0.997 F1 for new data citations).
Comparison: It outperformed other tested open models (GLM-5, gpt-oss-120b, Deepseek-R1).

3. Key Results

The model analyzed 4,475 PLOS articles published in Q1 2024.

Overall Reuse Rate: 43% of articles were found to reuse existing data. This is significantly higher than bibliometric estimates (9–16%) and closer to self-reported survey data (>50%).
Data Generation: 59% of articles generated new data.
Overlap: 11% of articles both generated new data and reused existing data.
Non-Data Articles: 10% of articles neither generated nor reused data (a higher rate than previous PLOS indicators, due to stricter definitions excluding pure simulations or systematic reviews).

Regional and Disciplinary Breakdowns

Region:
- Asia-Pacific: Highest reuse rate (46%).
- Africa: Lowest reuse rate (31%), with higher rates of new data generation (66%). The authors suggest this may relate to data sovereignty issues or "helicopter science."
- North America, Latin America, Europe: Reuse rates clustered around 39–42%.
Discipline:
- Social Sciences: Highest reuse rate (50%).
- Physical Sciences: High reuse (47%) but also the highest rate of articles with no data generation or reuse (18%).
- Life Sciences: Highest new data generation (76%) but lowest reuse (37%).

4. Key Contributions

Novel Indicator: Developed the first LLM-based indicator capable of measuring data reuse at scale, moving beyond simple citation counting to semantic analysis of text.
Contextual Understanding: Unlike previous methods, this approach extracts the reasoning behind reuse (how/why), providing richer metadata for impact assessment.
Performance Validation: Demonstrated that fine-tuned LLMs can achieve high accuracy in detecting complex research practices, outperforming traditional bibliometrics and earlier NLP models.
Refined Definitions: Clarified the distinction between "data generation" (moving physical world info to digital) and "reuse," correcting overestimations in previous PLOS Open Science Indicators (which reported 98% generation rates).

5. Significance and Implications

Re-evaluating Open Science Impact: The findings suggest that the positive impacts of data sharing and Open Science policies are currently underestimated by traditional metrics. The 43% reuse rate indicates a vibrant ecosystem of data reuse that bibliometrics fail to capture.
Policy and Funding: This tool provides funders (e.g., Michael J. Fox Foundation) and policymakers with a scalable mechanism to monitor the downstream effects of data sharing mandates, moving from measuring compliance (sharing) to measuring impact (reuse).
Citation Practices: The study highlights that researchers often describe data usage in ways that do not conform to formal citation standards, reinforcing the need for AI-driven content analysis rather than relying solely on structured metadata.
Future Directions: The authors note limitations regarding the distinction between reusing one's own data vs. others' data and the need for broader validation across different journals and disciplines.

In conclusion, this paper establishes that Generative AI is a viable and superior tool for monitoring the true prevalence and nature of research data reuse, offering a more accurate foundation for evaluating the success of Open Science initiatives.