💬 NLP

Scientific Knowledge Discovery in the Age of Large Language Models

This chapter surveys 34 peer-reviewed studies that explore how generative large language models can improve scientific knowledge discovery by automating literature retrieval and screening candidate studies against eligibility criteria, addressing the challenges posed by the rapid growth of scholarly literature.

Original authors: Eleni Adamidi, Serafeim Chatzopoulos, Thanasis Vergoulis

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Eleni Adamidi, Serafeim Chatzopoulos, Thanasis Vergoulis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of science as a massive, ever-expanding library. Every day, thousands of new books (research papers) are written and added to the shelves. In the past, a librarian (a researcher) could walk the aisles, find the books they needed, and check if they were good enough to use. But now, the library is growing so fast that the librarian is drowning. They can't read every book, and they might miss the most important ones or accidentally buy the same book twice. This is the problem of "scientific knowledge discovery." To help, scientists are trying out a new kind of tool: Large Language Models (LLMs). Think of these not as simple search engines that just match keywords like "dog" or "cat," but as super-smart, reading robots that can understand the meaning of a question, skim through millions of pages in a blink, and tell you, "Hey, this book is exactly what you're looking for," or "Nope, this one doesn't fit the rules."

This paper is like a detective's report on how well these reading robots are actually doing their job. The authors, Eleni Adamidi, Serafeim Chatzopoulos, and Thanasis Vergoulis, gathered 34 recent studies (published between 2024 and 2026) to see how these AI tools are being used to solve two specific problems: Literature Retrieval (finding the right books) and Literature Screening (deciding which books are good enough to keep). They wanted to know: Are these robots smart enough to replace the tired librarian? What kind of robots are people using? And how do we know if they are actually doing a good job?

The Great Library Hunt: Finding the Right Books

The first half of the report looks at Literature Retrieval. This is the "search" phase. Imagine you ask a librarian, "I need a book about how to fix a broken robot arm." A traditional search engine might just grab every book with the words "robot" and "arm" in the title, even if it's about a toy robot or a human arm. The new AI tools try to understand that you mean a mechanical arm for a machine.

The paper finds that researchers are trying many different tricks to make these robots smarter. Some use the AI to break a big question into smaller, easier questions (like a detective breaking a case down). Others use the AI to build a map of connections between different books, so it can find hidden links that a simple search would miss. Some systems even use multiple AI agents working together: one agent plans the search, another looks for the books, and a third checks if the results make sense.

However, the report suggests a tricky problem: while these AI tools are great at generating answers, they aren't always being tested on how well they find the answers. In many studies, the researchers only checked if the final story the AI wrote was good, not whether the AI picked the right books to write that story from. It's like judging a chef only by the taste of the soup, without checking if they used the right ingredients. The paper notes that while there are some clever new systems, we don't have enough proof yet that they are better than the old methods at actually finding the right papers. Also, most of these "robots" are just using pre-made, off-the-shelf models (like asking a famous chef to cook) rather than training a custom robot for the specific job.

The Gatekeepers: Deciding What to Keep

The second half of the report focuses on Literature Screening. This is the "filter" phase. Imagine you have a pile of 1,000 books, but you only want the ones that are about "robot arms" and were written in the last five years. A human has to read the title and summary of every single book to decide if it belongs. This is slow and boring.

Here, the AI robots are being used as gatekeepers. The paper finds that researchers are testing two main ways to do this. The first way is the "One-Shot" method: you hand the book to the robot, ask "Keep or toss?", and it gives an answer. The second way is the "Teamwork" method: you use a whole team of robots. One reads the book, another checks the facts, and if they disagree, a third robot acts as a judge to make the final call.

The report suggests that the "Teamwork" approach is looking more promising. It's like having a panel of judges instead of just one person; if one judge makes a mistake, the others can catch it. The paper also highlights that most of these tests are happening in the world of medicine and biology. Why? Because in medicine, there are strict rules about what counts as "evidence," and there are already huge piles of past decisions (like which studies were kept for a medical review) that researchers can use to test if their robots are accurate. In other fields, like history or engineering, we don't have these clear rules or test piles yet, so it's harder to know if the robots are doing a good job.

The Big Picture

So, what's the final verdict? The paper suggests that we are in the early, exciting days of using AI to manage the library. The robots are getting better at understanding complex questions and working together in teams. However, they are still mostly being used in the medical world, and we need more proof that they are actually finding the right books better than humans can.

The authors point out that while the technology is moving fast, we need to be careful. Just because a robot can write a nice summary doesn't mean it found the right sources. And while some people are using powerful, expensive "closed" robots (like the famous ones from big tech companies), others are starting to use open, customizable robots that can run on their own computers to save money and keep secrets safe.

In short, the AI librarian is a helpful new assistant, but it's not quite ready to take over the whole library just yet. It needs more training, better testing, and perhaps a little more teamwork before it can fully replace the human researchers who have been doing this hard work for so long. The future looks bright, but for now, we're still watching the robots learn how to read.

Technical Summary: Scientific Knowledge Discovery in the Age of Large Language Models

Problem Statement

The rapid expansion of scientific literature has rendered traditional search and review workflows increasingly inefficient. Conventional systems rely heavily on keyword matching, citation-based ranking, and labor-intensive manual screening, leading to risks of overlooked relevant work, duplicated efforts, and delayed evidence synthesis. While semi-automated tools exist, they primarily assist in prioritization rather than replacing human assessment. The authors identify a critical need for more flexible methods capable of interpreting natural language, capturing semantic relationships, and producing structured judgments to support scientific knowledge discovery, specifically focusing on two core tasks:

Literature Retrieval: Identifying, ranking, or recommending publications relevant to a query, seed paper, or user profile.
Literature Screening: Assessing candidate publications against predefined eligibility criteria, typically within systematic reviews and evidence-synthesis workflows.

Methodology

This chapter presents a systematic survey of 34 peer-reviewed papers published between 2024 and 2026 that apply generative Large Language Models (LLMs) to these tasks.

Search Strategy: The authors utilized BIP! Finder, a scholarly search engine indexing over 250 million publications via the OpenAIRE Graph. A Boolean search was conducted in May 2026 using terms related to "large language model" combined with specific retrieval and screening keywords.
Selection Criteria: From an initial pool of 1,589 records, studies were included if they: (i) addressed retrieval or screening; (ii) used a generative LLM as a primary component; (iii) reported sufficient methodological detail for comparison; and (iv) were written in English. Extended abstracts, preprints (unless a peer-reviewed version was found), and studies focusing on tasks outside the scope (e.g., summarization) were excluded.
Corpus Characteristics: The final corpus of 34 papers shows rapid growth (9 in 2024, 24 in 2025, 1 in early 2026). The literature is heavily concentrated in biomedical and clinical domains, with "Biomedical Text Mining and Ontologies" and "Topic Modeling" being the most frequent research topics.
Analysis Framework: The survey categorizes works into two thematic areas (Retrieval and Screening) and further subgroups them by technical approach. Each group is analyzed based on:
- Models Used: Specific LLMs, access methods (API, local, Web UI), and adaptation strategies (fine-tuning vs. prompting).
- Evaluation Trends: Ground truth sources (existing benchmarks vs. author-constructed) and metrics (ranking metrics, diagnostic accuracy, efficiency).

Key Contributions and Findings

1. LLMs for Literature Retrieval

The survey identifies two primary modes of retrieval application:

Core Search & Recommendation: This is the most diverse area, encompassing five research strands:
- Semantic Search: Using LLMs to extract multi-granular concepts (e.g., SemRank) or re-weight embeddings to penalize redundancy (e.g., Zheng et al.).
- Agentic Search: Systems where LLMs plan, route, and refine retrieval (e.g., LitChat, LITERAS). These often employ multi-agent architectures for query generation, searching, validation, and refinement.
- Structured Knowledge Grounding: Retrieval grounded in knowledge graphs (e.g., GPTscholar generating SPARQL queries; Liu et al. generating Cypher queries) to reduce hallucinations.
- Citation Auditing: Controlled studies examining the ability of LLMs to reproduce reference lists from memory without database grounding (e.g., Chelli et al.).
- Profile-Driven Recommendation: Using LLMs to refine similarity signals for graph-based recommenders (e.g., MCAP).
- Models & Evaluation: Most systems use commercial models (GPT-4, Claude, Gemini) via API without fine-tuning. Evaluation often relies on classic IR metrics (Recall, Precision, nDCG) against existing benchmarks. However, focused evaluation of the retrieval component is often absent, as retrieval is frequently treated as a supporting step for downstream tasks.
RAG-Style Retrieval for Downstream Tasks:
- These systems integrate retrieval into pipelines for automated peer review, clinical decision support, or survey generation.
- Retrieval is used to contextualize novelty, ground limitations, or support evidence-based decisions.
- Evaluation Gap: In these studies, retrieval quality is rarely evaluated against a dedicated ground truth. Evaluation focuses on the downstream output (e.g., the quality of a review or treatment plan), using indirect signals like citation proxies or ablation studies.

2. LLMs for Literature Screening

Screening approaches are divided into two architectural patterns:

Single-Pass Screening:
- Treats screening as a single model call per paper.
- Technical variation lies in prompt engineering: static criteria, few-shot examples, Chain-of-Thought (CoT), and iterative prompt optimization (e.g., Tao et al., Ruan et al.).
- Evaluation: Highly rigorous, utilizing standard diagnostic accuracy metrics (Precision, Recall, Specificity, F1, MCC) against existing systematic reviews or author-constructed ground truth.
- Models: A mix of commercial (GPT, Claude, Gemini) and open-weight models (Llama, Mistral, Gemma). Notably, Delgado-Chaves et al. conducted the broadest comparison of 18 models. No fine-tuning of model weights was observed; adaptation was limited to prompting.
Engineered Multi-Component Pipelines:
- Designed to mimic human review teams through pre-filtering, inter-model arbitration, or retrieval-augmented screening.
- Arbitration: Multiple models screen independently, with a third model or human expert resolving disagreements (e.g., Dai et al., Pan et al.).
- Retrieval-Augmented: Systems retrieve full-text passages to answer eligibility questions (e.g., Trad et al.).
- Agentic/Iterative: Multi-agent systems that refine prompts based on feedback or apply workload-reduction thresholds (e.g., GREP-Agent).
- Evaluation: Uses standard diagnostic metrics alongside agreement statistics (Cohen's $\kappa$ , PABAK) and efficiency metrics (workload reduction, cost). Ground truth is typically derived from existing systematic reviews or expert-labeled subsets.

Significance and Claims

The paper claims to provide a comprehensive overview of the rapidly evolving landscape of LLM-assisted scientific knowledge discovery. Its primary significance lies in:

Mapping the Landscape: It characterizes the shift from single-model calls to composed, often agentic, multi-component systems for both retrieval and screening.
Identifying Methodological Divergence:
- Retrieval is technically diverse but methodologically fragmented, with a lack of standardized evaluation for the retrieval component itself.
- Screening is more methodologically standardized, utilizing rigorous diagnostic-accuracy benchmarks and moving toward production-usable tools via multi-model consensus.
Domain Concentration: The survey highlights that current advancements are heavily concentrated in biomedical and clinical applications. This is attributed to the maturity of systematic review methodologies in biomedicine, which provide established ground truth and clear evaluation conventions.
Adoption Trends: The field is seeing a rise in open-weight models (often for privacy and cost control) alongside dominant closed API models. However, domain-specific fine-tuning remains rare, with adaptation primarily occurring at the prompt level.

The authors conclude that while LLM-assisted retrieval and screening are moving from isolated demonstrations toward established practice, the field remains domain-bound. The transferability of these approaches to non-biomedical domains is currently unclear due to the lack of comparable benchmarks and review conventions outside of biomedicine.