💻 computer science

An Auditable Large Language Model Pipeline for Data Extraction in Systematic Literature Reviews

This study demonstrates that an auditable large language model pipeline can effectively support systematic literature reviews by achieving high agreement with human experts in screening and data extraction, particularly excelling in complex fields while highlighting specific challenges with binary questions regarding unreported information.

Original authors: Philipp André Macháček, Daniel Segelcke, Esther M. Pogatzki-Zahn, Bruno Pradier

Published 2026-10-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Philipp André Macháček, Daniel Segelcke, Esther M. Pogatzki-Zahn, Bruno Pradier

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every time a doctor needs to make a critical decision about a new treatment, they often rely on a systematic literature review. This is a massive, labor-intensive process where researchers gather every relevant scientific study on a topic, read them carefully, and pull out specific facts to see what the total evidence says. Imagine trying to find a few specific needles in a haystack that keeps growing every day; that is the reality of modern medical research. As the volume of published science explodes, the manual work required to sift through thousands of articles and extract data point by point has become a bottleneck. It slows down the translation of lab discoveries into real-world cures and introduces the risk of human error, as tired reviewers might miss details or interpret ambiguous text differently. The question facing the scientific community is whether artificial intelligence can help shoulder this burden without losing the precision that human experts provide.

A team of researchers at University Hospital Münster and the University of Exeter set out to test this idea not with a small, simulated experiment, but by running a large language model through a complete, real-world scientific review. They took a published study that had already analyzed hundreds of preclinical experiments on pain in rodents and asked an artificial intelligence system to do the exact same work. The goal was to see if the machine could act as a reliable second pair of eyes, catching mistakes or confirming findings, rather than simply replacing the human researcher. The researchers fed the AI the same thousands of articles and asked it to answer the same hundreds of specific questions about animal models, treatments, and outcomes that the original human team had answered years earlier.

The results showed a high level of cooperation between the human and the machine. When the AI and the human reviewers looked at the titles and abstracts of 6,474 records to decide which ones to keep, they agreed on the vast majority of decisions. When they moved to the harder task of reading the full text of 674 studies and extracting 74,140 specific data points, they still agreed on nearly 90 percent of the answers. However, the story of their disagreement is where the true value of the study lies. In the cases where the human and the AI disagreed, the researchers brought in independent experts to act as judges, looking at the original articles to decide who was right. These experts sided with the artificial intelligence in more than 40 percent of the conflicts, proving that the machine had spotted errors the human reviewers had missed. Conversely, the human experts were right more often when the AI struggled with questions that required figuring out what was not reported in a text, such as determining if a specific safety measure was missing or if a study lacked a necessary detail.

The study revealed that the AI's performance depended heavily on the type of question being asked. It was exceptionally good at finding facts that were clearly written down, such as the specific strain of animal used or the exact drug administered, achieving near-perfect accuracy on these free-text and multi-choice fields. It also excelled at pulling out numbers and lists of options. The machine struggled most with yes-or-no questions that required it to conclude that information was absent or to interpret the author's intent behind a vague description. This is a crucial distinction because it suggests that the AI is not a magic wand that can do everything, but a powerful tool that works best when its strengths are matched to the right tasks. The researchers found that the difficulty of the task was determined more by how the question was asked than by the medical topic itself.

Ultimately, this work demonstrates that an artificial intelligence system can serve as an auditable second reader in the rigorous world of evidence synthesis. It does not replace the human expert but rather creates a parallel stream of extraction that highlights where the two disagree. By flagging these specific points of conflict, the system allows human reviewers to focus their attention on the most difficult parts of the data, such as judging missing information or interpreting complex study designs. The researchers concluded that this partnership offers a practical path forward: using the machine to handle the heavy lifting of finding explicit facts while keeping human oversight targeted at the areas where judgment and interpretation are most critical. This approach promises to make the process of reviewing scientific evidence faster and more reliable, ensuring that the conclusions drawn from vast amounts of data are as accurate as possible.

Technical Summary: An Auditable Large Language Model Pipeline for Data Extraction in Systematic Literature Reviews

Problem Statement

Systematic Literature Reviews (SLRs) are foundational to evidence-based decision-making but face significant scalability challenges due to the labor-intensive nature of manual data extraction. As the volume of scientific literature grows, the requirement for human reviewers to code study attributes field-by-field creates a throughput bottleneck and introduces inter-reviewer reliability issues. While automation has been proposed to mitigate this, existing solutions primarily focus on title-and-abstract screening (binary classification) or rely on supervised text classification for retrieving specific text spans. They lack the capability to perform generative, field-by-field extraction from heterogeneous full-text articles. Furthermore, rigorous evaluations of Large Language Models (LLMs) in this context are scarce; most studies rely on simulated tasks or benchmark subsets rather than complete, real-world systematic reviews. The critical gap is not the existence of generative pipelines, but the lack of an evaluation at the evidential standard required for real-world practice: a complete comparison against archived human extractions with every disagreement resolved by blinded expert adjudication.

Methodology

The authors evaluated an end-to-end, auditable LLM pipeline against a published preclinical SLR on post-surgical pain models (6,474 records screened; 674 studies extracted). The study utilized a "human-in-the-loop" verification process where the LLM acted as a second reader alongside trained human reviewers.

Pipeline Architecture:

Models: Two distinct models were employed based on task requirements:
- Screening: MiniMax-M2.5 (one-shot prompting).
- Full-text Extraction: MiniMax-Text-01 (few-shot, question-specific workflow within a persistent conversation thread to maintain context).
Workflow:
1. Preprocessing: PDFs were converted to plain text using PyMuPDF, with tables appended as text and images marked as placeholders. Text was chunked (max 35,000 characters) and fed into the model.
2. Extraction: The pipeline processed 47 questions across 110 fields (74,140 data points total). Questions were submitted sequentially within a single conversation thread, carrying the full accumulated context of prior exchanges.
3. Parsing & Normalization: Raw model outputs were parsed into structured fields matching the human extraction forms. Free-text outputs were normalized to controlled vocabularies where necessary.
4. Adjudication: All disagreements between the human reviewer and the LLM were forwarded to blinded domain experts (>15 years experience). Experts reviewed the source full text and selected the correct answer (Human, LLM, or Neither).
Evaluation Metrics:
- Agreement: Percentage of matching data points.
- Estimated Correctness: Calculated as $(M + D_{LLM}) / N$ , where $M$ is the number of matching fields, $D_{LLM}$ is the number of disagreements resolved in favor of the LLM, and $N$ is the total data points. This metric acknowledges that the archived human extraction is not an error-free gold standard.
- Stratification: Performance was analyzed by answer format (Yes/No, Single-select, Numeric, Free-text, Multi-select) and topical domain.

Key Contributions

Comprehensive Real-World Evaluation: Unlike prior work using benchmark subsets, this study evaluated a complete, published SLR corpus (674 studies, 74,140 data points) field-by-field against archived human extractions.
Blinded Expert Adjudication: The study resolved 7,835 extraction disagreements using blinded experts, allowing for the estimation of LLM correctness independent of the assumption that the initial human extraction was always correct.
Failure Mode Analysis: The research identifies that performance is driven more by the type of cognitive operation required (e.g., detecting absence vs. extracting explicit facts) than by the biomedical topic.
Auditable Workflow Design: The paper presents a transparent pipeline where prompts, parsing rules, and adjudication decisions are retained as audit artifacts, supporting the "RAISE" principles for responsible AI in evidence synthesis.

Results

Overall Performance:

Screening: Human-LLM agreement was 94.2% (6,096/6,474 records). After adjudication, estimated LLM correctness was 96.29%, compared to 97.87% for the human reviewer.
Extraction: Human-LLM agreement was 89.4% (66,305/74,140 data points). After adjudication, estimated LLM correctness was 93.92%, compared to 95.14% for the human reviewer.
Conflict Resolution: In full-text extraction disagreements, experts favored the LLM in 42.4% of cases, the human in 54.0%, and neither in 3.6%.

Performance by Answer Format:

High Performance: Free-text and Multi-select fields achieved the highest estimated LLM correctness (99.0%). The model excelled at extracting explicitly reported information.
Low Performance: Yes/No fields showed the lowest estimated LLM correctness (76.9%). The primary failure mode was interpreting negative evidence (determining that information was absent, not reported, or explicitly negated).
- When the correct answer was "not reported," LLM correctness was 58.4% vs. 94.0% for humans.
- When the correct answer was an explicit "no," LLM correctness was 74.2% vs. 92.8% for humans.
Numeric Fields: The model outperformed humans in numeric extraction (94.6% vs. 89.8%), often correctly identifying values where humans missed them.

Operational Findings:

Time: LLM processing took 16.9 hours (API time) vs. 85.7 hours for manual extraction. However, the authors note this excludes prompt development, data cleaning, and quality control, so it does not represent a net labor saving in this retrospective setup.
Error Types: The dominant failure mode was not unparseable output (format violations were rare, <0.03% for Yes/No) but semantically incorrect, well-formed answers.

Significance and Claims

The paper claims that LLMs are viable as auditable second readers rather than autonomous replacements for human reviewers. The significance of the work lies in:

Targeted Oversight: The results demonstrate that reliability is not uniform; it depends heavily on the evidential operation required. Human oversight can be strategically targeted to high-risk fields (e.g., Yes/No questions regarding absence or author intent) while lower-risk fields (explicit facts, multi-select) can be handled with high confidence by the LLM.
Error Detection: The LLM acts as an independent stream that can detect human omissions and miscoding. In multi-select fields, the model identified applicable options more completely than trained human reviewers (higher sensitivity).
Methodological Transparency: The study provides a framework for evaluating AI in evidence synthesis that prioritizes transparency, auditability, and the resolution of disagreements over simple aggregate accuracy.
Realistic Expectations: The authors emphasize that current AI tools are not ready for fully autonomous deployment across all methodological tasks. The proposed workflow maintains meaningful human responsibility, aligning with the RAISE principles, by using the LLM to surface conflicts for expert review rather than replacing the review process entirely.

The paper concludes that while a single aggregate performance metric is insufficient, a structured pipeline that combines generative extraction with blinded adjudication offers a practical path forward for integrating LLMs into human-led evidence synthesis.