Accuracy and efficiency of using artificial intelligence for data extraction in systematic reviews. A noninferiority study within reviews

This noninferiority study demonstrates that AI-assisted data extraction using Elicit(R) is as accurate as, but significantly faster and more cost-effective than, human-only extraction for systematic reviews of RCTs, suggesting it can effectively replace one human extractor without compromising data quality.

The Gist

Imagine you are trying to build a giant, perfect puzzle of the world's health policies. To do this, you need to read hundreds of scientific studies and pull out specific facts (like "how many people were in the study" or "what medicine they took"). This process is called data extraction.

Traditionally, this job is done by humans. It's like having a team of scribes sitting in a library, reading every single book, and copying the important bits into a ledger. It takes forever, it's boring, and even the best scribes make mistakes when they get tired.

This paper asks a simple question: Can we hire a super-smart robot assistant to help the scribes, or even replace one of them, without ruining the quality of the puzzle?

Here is the story of their experiment, explained simply.

The Experiment: Humans vs. Humans + Robot

The researchers set up a race between two teams to extract data from 50 different scientific studies about children's health.

1. Team Human: Two experienced researchers read the studies and typed the data into a spreadsheet using only their eyes and brains.
2. Team Hybrid: Two researchers did the same job, but they had a secret weapon: an AI tool called Elicit®. This tool acts like a super-fast librarian. You ask it a question (e.g., "What was the age of the participants?"), and it scans the document, finds the answer, and even shows you the exact sentence where it found it. The human then checks the answer and types it in.

The Results: Who Won?

The researchers measured three things: Accuracy (did they get the facts right?), Speed (how long did it take?), and Cost (how much did it cost to run the race?).

1. Accuracy: The "Perfect Score" Test

Imagine a teacher grading a test.

• The Result: Both teams got almost the exact same score. The AI-assisted team didn't make more mistakes than the humans. In fact, for one specific part of the puzzle (describing the "intervention" or the treatment), the AI team was actually better at getting the details right.
• The Metaphor: Think of the AI as a spell-checker that never gets tired. It didn't introduce new typos; it just helped the human catch the ones they might have missed.

2. Speed: The "Fast Food" vs. "Slow Cook"

• The Result: The AI-assisted team was much faster. On average, they finished each study about 25 minutes sooner than the human-only team.
• The Metaphor: If the human-only team was walking to the grocery store, the AI-assisted team was driving a sports car. They didn't just walk faster; they skipped the traffic jams. Over 50 studies, this saved the team over 20 hours of work. That's like getting an entire work week back!

3. Errors: The "Hallucination" Myth

People often worry that AI will "hallucinate"—meaning it will make up facts that aren't true (like saying a study happened in 1990 when it actually happened in 2020).

• The Result: The AI made up facts just as rarely as the humans did. Both teams made similar types of mistakes, mostly small ones like missing a tiny detail or rounding a number slightly wrong.
• The Metaphor: The AI wasn't a crazy dreamer making up stories; it was a diligent intern who occasionally forgot to write down a phone number, just like a human would.

4. Cost: The "Wallet" Check

• The Result: The AI-assisted method was actually cheaper. Even though the AI tool costs money to subscribe to, the time saved meant the researchers spent less money on their salaries.
• The Metaphor: It's like buying a slightly expensive coffee machine. It costs more upfront than a kettle, but because it makes coffee so fast, you save money on the barista's wages in the long run.

The Big Takeaway

This study is a "non-inferiority" trial. In plain English, that means they weren't trying to prove the AI was better than humans; they just wanted to prove it wasn't worse.

The verdict? The AI assistant is non-inferior. It is just as accurate, but much faster and cheaper.

Why This Matters

Systematic reviews are the "gold standard" for making health decisions. If we can use AI to do the boring, time-consuming data gathering, human experts can stop acting like data-entry clerks and start doing what they do best: thinking, analyzing, and making sense of the big picture.

The researchers suggest that in the future, we might see a workflow where one human and one AI work together as a team, or where two AIs check each other's work, with a human just stepping in to double-check the final result. It's not about replacing humans; it's about giving them a superpower.

Technical Summary

1. Problem Statement

Systematic reviews are critical for evidence-based public health policy but are notoriously labor-intensive and prone to human error, particularly during the data extraction phase. Traditional best practices require two independent human extractors and a consensus process, which can take an average of 107–172 minutes per study. While Artificial Intelligence (AI) and Generative AI (GenAI) offer potential solutions to automate these tasks, there is a lack of robust, noninferiority evidence validating their accuracy, efficiency, and cost-effectiveness compared to human-only extraction. Specifically, there is a need to determine if AI-assisted extraction can replace one of the two human extractors without compromising data validity.

2. Methodology

The study employed a two-arm, parallel-group, noninferiority Study Within a Review (SWAR) design.

• Study Population: 50 Randomized Controlled Trials (RCTs) regarding obesity prevention interventions in children (aged 6–18) were randomly selected from a Cochrane systematic review.
• Participants: Two graduate/post-doctoral researchers with systematic review experience but no prior experience with the specific AI tool.
• Intervention Arms:
  • AI-Assisted Arm: Participants used Elicit® (Pro subscription, "high-accuracy mode") to extract data. They utilized pre-existing and custom-engineered prompts to extract 51 variables. Participants could verify, amend, or reject AI outputs but were not allowed to modify the prompts.
  • Human-Only Arm: Participants extracted the same 51 variables using only the source PDFs and standard data extraction forms, without AI assistance.
• Outcomes Measured:
  • Primary: Accuracy (assessed by a blinded, independent post-doctoral researcher using a 3-point rubric: 0=incorrect, 1=partial, 2=correct, standardized to a 0–100 scale) and Time-to-completion (including preparation and extraction).
  • Secondary: Error types (missed, misallocated, incomplete, false/hallucination, calculation errors), error severity (major, minor, inconsequential), and total cost (labor + software subscription).
• Statistical Analysis: A noninferiority margin was set at 10% (equivalent to a 5–10 point difference on the 0–100 accuracy scale). Paired t-tests were used to compare mean differences between arms.

3. Key Contributions

• First Noninferiority SWAR for GenAI: This is the first study to rigorously test GenAI-assisted data extraction against human-only extraction using a noninferiority trial design, moving beyond proof-of-concept studies.
• Blinded Independent Assessment: Unlike many previous studies that compared AI to another human extractor (introducing benchmark bias), this study used a blinded, independent assessor to verify extracted data against the source PDFs, establishing a true "gold standard."
• Comprehensive Cost-Benefit Analysis: The study provided a granular breakdown of costs, including labor, prompt engineering, file preparation, and software subscription fees, demonstrating the economic viability of AI integration.
• Granular Error Analysis: It categorized errors by type (including "hallucinations") and severity, providing specific data on the nature of AI failures compared to human errors.

4. Key Results

Accuracy

• Overall Accuracy: The AI-assisted arm was noninferior to the human-only arm.
  • Mean Difference (MD): 0.57 (95% CI: -1.29 to 2.43).
  • Both arms scored similarly on a 0–100 scale (AI: 85.8 vs. Human: 85.3).
• Subgroup Analysis:
  • The AI-assisted arm was significantly more accurate for the "Intervention and Control Group" data variables (MD 4.75, 95% CI 2.13–7.38).
  • No significant differences were found for other data groups (Methods, Participants, Data/Analysis) or data types (Alpha, Numeric).

Efficiency (Time-to-Completion)

• The AI-assisted arm was significantly faster.
  • Mean time saved: 24.82 minutes per study (95% CI 18.80–30.84).
  • AI-assisted mean time: ~48.5 mins; Human-only mean time: ~73.3 mins.
  • This represents a ~34% reduction in time per study.

Error Types and Severity

• Total Error Rate: Similar between arms (AI: 9.7% vs. Human: 10.1%).
• Hallucinations (False Data): Extremely rare and identical in both arms (1.0% each).
• Most Common Errors: "Missed/omitted data" and "Incomplete data" were the most frequent error types for both groups.
• Severity: The majority of errors in both arms were "Minor" (AI: 34.1% of errors; Human: 33.1% of errors).

Cost

• The AI-assisted arm was less expensive overall.
  • Total Cost: AI-assisted ($3,469 AUD) vs. Human-only ($3,651 AUD).
  • Savings: $181.98 AUD saved across 50 studies, primarily driven by reduced labor hours for data extraction, despite the cost of the Elicit® Pro subscription and prompt engineering time.

5. Significance and Implications

• Workflow Optimization: The findings suggest that AI-assisted extraction (using a "human-in-the-loop" model) can effectively replace one of the two human extractors in systematic reviews of RCTs without sacrificing accuracy. This could halve the labor burden and time required for data extraction.
• Scalability: For large reviews (e.g., 195 studies), the time savings could translate to thousands of hours, significantly accelerating the production of evidence for policy-making.
• Cost-Effectiveness: The reduction in labor costs outweighs the software subscription fees, making AI a financially viable tool for research teams with limited funding.
• Future Directions: The authors recommend exploring hybrid models, such as using two AI-assisted extractors with human consolidation, or AI-only extraction with human verification, to further optimize the balance between efficiency and accuracy.

Conclusion: The study concludes that Elicit®-assisted data extraction is noninferior to human-only extraction in terms of accuracy, significantly faster, and more cost-effective, supporting its adoption as a standard component in systematic review workflows.