meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis

Technical Summary: meta-pipe

Problem Statement
Systematic reviews and meta-analyses (SR/MA) are resource-intensive processes, traditionally requiring approximately 1,100 person-hours and 67 weeks to complete. While several automation tools exist (e.g., Covidence, TrialMind, otto-SR), they typically address only specific segments of the workflow. A scoping review of 37 studies indicated that while Large Language Model (LLM) approaches cover 10 of 13 SR steps, they are "not yet ready for standalone use," specifically lacking in protocol development, certainty of evidence assessment, and interpretation. Furthermore, no existing open-source tool integrates statistical analysis, manuscript generation, and quality assurance into a unified, end-to-end pipeline.

Methodology
The authors developed meta-pipe, an open-source, 10-stage modular pipeline designed to orchestrate the complete SR/MA workflow. The system architecture relies on a "human-in-the-loop" design, embedding five mandatory human decision points to ensure oversight at critical junctures (PICO definition, screening disagreements, analysis type selection, GRADE assessment, and interpretation).

The technical stack integrates four primary components:

1. LLM Orchestration: Utilizes Anthropic's Claude models. Claude Opus 4 (with extended thinking) handles complex reasoning tasks such as screening adjudication, risk-of-bias assessment, and manuscript generation. Claude Haiku 3.5 is employed for high-throughput, simple tasks like title/abstract classification and data validation.
2. Automation & Processing: Approximately 3,600 lines of Python code (managed via uv) handle data processing, LLM-assisted screening/extraction, and quality assurance.
3. Statistical Analysis: R scripts (using meta, metafor, gemtc, and netmeta packages) perform pairwise and network meta-analyses. The pipeline supports a dual-paradigm approach for Network Meta-Analysis (NMA), running both Bayesian (MCMC) and frequentist models to check for discordance.
4. Manuscript Rendering: Quarto is used to assemble and render manuscripts (HTML, DOCX, PDF) directly from R output files, ensuring statistical results are not hallucinated.

Workflow Stages
The pipeline proceeds through 10 sequential stages with standardized data contracts (BibTeX, CSV, JSON, Quarto Markdown):

• Stages 00–02: Topic intake, PICO definition, and literature search (PubMed/Scopus default).
• Stages 03–03b: LLM-assisted screening using a test-retest reliability approach (two independent passes at temperature 0.3) with a Cohen's kappa threshold of ≥0.60. A human gate resolves disagreements.
• Stages 04–05: Full-text retrieval (via Unpaywall API) and LLM-assisted data extraction with automated validation (range checks, consistency). Risk of bias is assessed using RoB 2 or ROBINS-I with mandatory human review.
• Stage 06: Statistical analysis (Pairwise or NMA).
• Stages 07–09: Automated manuscript generation, semi-automated GRADE assessment, and quality assurance (PRISMA 2020 checklist validation and overclaim detection).

Key Contributions
As a system description rather than a validation study, the paper contributes:

1. Architectural Blueprint: A standardized, open-source pipeline covering all 10 SR/MA stages with defined data contracts and quality gates.
2. Automated Manuscript Generation: The unique capability to generate structured IMRaD manuscripts directly from statistical outputs, reading effect estimates from R files to prevent hallucination.
3. Quality Assurance Layer: Implementation of semi-automated GRADE assessment and an automated "overclaim detection" system that scans for 12 predefined patterns of unsupported claims (e.g., causal language for non-randomized data, ignoring heterogeneity).
4. Dual-Paradigm NMA: Integration of both Bayesian and frequentist approaches for network meta-analysis with automatic discordance checking.

Results
The paper reports on the implementation and design of the system, not empirical performance validation.

• Implementation: The pipeline consists of 12 skill modules, 24 Python scripts, 5 R scripts, and 74 methodology documents.
• Cost: Theoretical API costs are estimated at $15–30 per typical review (5–10 studies), driven primarily by full-text PDF processing for extraction.
• Capabilities: The system successfully demonstrates the technical feasibility of end-to-end AI-assisted evidence synthesis with mandatory human oversight. It offers four capabilities not found in any single existing tool: manuscript generation, semi-automated GRADE, overclaim detection, and dual-paradigm NMA.

Significance and Claims
The authors modestly position meta-pipe as an architectural proof-of-concept rather than a validated clinical tool.

• Feasibility: The paper demonstrates that end-to-end AI-assisted evidence synthesis is architecturally feasible as an open-source tool when designed with mandatory human oversight.
• Complementarity: meta-pipe is presented as complementary to existing tools (e.g., Covidence, otto-SR), filling gaps in manuscript generation, quality assurance, and NMA rather than claiming superiority in screening or extraction accuracy.
• Limitations: The authors explicitly acknowledge the lack of validation data, the reliance on a proprietary commercial API (Claude), the use of test-retest reliability rather than PRISMA 2020 dual-review for screening, and the exclusion of certain databases (CENTRAL, Embase) in the default configuration.
• Future Work: Formal validation reproducing published Cochrane reviews is identified as the essential next step before routine use. The authors invite independent groups to validate the system and encourage the development of interoperability standards.