EviSearch: A Human in the Loop System for Extracting and Auditing Clinical Evidence for Systematic Reviews

1. Problem Statement

The extraction of structured clinical evidence from trial publications (PDFs) is a critical bottleneck in evidence synthesis, meta-analysis, and the development of clinical guidelines. Current workflows rely heavily on manual curation by experts, which is time-consuming and prone to fatigue. While Large Language Models (LLMs) show promise, they face significant challenges in this domain:

• Multimodal Complexity: Clinical evidence is distributed across narrative text, complex tables, Kaplan–Meier plots, and figure captions. Standard text-based extraction often fails to interpret these heterogeneous modalities.
• Hallucination and Reliability: LLMs can hallucinate values or misattribute data, which is unacceptable in clinical settings where errors propagate into meta-analyses and treatment decisions.
• Lack of Provenance: Existing automated methods often lack "per-cell provenance," meaning extracted values cannot be easily traced back to the specific source (page, modality, or quote) for human verification.
• Schema Constraints: Clinical ontologies require specific reporting conventions (e.g., distinguishing between "not reported" and "zero"), which simple pattern matching cannot handle.

2. Methodology: The EviSearch System

EviSearch is a multi-agent, human-in-the-loop framework designed to automate evidence extraction while guaranteeing auditability. The system operates through four distinct stages:

A. System Architecture & Agents

The pipeline utilizes a dual-agent strategy followed by a reconciliation module:

1. PDF Query Agent (Agent A):
   • Input: The native PDF binary (preserving layout, figures, and multi-column structures).
   • Mechanism: Uses the File API to submit the full PDF to an LLM (Gemini-2.5-Flash) alongside column definitions.
   • Strength: Excels at global document context, trial design, and figure-heavy content where layout matters.
2. Search Agent (Agent B):
   • Input: A parsed, chunked representation of the document (text, tables, figures) with an embedded semantic index.
   • Mechanism: Operates via a tool-based agentic loop. It uses tools like search_chunks (semantic search), get_chunks_by_page, and submit_extraction.
   • Strength: Specializes in fine-grained retrieval from specific tables and results sections, using a "extract-first" policy before invoking retrieval tools.
3. Reconciliation Agent:
   • Function: Adjudicates disagreements between Agent A and Agent B.
   • Protocol:
     • Pass 1 (Agreement): If agents agree on value, page, and modality, the result is accepted.
     • Pass 2 (Conflict Resolution): If agents disagree or one reports "Not reported," the agent is forced to call get_page to retrieve the full text and rendered image of the disputed page. It then performs multimodal verification to determine the correct value or label the extraction as both_wrong.

B. Human-in-the-Loop Interface

• Automated Mode: The system runs autonomously, storing every value with explicit provenance (source page, modality, verbatim quote).
• Human-Assisted Mode: For uncertain extractions (e.g., both_wrong labels) or random sampling, a web interface presents the human reviewer with:
  • Agent A's answer.
  • Agent B's answer.
  • The Reconciler's judgment.
  • The attributed source content (highlighted in the PDF).
• Feedback Loop: Reviewer edits and reconciliation decisions are logged as structured supervision signals to bootstrap iterative model improvement.

C. Data Processing

• Schema: The system targets a 133-column schema derived from the Living Evidence Synthesis Platform (LISP) for metastatic castration-sensitive prostate cancer (mCSPC).
• Batching: Columns are grouped by clinical section (e.g., efficacy, adverse events) into batches of up to 15 to maintain context while enabling parallel processing.
• Parsing: Uses Landing AI's dpt-2-latest model for element-level chunking and OpenAI's text-embedding-3-large for semantic indexing.

3. Key Contributions

1. Unified Multimodal Pipeline: Unlike prior work that isolates table reasoning, chart interpretation, or text retrieval, EviSearch unifies these into a single pipeline specifically for ontology-aligned clinical evidence.
2. Per-Cell Provenance: The system guarantees that every extracted value is traceable to a specific page, modality (text/table/figure), and verbatim quote, enabling rigorous auditing.
3. Forced Verification Mechanism: The "forced page-level verification" in the reconciliation step ensures that disagreements are resolved via explicit multimodal evidence rather than probabilistic guessing.
4. Human-in-the-Loop Design: It shifts the paradigm from "black box" automation to a transparent workflow where clinicians can inspect the evidence chain, correct errors, and generate high-quality training data.

4. Results and Performance

The system was evaluated on a clinician-curated benchmark of oncology trial papers (mCSPC).

• Overall Accuracy: EviSearch achieved an Overall Score of 91.3% (90.9% Correctness, 91.6% Completeness).
  • This represents a 7.2 percentage point improvement over the strongest baseline (GPT-4.1 on parsed text, which scored 84.1%).
• Modality Robustness:
  • Text: EviSearch scored 91.2%.
  • Tables: EviSearch scored 91.7%.
  • Figures: EviSearch scored 86.7%.
  • Comparison: Baselines degraded significantly on figures (e.g., Parsed Doc baseline dropped to 51.6% on figures) and tables (GPT-4.1 dropped 13.8 points from text to tables). EviSearch maintained near-constant performance across all modalities.
• Cost-Efficiency Trade-off:
  • EviSearch uses more tokens (642k per doc) than native PDF prompting (149k) due to the dual-agent and reconciliation overhead.
  • However, it is significantly more efficient than parsed-text baselines (~1M tokens) because it uses retrieval to constrain context rather than feeding the full document repeatedly.
  • The authors argue the higher cost is justified by the massive gains in accuracy and the critical value of auditability in clinical settings.

5. Significance and Impact

• Accelerating Living Systematic Reviews: EviSearch reduces the manual burden of curating "living" evidence tables, allowing for faster updates as new trials are published.
• Safe Integration of LLMs: By enforcing a "human-on-the-loop" architecture with forced verification, it provides a safe pathway for integrating LLMs into high-stakes clinical decision support systems.
• Data Flywheel: The system generates structured preference data (reviewer corrections vs. model outputs) that can be used to fine-tune models for future iterations, creating a self-improving cycle.
• Clinical Trust: The ability to click a cell and see the exact source page and quote builds the necessary trust for clinicians to rely on automated extraction tools.

In conclusion, EviSearch demonstrates that a multi-agent approach combining native PDF querying, retrieval-augmented search, and forced human-verifiable reconciliation can significantly outperform single-pass LLM baselines in complex, multimodal clinical extraction tasks.