💬 NLP

From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFs

This paper presents a schema-constrained AI system that transforms full-text biomedical PDFs into structured, auditable evidence records by employing typed schemas, provenance tracking, and conflict-aware consolidation to overcome the scalability and reliability limitations of existing document AI in evidence synthesis.

Original authors: Pouria Mortezaagha, Joseph Shaw, Bowen Sun, Arya Rahgozar

Published 2026-01-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pouria Mortezaagha, Joseph Shaw, Bowen Sun, Arya Rahgozar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive mystery. You have a library filled with thousands of old, messy books (scientific PDFs). These books are written in a strange way: the text is split across columns, important clues are hidden inside pictures and tables, and the pages are sometimes scanned poorly, making the letters look like gibberish.

Your job is to find specific facts in every single book—like "What drug was used?" or "How long did the study last?" and write them down in a neat spreadsheet. Doing this by hand would take a lifetime, and you'd likely make mistakes because the books are so messy.

This paper introduces a new kind of AI detective designed to do this job automatically, but with a very specific set of rules to ensure it doesn't just "guess" or make things up.

Here is how the system works, broken down into simple parts:

1. The Problem: The "Messy Library"

Scientific papers are saved as PDFs. To a computer, a PDF is just a picture of a page, not a list of words. It's like looking at a photo of a newspaper; the computer sees pixels, not sentences.

The Challenge: The text is often in two columns, mixed with charts, and the most important numbers might be hidden in a tiny caption under a graph.
The Risk: If you just ask a standard AI to "read" these, it might get confused by the layout, miss the hidden numbers, or confidently invent facts that aren't there (a problem called "hallucination").

2. The Solution: The "Strict Rulebook" (Schema Constraints)

Instead of letting the AI guess, the researchers gave it a strict rulebook (called a "schema").

The Analogy: Imagine filling out a tax form. You can't just write whatever you want in the "Income" box; you must write a number, and if you don't have one, you must write "N/A." You can't write "I think I made a lot."
How it works here: The AI is forced to choose answers only from a pre-approved list of words (like specific drug names) and must provide the exact sentence from the book that proves its answer. If the book doesn't say it, the AI must leave the box blank. It cannot invent facts.

3. The Process: The "Assembly Line"

The system doesn't try to read the whole 50-page book at once. That would overwhelm the AI's memory.

Chopping it up: The system cuts the book into small, manageable chunks (like 8 pages at a time).
The Assembly Line: It processes these chunks one by one, like items on a factory conveyor belt. It uses a "traffic cop" (rate limiting) to make sure it doesn't ask the AI too many questions too fast, which would cause the system to crash.
The "Resume" Feature: If the power goes out or the internet cuts, the system remembers exactly where it left off. When it starts again, it doesn't re-read the books it already finished; it just picks up the new ones.

4. The "Proof" (Provenance)

This is the most important part for trust.

The Analogy: In a courtroom, a witness can't just say, "I think the suspect was there." They must point to the specific moment in the video footage and say, "Look at 2:04 PM."
How it works here: For every fact the AI extracts (e.g., "The study lasted 6 months"), it must also save the exact sentence from the original PDF that says "6 months." This allows a human expert to quickly check the work without reading the whole book again.

5. The Results: From Chaos to Clarity

The researchers tested this system on 734 scientific papers about blood-thinning drugs (DOACs).

Speed: It processed the entire library in a fraction of the time it would take a human.
Accuracy: When they checked the work, the system was very good at following the rules. However, the biggest improvement came from iterative refinement.
- The "Practice" Analogy: At first, the rulebook wasn't perfect. The researchers looked at the AI's mistakes, realized the rules were vague, and rewrote the rulebook to be clearer. After a few rounds of this "practice," the AI's accuracy jumped significantly (for example, correctly identifying study outcomes went from about 33% to 95%).
The Output: The system produced two things:
1. A Spreadsheet: Clean data ready for analysis.
2. A "Reconstructed" Book: A digital version of the original paper where the AI's notes are highlighted right next to the original text and images, making it easy for humans to verify.

The Bottom Line

This paper doesn't claim the AI is perfect or that it can replace human doctors. Instead, it claims to have built a reliable, auditable tool that turns messy, unreadable scientific PDFs into clean, organized data.

It does this by:

Forcing the AI to follow strict rules (no guessing).
Making the AI show its work (providing the source sentence).
Allowing humans to fix the rules to make the AI smarter over time.

This turns the impossible task of reading thousands of messy papers into a manageable workflow where humans only need to check the AI's "homework" rather than doing the whole assignment themselves.

Technical Summary: From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFs

Problem Statement

Biomedical evidence synthesis relies on the accurate extraction of methodological, laboratory, and outcome variables from full-text research articles. However, this information is predominantly embedded in complex scientific PDFs that interleave multi-column text, tables, figures, and captions. This format, optimized for visual presentation rather than machine-readable structure, renders manual abstraction time-intensive, error-prone, and unscalable for contemporary systematic reviews.

Existing solutions face significant limitations:

Layout and OCR Challenges: Scientific PDFs encode positioned glyphs rather than logical reading order. Errors in reading-order reconstruction, table detection, or caption association can cascade into downstream extraction failures.
Model Limitations: While layout-aware transformers and multimodal models have advanced, end-to-end extraction systems often suffer from limited throughput, OCR error propagation, and insufficient auditability.
Auditability Gap: In high-stakes biomedical applications, accuracy alone is insufficient. Extracted variables must be auditable, reproducible, and explicitly linked to supporting evidence in the source document. Current systems often lack explicit guarantees of traceability or mechanisms to control inference under ambiguity.

Methodology

The authors propose an end-to-end, OCR-driven pipeline that couples high-accuracy document transcription with schema-constrained structured extraction. The system is designed to process full-text biomedical PDFs into structured, analysis-ready records while explicitly restricting model inference to ensure auditability.

1. Pipeline Architecture and Orchestration

Ingestion: Documents are ingested using deterministic file discovery and resume-aware hashing (SHA-1 of canonicalized file identifiers) to prevent redundant processing. PDFs are converted to base64-encoded data URLs.
Chunking: To respect model context limits, documents are partitioned into non-overlapping page-level chunks (default $k=8$ pages).
Asynchronous Execution: The pipeline uses a semaphore-based concurrency control and a time-based rate limiter to manage interaction with the Mistral OCR engine. It employs an exponential backoff strategy for transient errors (e.g., rate limits), ensuring stable throughput under service constraints.
Caption Awareness: Image captions are isolated as independent semantic units and processed with the same schema constraints as body text, acknowledging that captions often contain substantive methodological or numerical information.

2. Schema-Constrained Inference
Instead of unstructured text generation, the OCR stage is coupled with schema-aware annotation using typed schemas (implemented via Pydantic).

Payloads: Five major domain-specific payloads are defined: (1) Bibliographic/Study Design, (2) Population/Indications, (3) Methods/Assays, (4) Outcomes/Timing/Thresholds, and (5) Diagnostic Performance.
Constraints: Schemas utilize controlled vocabularies, closed sets of labels, and explicit negative rules (e.g., "do not infer," "set to null if ambiguous").
Evidence Gating: For every high-level classification, the model is required to populate a parallel "sentence from text" field, quoting the exact sentence(s) or paragraph(s) justifying the decision. This ensures sentence-level provenance.

3. Record Construction and Consolidation

Merging: Annotations from page-level chunks are deterministically merged into single study-level records.
- Scalar fields: Require agreement across all non-null chunks; conflicts trigger manual review flags.
- List-valued fields: Values are aggregated into sets to remove duplicates while preserving the order of first appearance.
- Evidence fields: All distinct quoted sentences are retained to enable traceability.
Outputs: The system exports tabular data (CSV, Parquet) and human-readable, multimodal markdown reconstructions that integrate OCR text, figures, and structured annotations.

Key Contributions

Schema-Constrained Extraction System: A system that governs long-document inference through typed schemas, controlled vocabularies, and explicit negative rules, reducing over-inference under heterogeneous layouts and OCR noise.
Deterministic Consolidation Strategy: A formalized method for reconciling page-level annotations into coherent study-level records, enforcing consistency for scalar fields and set-based aggregation for list-valued fields.
First-Class Provenance: Treating sentence-level evidence as a primary output, binding synthesis-critical variables to explicit supporting text to enable transparent audit and expert adjudication.
Scalable, Auditable Workflow: Validation of the approach on a challenging laboratory medicine domain (Direct Oral Anticoagulant [DOAC] level measurement), demonstrating stable throughput under service constraints and improved fidelity via expert-in-the-loop schema refinement.

Results

The pipeline was evaluated on a corpus of 734 biomedical PDFs (7,228 pages) related to DOAC level measurement.

Execution: The system processed all documents without manual intervention. It maintained a steady-state throughput of 3.1 OCR requests per second under strict concurrency (3 concurrent calls) and rate-limiting (5 RPS) constraints.
Robustness: The system handled transient failures automatically via retry logic, with no fatal document-level failures.
Extraction Fidelity: Expert review of a random sample of 50 studies showed substantial improvements in extraction correctness following iterative schema and prompt refinement.
- Clinical outcome definitions: Improved from 33% to 95%.
- Study design classification: Improved from 75% to 100%.
- Assay types: Improved from 70% to 95%.
- Global coagulation testing: Improved from 70% to 100%.
Completeness: Payload-level completeness was high (e.g., 99.9% for bibliographic metadata, 98.6% for methods), though field-level completeness for complex constructs (e.g., follow-up duration) reflected the inherent heterogeneity of reporting in the source literature.
Quality: A proxy transcription quality score indicated a median of 98.6%, sensitive to error modes affecting downstream schema extraction.

Significance and Claims

The paper posits that the primary challenge in biomedical evidence synthesis is not incremental accuracy in language modeling, but robust end-to-end reliability under heterogeneous layouts and document quality.

Shift in Paradigm: The authors argue that reliable extraction arises not solely from increased model capacity, but from explicit specification, conservative inference, and enforceable traceability. By constraining the hypothesis space through schemas and requiring evidence gating, the system mitigates the risk of "silent over-inference" common in unconstrained generative models.
Practical Utility: The pipeline shifts the human expert's role from exhaustive manual abstraction to targeted audit and adjudication. By providing sentence-level provenance and multimodal reconstructions, the system allows reviewers to verify extracted values against source text rapidly.
Scalability: The approach demonstrates that AI-driven extraction can be scaled to corpus-level volumes while maintaining controllability and auditability, addressing a critical bottleneck in systematic reviews and guideline updates.

The work concludes that this schema-constrained, provenance-centric design provides a practical foundation for scalable, traceable, and reproducible scientific annotation, aligning modern document understanding capabilities with the transparency demands of biomedical evidence synthesis.