📄 medicine

VIGIL: A Validation-First Research Infrastructure for Global Infectious Disease Intelligence

The paper introduces VIGIL, a validation-first research infrastructure that systematically extracts infectious disease intelligence from diverse open-source documents and continuously validates it against WHO reference data to quantify reporting biases and establish open-source signals as testable hypotheses rather than direct evidence.

Original authors: Yen-Hsaing Wang

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yen-Hsaing Wang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out how many monsters are hiding in a giant, messy library. Every day, new books, newspapers, and blog posts arrive, shouting about where these monsters (germs) are and how they are fighting back against our medicine. For a long time, scientists thought: "If we just read more books, we'll know exactly where the monsters are!" They assumed that a library with a huge pile of books about a specific monster meant that monster was everywhere and dangerous.

But a new tool called VIGIL (which stands for something like "Vigilance Intelligence for Global Infectious-disease & Resistance Landscape") says, "Wait a second. Let's not just trust the pile of books. Let's check if the books are actually telling the truth."

The "Trust No One" Library

Think of VIGIL as a super-smart librarian who doesn't just stack books; she has a secret, independent checklist. While other systems might just count how many times a word appears in a book and say, "Aha! This monster is a big problem here!", VIGIL grabs that claim and runs it against a strict, official scorecard kept by the World Health Organization (WHO).

The paper describes VIGIL as a "validation-first" machine. This means it doesn't care how cool or fast it is at finding information in text. Its main job is to ask: "Is this story true?" before it lets anyone use the information. It treats every new piece of news not as a fact, but as a "hypothesis" that needs to be tested.

The Big Surprise: More Books ≠ More Monsters

Here is the most exciting part of the story. The researchers fed VIGIL a snapshot of 2,075 documents from places like the CDC, medical journals, and preprint servers. The tool successfully pulled out 31 different pathogens (the monsters), 12 resistance mechanisms (how they fight back), and 62 countries from the text. It even built a giant "knowledge graph"—think of it as a massive web of connections showing which germ is linked to which drug and which country.

But then, they did the test. They compared what the books said against the official WHO data for 43 countries.

The result was a total shock. The paper found that the number of documents written about a germ had almost no connection to how much resistance was actually measured in real life.

When they looked at the raw number of documents, the connection was basically zero (a statistical number called Spearman's ρ of 0.07).
Even when they tried to fix the math by adjusting for how many total papers a country writes (publication-normalized), the connection only got slightly better, but it was still very weak (ρ of 0.16).

What this means: Just because a country writes a lot of articles about a super-bug doesn't mean that super-bug is actually causing more problems there. It just means that country has a lot of writers! The paper explicitly rules out the idea that "literature volume" (how many articles exist) is a good way to guess how bad a disease is. The authors suggest that relying on book counts alone is like trying to guess how hungry a city is by counting how many people are talking about food—it's a bad proxy.

Why This Matters (And Why It's Not a Magic Bullet)

The paper argues that we need to stop treating text-based news as "evidence" and start treating it as "clues" that need checking. VIGIL is a research infrastructure, which is a fancy way of saying it's a playground for scientists to test their ideas.

The authors are careful to say this isn't a finished, perfect solution yet. They call their current results "proof-of-concept," meaning they proved the idea works, but the data is still "young" and "sparse."

They haven't finished checking if their computer code (using Large Language Models) is perfect; they are still working on getting doctors to double-check the answers.
They admit that for some countries, there simply isn't enough data in the system yet to make a strong call.
They note that the tool can't see everything; for example, it can't check data for Taiwan because the official reference data doesn't include it, not because the tool is broken.

The Takeaway

VIGIL is like a new kind of detective that refuses to solve a case until it has checked the alibi against a police database. It shows us that in the world of infectious diseases, more noise doesn't mean more signal.

The paper suggests that the future of tracking diseases shouldn't be about who can read the most books, but about who can build the best "truth-checking" systems. It turns the problem of "bias" (where some places write more than others) into something we can actually measure and track, rather than just ignoring it. It's a tool for the future, designed to help us stop guessing and start knowing, but for now, it's a very promising start, not a solved mystery.

Technical Summary: VIGIL – A Validation-First Research Infrastructure for Global Infectious Disease Intelligence

Problem Statement
Open-source infectious disease and antimicrobial resistance (AMR) intelligence is increasingly derived from heterogeneous sources, including outbreak reports, biomedical literature, preprints, and clinical trial registries. However, signals extracted from these text sources are inherently susceptible to reporting and publication biases. A critical methodological gap exists: current surveillance systems typically treat text-derived signals as evidence to be trusted, with validation against independent reference data treated as a downstream evaluation or omitted entirely. Furthermore, existing platforms (e.g., WHO EIOS, HealthMap) focus primarily on outbreak detection rather than structured AMR relationships, while curated systems (e.g., WHO GLASS) suffer from reporting delays and incomplete geographical coverage. There is a lack of an openly reproducible framework that integrates automated ingestion, structured extraction, and systematic validation as a primary operational principle.

Methodology
The authors developed VIGIL (Vigilance Intelligence for Global Infectious-disease & Resistance Landscape), a modular research infrastructure designed around a "validation-first" paradigm. The system operates through five sequential stages:

Data Collection: VIGIL continuously ingests data from eight categories of open sources via APIs and syndicated feeds: WHO Disease Outbreak News, US CDC, ECDC, UK Health Security Agency, PubMed, bioRxiv/medRxiv (via Europe PMC), ClinicalTrials.gov, and WHO GHO/GLASS reference data. All documents are stored in a unified SQLite database with complete provenance (original text, source, URL, timestamp).
Structured Information Extraction: The platform employs two interchangeable pipelines to produce an identical JSON schema:
- A rule-based extractor using frozen, version-controlled codebooks, word-boundary matching, and canonicalization. This serves as a transparent, deterministic baseline.
- A Large Language Model (LLM) extractor (currently Google Gemini) instructed to generate the same schema.
- Both outputs are stored independently to enable reproducible benchmarking under identical inputs.
Knowledge Graph Integration: Extracted entities (pathogens, resistance mechanisms, antimicrobial agents, diseases, countries) are integrated into a queryable knowledge graph representing relationships and co-occurrences.
Validation-First Analytics: Rather than presenting isolated features, VIGIL formulates analytical modules (e.g., trend analysis, outbreak surveillance, AMR risk assessment) as research questions paired with explicit validation strategies. The core principle is that analytical outputs are evaluated, not asserted.
Validation Strategy: Text-derived signals are systematically compared against independent reference data from the WHO Global Health Observatory (GHO) and GLASS. Construct validity is assessed via cross-sectional comparisons (e.g., Spearman correlation) between literature signals and measured resistance rates (e.g., MRSA bloodstream infections).

Key Contributions
The paper outlines four principal contributions:

Validation as the Primary Operation: VIGIL shifts the focus from information extraction to the continuous evaluation of intelligence signals against independent reference standards (WHO GHO/GLASS).
Open, Provenance-Complete Pipeline: The system records complete provenance for every extraction (source, method, model version, prompt version) and supports reproducible head-to-head benchmarking of rule-based vs. LLM-based extraction.
Validation-First Analytical Framework: The platform provides a knowledge graph and analytical modules explicitly designed as research questions with validation strategies, rather than just software features.
Empirical Evidence on Bias: Using the platform as a testbed, the authors demonstrate that literature volume is a poor proxy for AMR burden, transforming publication bias into a measurable research target.

Results
Using a snapshot of 2,075 documents from seven active sources (dominated by US CDC alerts), the system extracted:

31 distinct pathogens (including MRSA, VRE, CRE, CRAB, Candida auris).
12 resistance mechanisms and 23 antimicrobial agents.
62 countries and 1,060 knowledge graph relationships.

Validation Findings:

Construct Validity: When comparing raw AMR-related document counts against WHO GHO MRSA resistance data across 43 countries, there was virtually no association (Spearman's $\rho$ = 0.07).
Normalization: Normalizing document counts by overall publication volume improved the association only modestly ( $\rho$ = 0.16).
Interpretation: These results indicate that raw literature volume primarily reflects research activity rather than underlying disease burden. Publication bias is not merely a limitation but a quantifiable discrepancy between text-derived signals and independent surveillance data.
Outbreak Analysis: Literature-derived signals failed to discriminate countries experiencing officially reported outbreaks, which were concentrated in lower-resource regions with limited research output.

Significance and Claims
The authors position VIGIL not as a replacement for existing surveillance platforms, but as a validation-first research infrastructure that addresses the critical need for trustworthiness in automated surveillance.

Paradigm Shift: The paper argues that as Large Language Models make signal generation accessible, the primary scientific challenge shifts from generating signals to determining which are trustworthy.
Measurable Bias: By explicitly quantifying the discrepancy between text-derived signals and reference data, VIGIL transforms reporting bias into a continuous, testable research target.
Reproducibility and Extensibility: The platform is fully open-source (MIT License), archived with a DOI, and designed to support future clinician-annotated benchmarking, temporal forecasting, and lead-time evaluation as longitudinal data accumulates.
Modesty on Current State: The authors acknowledge that current analyses are proof-of-concept due to the "young" and unevenly distributed corpus. They state that trend detection, forecasting, and novelty identification are preliminary and expected to improve as historical data accumulates. They also note that reference data limitations (e.g., missing territories like Taiwan in WHO GHO) reflect constraints of the reference system, not the platform itself.

In conclusion, VIGIL provides an extensible framework for continuously evaluating emerging surveillance methodologies, advocating that validation against independent reference data must become a fundamental design principle for next-generation infectious disease intelligence systems.

The "Trust No One" Library

The Big Surprise: More Books ≠ More Monsters

Why This Matters (And Why It's Not a Magic Bullet)

The Takeaway

Technical Summary: VIGIL – A Validation-First Research Infrastructure for Global Infectious Disease Intelligence

More like this