Large language models enable consensus-level interpretation in metagenomic diagnostics

Technical Summary: Large Language Models Enable Consensus-Level Interpretation in Metagenomic Diagnostics

Problem Statement
Metagenomic next-generation sequencing (mNGS) offers broad-spectrum pathogen detection but faces a critical bottleneck in clinical deployment: the interpretation of results. Unlike quantitative assays with fixed thresholds, mNGS generates complex data containing true pathogens, background commensals, and environmental contaminants. Determining clinical relevance requires expert adjudication that synthesizes sample type, organism identity, relative abundance, contamination risk, and patient history. This process is currently manual, labor-intensive, difficult to scale, and prone to inter-reviewer variability and cognitive biases (e.g., anchoring). Furthermore, the reliance on human panels hinders systematic quality assurance, regression testing of computational pipelines, and the optimization of analytical parameters.

Methodology
The authors present a structured framework, META-GPT, which formalizes expert adjudication by combining a tiered computational filtering strategy with open-weight Large Language Models (LLMs).

1. Assay and Stratification: A short-read metagenomic assay was developed for sterile-site specimens (cerebrospinal fluid and ocular fluid). A tiered filtering scheme assigns taxonomic detections to three tiers based on confidence and abundance:

   • Primary: Above-threshold, high-abundance, high-confidence detections.
   • Secondary: Below-threshold, low-abundance, medium-high confidence.
   • Target: High-priority organisms (e.g., specific viral families) regardless of abundance.
     This system integrates negative control comparisons, prevalence-based contamination filters, domain-specific abundance thresholds, and ensemble classifier congruence to reduce false positives while retaining sub-threshold signals.

2. Diagnostic Classifiers: The core innovation is the use of locally deployed, open-weight reasoning models (Qwen3 series, specifically a 32B parameter model with 8-bit quantization) integrated into a decision-tree architecture.

   • Architecture: The models traverse a configurable decision tree, first evaluating primary tier detections, then secondary, then target tiers.
   • Prompting: At each node, the model is prompted to classify the sample as infectious or non-infectious and, if infectious, to select a single pathogen candidate.
   • Context: The models were evaluated both with and without short clinical notes (synthetic for spike-ins, real for clinical cases) to assess the impact of clinical context on reasoning.
   • Implementation: Inference was performed on consumer-grade GPUs using the candle stack, ensuring data privacy by keeping all processing local.

3. Evaluation Strategy:

   • Validation Cohort: A dataset of 96 samples (87 clinical/spike-in, 9 controls) from the META-GP study (Victoria, Australia, 2024-2025) was used.
   • Human Benchmark: Ten untrained clinical microbiologists adjudicated the stratified results without clinical notes to establish a baseline of individual variability and a consensus benchmark.
   • Model-Assisted Adjudication: Reviewers repeated the task six months later with clinical notes and a pre-populated classifier diagnosis to measure the impact of model assistance on human performance.
   • Systematic Benchmarking: The automated system was used to perform ablation studies on contamination controls, host-genome handling in reference databases, and regression testing of taxonomic classifier updates (Metabuli).

Key Results

• Performance vs. Human Consensus: In the absence of clinical notes, the best-performing classifier (Qwen3-32B) achieved 94.4% sensitivity and 95.4% specificity via consensus across replicates. This performance was comparable to the consensus of ten human reviewers (97.2% sensitivity, 97.7% specificity) and significantly outperformed individual untrained reviewers (average 77.9% sensitivity, 97.3% specificity).
• Impact of Clinical Context: Providing clinical notes to the classifier improved performance to 97.2% sensitivity and 100% specificity, effectively reaching the performance ceiling established by the human consensus. The notes primarily served to disambiguate plausible-but-spurious detections (e.g., commensal skin organisms in CSF).
• Human-in-the-Loop: When reviewers were assisted by the classifier's pre-populated diagnosis, individual performance rose to match the consensus ceiling (95.1% sensitivity, 99.1% specificity), and inter-reviewer variability was markedly reduced. However, the study noted that this "anchoring" effect could lead to errors if the classifier were incorrect, as some strong-performing reviewers incorrectly overrode correct classifier calls.
• Systematic Optimization: Automated adjudication enabled rigorous benchmarking. Disabling contamination controls caused specificity to collapse (95.4% to 88.4%), while clinical notes largely restored specificity. Conversely, removing the prevalence filter reduced sensitivity, which clinical notes could not recover. The study also demonstrated that updating the taxonomic classifier (Metabuli v1.0.5 to v1.1.1-dev with syncmer support) did not significantly alter diagnostic performance, validating the utility of automated regression testing.

Significance and Claims
The paper claims to demonstrate that diagnostic classifiers can emulate expert adjudication with high fidelity, providing a scalable, standardized alternative to the resource-intensive human review panels currently required for mNGS.

• Formalizing Reasoning: The study establishes that LLMs, when guided by structured decision trees and clinical context, can replicate the contextual reasoning of experts, moving beyond simple abundance thresholds to assess clinical plausibility.
• Scalability and QA: By replacing manual review with automated adjudication, the framework enables systematic benchmarking of computational parameters, contamination strategies, and pipeline updates—tasks that are impractical to scale with human experts.
• Consensus-Level Accuracy: The authors assert that a single, well-configured classifier can achieve performance comparable to a multi-expert consensus, potentially democratizing access to high-quality metagenomic interpretation outside of tertiary centers.
• Limitations and Future Work: The authors modestly note that the study was retrospective and limited to low-biomass samples. They emphasize that while classifiers can replicate expert reasoning, they do not replace the need for human oversight, particularly for emerging pathogens or unusual presentations. They suggest future work should focus on prospective multi-site trials and "human-in-the-loop" designs that mitigate the risks of automation bias.

In conclusion, the paper presents a proof-of-concept framework where open-weight LLMs, integrated with structured bioinformatic filtering, can serve as a robust, reproducible, and scalable engine for metagenomic diagnostic interpretation, addressing the primary barrier to the clinical adoption of mNGS.