💻 computer science

Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews

This paper demonstrates that integrating an auxiliary BERT+GCN classifier to provide structured uncertainty signals significantly improves the efficiency and recall of LLM-assisted systematic review screening, identifying a Pareto-optimal strategy that routes only "MAYBE" cases to the LLM while revealing that current instruction-tuned models lack the ability to self-triage their own decisions.

Original authors: Arya Rahgozar, Pouria Mortezaagha

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arya Rahgozar, Pouria Mortezaagha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medical research, the most trusted way to understand a treatment is the systematic review. This is a massive, rigorous effort where experts gather every study ever published on a specific topic, read them, and decide which ones are good enough to count toward the final answer. The first and most grueling step of this process is screening. Researchers must look at thousands of titles and abstracts—short summaries of papers—and decide which ones to read in full and which to discard. It is a task that consumes hundreds of hours of expert time and often feels like searching for a needle in a haystack. Recently, scientists have turned to large language models, the same powerful computer programs that can write essays and answer questions, to help with this screening. These models can read quickly and suggest which papers to keep, but they have a significant flaw: they do not know when they are unsure. When a computer program says "keep this paper," it might be absolutely certain, or it might be guessing, and the human reviewer has no way to tell the difference. This uncertainty creates a bottleneck, because reviewers cannot easily know which of the computer's suggestions deserve a second look by a human.

A team of researchers at the University of Ottawa and the Ottawa Hospital Research Institute set out to solve this problem by giving the computer a second opinion. They built a specialized system that acts as a safety net for the main screening program. This system, which combines a text-reading engine with a network that understands how papers are connected to one another, reads every paper before the main computer does. It then assigns a simple label: keep, discard, or maybe. The "maybe" label is the most important part; it signals that the system is confused or that the paper sits on the edge of being relevant. The researchers then tested different ways to hand this "maybe" signal to the main computer program. They wanted to see if telling the main program, "Hey, this one is tricky, here is a note from our safety net," would help it make better decisions without wasting time or money. They ran this experiment on eight different sets of medical data, involving thousands of papers, to see which method worked best.

The results showed that simply giving the main computer the "maybe" warning was the most effective strategy. When the researchers told the computer to pay extra attention only to the papers flagged as "maybe," the system became more accurate at finding the right papers while costing almost the same amount of money as doing nothing special. In fact, this targeted approach was the best balance of performance and cost. It achieved the highest rate of finding the correct papers and did so with only a tiny increase in computing resources. In contrast, the researchers found that telling the computer about the safety net for every single paper, even the easy ones, did improve accuracy slightly, but it cost significantly more to run. The most surprising finding, however, was about how the computer handles its own uncertainty. The researchers tried a method where the computer would read a paper, say "I'm not sure," and then immediately read it again with the safety net's notes to see if it would change its mind. This approach failed completely. Even when the computer admitted it was unsure and was given the extra help, it never changed its original decision. It stuck to its first guess, even when that guess was likely wrong. This suggests that these computer programs cannot effectively re-evaluate their own uncertainty on the fly; they need a human to step in when they are confused, rather than trying to fix the problem themselves.

The study also revealed that the complex safety net they built could be simplified. The system used two different tests to decide if a paper was "maybe," but the researchers found that one of those tests never actually triggered on the data they used. The entire safety net functioned just as well if it relied on a single, simple check of how confident the system was in its answer. This means that other researchers can build similar tools without needing the complex, multi-part machinery originally designed. Furthermore, the benefit of this safety net did not depend on how powerful the main computer program was. Whether they used a newer, more advanced version of the software or an older one, the safety net provided the same boost in accuracy. This implies that the safety net acts as a reliable, independent helper that works alongside the main program, rather than just filling in gaps for a weaker model.

For the teams that conduct these medical reviews, the path forward is clear. They should not rely on the screening computer to recognize its own confusion and ask for help. Instead, they should use a separate, simpler system to flag the difficult papers and then feed those specific flags to the main computer. This approach captures the benefits of having a second opinion without the high cost of checking every single paper twice. The researchers have made all their code, data, and tools available to the public, allowing other scientists to replicate the work and apply these methods to their own reviews. By understanding exactly where the computer needs help and where it does not, medical researchers can speed up the process of finding life-saving evidence, ensuring that the right studies are reviewed without overburdening the human experts who do the final work.

Technical Summary: Auxiliary Uncertainty Signals for LLM-Assisted Systematic Review Screening

1. Problem Statement

Systematic reviews are the gold standard for evidence synthesis in healthcare, yet the title–abstract screening phase remains a major bottleneck, often requiring hundreds of hours of expert time. While Large Language Models (LLMs) have demonstrated promise as automated screeners, they suffer from a critical limitation: their decisions lack calibrated uncertainty. An LLM outputting "INCLUDE" with hedging language (e.g., "this might be relevant") is indistinguishable in downstream action from a certain decision. This absence of structured uncertainty prevents reviewers from efficiently routing ambiguous records to human verification and leaves the LLM without a mechanism to request additional evidence when its assessment is ambiguous.

2. Methodology

2.1 Experimental Design

The study evaluates five prompt-delivery conditions across eight drug-class datasets from the Cohen (2006) benchmark collection. The evaluation utilizes 3 random seeds and 5-fold stratified cross-validation, resulting in 600 fold-level experimental runs.

2.2 The Auxiliary Uncertainty Signal (BERT+GCN)

A hybrid BERT+GCN (Graph Convolutional Network) model is trained per fold to generate structured uncertainty signals. The pipeline represents papers using:

Content: PubMedBERT embeddings of the title and abstract.
Relational Context: GCN embeddings learned over a $k$ -nearest-neighbor similarity graph with optional citation edges, using Relational Graph Convolution (RGCNConv).

The model classifies papers into three states: INCLUDE, EXCLUDE, or MAYBE. The MAYBE state is triggered by two complementary spectral tests:

Algebraic Radical Test: Flags uncertainty if the gap between include and exclude logits is below a threshold ( $|P(\text{Include}) - P(\text{Exclude})| < \epsilon$ , where $\epsilon=0.15$ ).
Categorical Paradox Test: Flags uncertainty if the cosine similarity between BERT (content) and GCN (relational) embeddings is low ( $\cos(b, g) < \delta$ , where $\delta=0.0$ ), indicating a disagreement between textual content and citation neighborhood.

2.5 Prompt-Delivery Conditions

The study investigates how spectral context is injected into the LLM prompt:

Baseline: Title and abstract only.
Full Spectral: Title, abstract, and full spectral context (label, confidence gap, model confidence, paradox type) for every paper.
Decision Only: Title, abstract, and the spectral label (INCLUDE/EXCLUDE/MAYBE) without numerical scores.
MAYBE Only: Full spectral context appended only when the BERT+GCN flags the paper as MAYBE; baseline prompt for all others.
Two-Pass: A baseline prompt is sent first. If the LLM response contains hedging language or low confidence, a second pass sends the full spectral context to "reconsider" the decision.

2.6 Models and Metrics

LLMs: Primary screener is gpt-5.4-mini (snapshot Jan–Apr 2026); a cross-model pilot uses gpt-4.1-mini.
Metrics: Recall, AUC-ROC (using LLM confidence scores), F1, Work Saved over Sampling at 95% recall (WSS@95), and token cost.
Analysis: A pre-registered cross-model pilot tests signal portability. A per-paper ablation (20,796 observations) analyzes the contribution of the dual paradox test.

3. Key Results

3.1 Efficiency Gains with Full Context

Delivering full spectral context yields statistically significant improvements over the baseline:

F1 Score: +0.011 improvement ( $p=0.008$ ).
WSS@95: +0.050 improvement ( $p=0.039$ ), meaning a 5% reduction in the screening burden while maintaining 95% recall.
Cost: This comes at a 1.28× token-cost premium.
Recall: Recall is preserved across all conditions ( $p \ge 0.12$ ), indicating no safety trade-off.

3.2 The Pareto-Optimal Strategy: MAYBE-Only

The MAYBE-only routing strategy is identified as the most cost-effective configuration:

Performance: Achieves the highest mean recall (0.92) and AUC-ROC (0.54) of all conditions.
Cost: Incurs only a 1.05× baseline cost (one-sixth of the overhead of full-context delivery).
Statistical Note: While full-context delivery shows significant gains in F1 and WSS@95, MAYBE-only does not differ significantly from baseline or full-context on these specific metrics but dominates the cost–quality frontier.

3.3 Failure of LLM Self-Triage (Two-Pass)

The Two-Pass design, intended to let the LLM self-triage uncertain records, yielded a decisive negative result:

Escalation: 22.2% ± 8.8% of records triggered a second pass due to detected hedging language.
Flip Rate: 0.0% of escalated papers changed their decision on the second pass.
Implication: Current instruction-tuned LLMs cannot effectively self-triage; they anchor on their initial decision even when presented with new uncertainty signals.

3.4 Cross-Model Portability

A pilot study comparing gpt-4.1-mini and gpt-5.4-mini on three datasets showed an identical +0.8% recall uplift from full spectral context for both generations. This suggests the BERT+GCN signal acts as a stable, model-agnostic auxiliary classifier rather than a capability-dependent crutch that only benefits weaker models.

3.5 Classifier Ablation Findings

An ablation study on the spectral engine revealed that the Categorical Paradox Test (cosine similarity) did not fire on any paper in the 8-dataset corpus at the default threshold ( $\delta=0.0$ ), as all BERT/GCN cosine similarities were positive. Consequently, the dual-test reduces empirically to a single Logit-Gap criterion ( $|P(\text{Include}) - P(\text{Exclude})| < 0.15$ ) for this specific benchmark.

4. Contributions and Significance

The paper claims the following contributions and significance:

Practical Guidance for Cost-Efficient Screening: It identifies MAYBE-only routing as the recommended configuration for cost-sensitive deployments. It captures the spectral benefit (high recall and AUC-ROC) at near-baseline API cost, avoiding the 28% premium of full-context delivery.
Quantified Benefit of Spectral Context: It provides evidence that structured uncertainty from an auxiliary classifier significantly improves screening efficiency (F1 and WSS@95) without compromising recall.
Definitive Negative Result on Self-Triage: It provides "decisive empirical evidence" that instruction-tuned LLMs cannot self-triage. The 0% flip rate in the two-pass design rules out a class of pipeline designs that rely on LLMs to recognize their own uncertainty and benefit from targeted re-evaluation.
Stable Auxiliary Signal: The cross-model pilot supports the interpretation of the BERT+GCN output as a stable external signal that transfers across LLM generations, rather than a fix for reasoning gaps in weaker models.
Reproducible Implementation: The authors contribute a fully reproducible 600-run benchmark, a deterministic analysis script, and a publicly available pipeline that can be replayed in under one hour from cached responses.

5. Limitations and Scope

The authors explicitly note several scope conditions:

Domain: Results are based on biomedical drug-class reviews; generalization to social sciences or environmental science is untested.
Model Coverage: The main matrix uses gpt-5.4-mini; results may differ for larger frontier models (e.g., GPT-4o) or open-source alternatives, particularly regarding numerical reasoning capabilities.
Thresholds: Spectral thresholds were fixed and not tuned per dataset; adaptive thresholds (e.g., percentile-based cosine thresholds) remain a future direction.
Human-in-the-Loop: The study evaluates against ground-truth labels, not human reviewer decisions under time pressure.
Deterministic Output: Temperature 0 was used, preventing analysis of inter-run variance.

In conclusion, the paper argues that while LLMs are effective screeners, they benefit most from targeted, cost-efficient integration of auxiliary uncertainty signals (specifically MAYBE-only routing) rather than relying on their own self-assessment or blanket context injection.