Performance Characteristics of Reasoning Large Language Models for Evidence Extraction from Clinical Genomics Literature

This study demonstrates that reasoning-capable large language models can achieve high accuracy in automating guideline-constrained PS4 evidence extraction from clinical genomics literature, though performance varies by model and prompt, supporting their use in a hybrid workflow with expert oversight.

The Gist

Imagine you are trying to solve a massive medical mystery: Why do some people get sick with a specific genetic condition while others with the same genetic "typo" stay perfectly healthy?

To solve this, doctors need to find proof in thousands of old research papers. They are looking for a specific type of evidence called PS4. Think of PS4 as a "Gold Star" that says, "Hey, we found this genetic typo in 10 sick people, but zero healthy people. That's a strong clue!"

The Problem: The Human Bottleneck

Right now, finding these "Gold Stars" is like trying to find a needle in a haystack, but the haystack is made of 275 different books (scientific papers), and the needles are hidden in tiny paragraphs. Human experts have to read every single word, count the sick people, check their family trees, and make sure the rules are followed. It's slow, exhausting, and creates a huge backlog.

The Experiment: Can AI Do the Reading?

The researchers asked: "Can smart computer brains (AI) do this reading for us?"

They didn't just use any AI; they tested the newest, most "reasoning-capable" models (think of them as the top students in the class who are good at logic, not just memorizing facts). They gave five different AI models a test:

1. The Search: "Find the specific genetic typo in this paper."
2. The Count: "How many sick people with this typo are mentioned here, and does it fit the strict rules?"

They compared the AI's answers against a "Truth Set"—a list of answers already graded by a team of human experts.

The Results: The AI Report Card

Here is how the AI students performed:

• Finding the Needle (Variant Detection): The AI was amazing at this. It found the genetic typos in the text with 93% to 98% accuracy. It was like a super-powered metal detector that rarely missed a signal.
• Counting the Gold Stars (PS4 Evidence): This was the tricky part. The AI had to not just count people, but understand why they were counted (e.g., "Are these people from the same family? Do they have the right symptoms?").
  • Top Performers: Gemini 2.5 Pro and GPT-5 were the valedictorians, getting the count right 90%+ of the time.
  • Middle Pack: o3 did well (86%), while o4-mini and Claude Sonnet 4 struggled a bit more (73-79%).

Where Did They Stumble?

The AI didn't fail because it couldn't read; it failed because it sometimes got confused by the rules.

• The Metaphor: Imagine a robot trying to follow a recipe. It can perfectly chop the onions (find the text), but if the recipe says "only use onions from the red basket," the robot might accidentally grab a green one if it doesn't understand the nuance of the family history or the specific symptoms.
• The Fix: The researchers tried changing the instructions (prompts) given to the AI. For most models, clearer instructions made them smarter. However, for one model (Claude), the new instructions actually made it worse, proving that every AI has a different "personality" and needs different ways of being talked to.

The Conclusion: The "Human-in-the-Loop" Team

The big takeaway isn't that AI will replace doctors tomorrow. Instead, think of it as a super-efficient intern.

• The Workflow: The AI does the heavy lifting first. It scans the papers, finds the typos, and does a rough count of the sick people. It highlights the most promising evidence.
• The Safety Net: A human expert then steps in to double-check the AI's work, especially for the tricky parts where the rules are complex.

In short: These reasoning AI models are powerful tools that can speed up the process of finding genetic clues by a huge margin, but they still need a human supervisor to catch the subtle mistakes. It's a team sport, not a solo act.

Technical Summary

1. Problem Statement

The implementation of Genomic Medicine relies heavily on genetic variant curation, a process that determines the pathogenicity of genetic variants. A critical component of this process is the PS4 evidence code (per ACMG/AMP guidelines), which requires comparing the prevalence of a variant in affected individuals (probands) versus healthy controls.

Currently, extracting this evidence from scientific literature is a manual bottleneck. Curators must read through thousands of publications to identify specific variants and count independent probands that meet strict guideline criteria (e.g., specific phenotypes, family structures). This study addresses the need to automate this process using reasoning-capable Large Language Models (LLMs) to determine if they can reliably extract guideline-constrained evidence.

2. Methodology

The researchers designed a rigorous benchmarking framework to evaluate the performance of LLMs in a clinical context.

• Truth-Set Construction: An expert-curated dataset was assembled containing 281 publication-variant pairs derived from 275 peer-reviewed publications. This set covered 58 genes and 128 distinct variants, serving as the ground truth for evaluation.
• Model Selection: Five state-of-the-art LLMs were selected to represent different architectural classes:
  • Frontier-scale: Gemini 2.5 Pro, GPT-5.
  • Reasoning-optimized: o3, o4-mini.
  • Efficiency-oriented: Claude Sonnet 4.
• Experimental Setup:
  • Inputs: Identical inputs were provided to all models.
  • Prompting: A unified prompt template was used, constrained by a specific JSON schema to ensure structured output.
  • Tasks:
    1. Variant Detection: Determining if a prespecified variant was correctly identified in the text.
    2. PS4 Proband Counting: Counting the number of independent, PS4-eligible probands based on ACMG/AMP and ClinGen Variant Curation Expert Panel (VCEP) guidance.
• Evaluation Metrics:
  • Task 1: Accuracy of variant detection.
  • Task 2: Exact-count concordance (the model's count must exactly match the expert truth-set count).
  • Secondary Analysis: Prompt sensitivity, error mode classification, and output variability.

3. Key Contributions

• First Comprehensive Benchmark: This study provides one of the first systematic evaluations of reasoning-optimized LLMs specifically for the complex, guideline-driven task of clinical variant curation (PS4 evidence extraction).
• High-Fidelity Truth-Set: The creation of a large, expert-curated dataset (281 pairs) specifically for benchmarking LLMs against clinical guidelines fills a gap in existing literature.
• Granular Error Analysis: The study moves beyond simple accuracy to analyze why models fail, specifically identifying failures in applying complex clinical logic (phenotype evaluation, family structure) rather than simple information retrieval.
• Prompt Sensitivity Insights: The research highlights that prompt engineering is not "one-size-fits-all"; refinements that helped some models degraded the performance of others (e.g., Claude Sonnet 4).

4. Key Results

• Variant Detection (Task 1): All models demonstrated high proficiency in identifying the presence of a variant within a publication, achieving accuracy rates between 93.6% and 97.9%.
• Proband Counting (Task 2): Performance varied significantly based on the model's reasoning capabilities:
  • Top Performers: Gemini 2.5 Pro achieved the highest exact-count concordance at 91.1%, followed closely by GPT-5 at 90.0%.
  • Mid-Tier: o3 achieved 86.5%.
  • Lower Tier: o4-mini scored 79.4%, and Claude Sonnet 4 scored 73.0%.
• Error Modes: The primary source of failure was not information retrieval but the application of clinical guidelines. Models struggled with nuanced logic, such as correctly evaluating phenotypes and determining family structures to count independent probands.
• Prompting Impact: While prompt refinements improved concordance for most models, they negatively impacted Claude Sonnet 4, suggesting that model-specific prompting strategies are necessary for optimal clinical deployment.

5. Significance and Conclusion

The study concludes that reasoning-capable LLMs are viable tools for automating guideline-based PS4 evidence extraction, achieving high concordance with expert human curators. However, the technology is not yet fully autonomous due to model-dependent performance and specific failure modes in complex guideline application.

Practical Implication: The authors advocate for a hybrid workflow in clinical settings. In this model, LLMs would serve as a force multiplier to accelerate the initial evidence extraction and counting, with expert human escalation reserved for reviewing model outputs, particularly in cases involving complex guideline interpretation. This approach promises to significantly reduce the manual bottleneck in genomic medicine without compromising the rigor required for clinical decision-making.