Evaluation of somatic variant calling methods on high coverage tumour-only amplicon sequencing data in a clinical environment

This study evaluates the performance of six open-source somatic variant callers on high-coverage tumour-only amplicon sequencing data using a commercial reference standard, identifying FreeBayes, VarScan, MuTect2, and Pisces as the top-performing tools despite FreeBayes' tendency to generate more artifacts.

The Gist

Imagine you are a detective trying to find a few specific clues (mutations) hidden inside a massive library of books (DNA) from a cancer patient. The problem? You only have the patient's book (the tumor), and you don't have a "clean" copy of the book to compare it against (the healthy tissue). This is called "tumor-only" sequencing, and it's like trying to find typos in a manuscript when you don't have the original draft to check against.

To make this easier, scientists use a special magnifying glass called Next-Generation Sequencing (NGS). In this study, they used a specific type of magnifying glass called an Amplicon Panel, which zooms in very closely on just 47 important chapters (genes) known to be involved in cancer. Because they zoomed in so much, they got a super-high-definition picture (high coverage), allowing them to see even the tiniest, rarest clues.

The Big Question: Which Detective is Best?

There are many different software programs (called variant callers) that act as the "detectives" to find these clues. The authors of this paper wanted to know: Which detective is the most accurate when working alone with just the tumor sample?

They tested six famous detectives:

1. FreeBayes
2. MuTect2
3. Pisces
4. Platypus
5. VarDict
6. VarScan

To test them, they used a "Gold Standard" box of clues called HD789. This is a commercial sample where the scientists already know exactly what the clues are. It's like giving the detectives a test with an answer key. They ran the test three times (replicates) and even diluted the sample (making the clues harder to find) to see how well the detectives performed under pressure.

The Results: The Detective Report Card

Here is how the detectives fared, explained with some metaphors:

• FreeBayes (The Over-Enthusiastic Detective):
  This detective found the most clues. In fact, it found more than anyone else. However, it was a bit too eager. It often shouted, "I found a clue!" when it was actually just a smudge on the page (a false positive/artifact). It's great if you don't want to miss anything, but you have to be very careful to double-check its work because it makes a lot of mistakes.

• Platypus (The Conservative Detective):
  This detective was very quiet. It barely called any variants. It missed a lot of the real clues that the other detectives found. It was too strict and didn't want to take any risks.

• MuTect2, Pisces, VarScan, and VarDict (The Balanced Team):
  These four detectives were the stars of the show. They found almost all the real clues that were supposed to be there, without calling too many fake ones. They struck the perfect balance between being thorough and being accurate. They are the "Goldilocks" detectives—not too eager, not too shy.

• The Dilution Challenge:
  When the scientists made the sample weaker (the 1:4 dilution), it was like turning down the lights in the library. All the detectives struggled a bit more, but the "Balanced Team" (MuTect2, Pisces, etc.) still managed to find the most important clues, whereas the others missed more.

The "Snakemake" Pipeline

The researchers didn't just run these programs one by one; they built a conveyor belt (called a Snakemake pipeline) that automatically fed the data through all six detectives at once. This ensured that every detective was working with the exact same clean data, making the comparison fair.

The Takeaway for Real Life

In a hospital, doctors need to know which mutations a patient has to choose the right medicine. If a detective misses a clue, the patient might get the wrong treatment. If a detective invents a fake clue, the patient might get unnecessary treatment.

The main lesson from this paper is:
Don't rely on just one detective. The best strategy is to use a team approach. If you combine the results of the top performers (FreeBayes, MuTect2, Pisces, and VarScan) and only trust the clues that at least two of them agree on, you get the most reliable answer.

In short:

• FreeBayes finds everything but makes noise.
• Platypus is too quiet and misses things.
• The others (MuTect2, Pisces, VarScan, VarDict) are the reliable workhorses.
• The Solution: Use a combination of them to get the best results for cancer patients.

Technical Summary

1. Problem Statement

• Context: Amplicon-based targeted sequencing is a standard workhorse in clinical molecular diagnostics for profiling somatic mutations in tumors due to its cost-effectiveness and ability to achieve very high sequencing depths.
• Challenge: While numerous open-source somatic variant callers exist, their performance in tumour-only (without a matched normal sample) clinical settings using high-depth amplicon data is under-explored. Most existing comparisons focus on Whole Genome Sequencing (WGS) or Whole Exome Sequencing (WES) with matched normal-tumor pairs.
• Specific Gap: There is a lack of guidance on which variant callers perform best for detecting low-frequency somatic variants in highly heterogeneous tumors using clinical-grade amplicon panels, particularly regarding the trade-off between sensitivity (detecting true low-frequency variants) and specificity (avoiding artifacts/false positives).

2. Methodology

The authors developed a unified workflow to evaluate six widely used open-source variant callers under consistent conditions.

• Workflow (SSVCC): A Snakemake pipeline named SSVCC was created to integrate the following tools:

  • Preprocessing: FASTQC for quality control and FASTP for adapter/quality trimming.
  • Alignment: BWA-MEM2 for mapping reads to the hg38 reference genome.
  • Post-processing: Read grouping, coordinate sorting, and Base Quality Score Recalibration (BQSR) using GATK BaseRecalibrator (excluding known polymorphic sites from dbSNP and Gnomad).
  • Variant Calling: The pipeline executed six callers: FreeBayes, MuTect2, Pisces, Platypus, VarDict, and VarScan.
  • Downstream Analysis: Custom Python/R scripts were used for normalization, annotation (using Ensembl VEP), and visualization (violin plots, UpSet plots).

• Dataset:

  • Sample: HD789, a commercial FFPE (Formalin-Fixed Paraffin-Embedded) reference standard containing well-characterized SNVs, indels, and structural variants with allele frequencies (AF) ranging from 5% to 15%.
  • Sequencing: Performed on Illumina MiSeq and NextSeq 2000 using the AmpliSeq Focus panel (269 amplicons across 47 clinically significant genes).
  • Replicates: Three independent replicates (S1, S2, S3) and one 1:4 dilution sample (S-dil) to test detection limits.

• Evaluation Metrics:

  • Concordance between callers.
  • Performance on known variants (sensitivity).
  • Analysis of depth of coverage, variant allele frequency (VAF), and filtering flags (e.g., PASS vs. filtered).

3. Key Contributions

• Unified Benchmarking Framework: Introduction of the SSVCC pipeline, which standardizes the preprocessing and execution of six distinct variant callers, allowing for a fair, apples-to-apples comparison in a tumour-only context.
• Clinical Relevance: The study specifically addresses the "tumour-only" scenario common in clinical diagnostics where matched normal tissue is unavailable, a scenario where distinguishing somatic mutations from germline variants and artifacts is critical.
• Performance Characterization: Comprehensive evaluation of how different callers handle high-depth data, low-frequency variants, and complex indels in a targeted amplicon setting.

4. Key Results

• Variant Call Volume & Artifacts:

  • FreeBayes called the highest number of variants (2,433 total) but also included the most potential artifacts and complex indels. It detected the most variants across all samples but had a higher false-positive rate.
  • Platypus called the fewest variants (55) and had the lowest sensitivity for known variants.
  • MuTect2, Pisces, VarDict, and VarScan showed more conservative calling patterns with fewer artifacts.

• Concordance:

  • Only 609 of the 2,655 total unique variants were called by at least two different callers, indicating significant divergence in calling strategies.
  • FreeBayes and VarDict showed the highest concordance with each other, largely because both call a high volume of variants.

• Performance on Known Variants (Sensitivity):

  • FreeBayes successfully detected the highest number of known HD789 mutations across all replicates (except for one GNA11 mutation in the dilution sample).
  • Platypus performed the worst, detecting only 3 known mutations in standard replicates and 1 in the dilution sample.
  • MuTect2, Pisces, VarDict, and VarScan demonstrated comparable performance to one another, successfully identifying most known variants in standard replicates but showing reduced sensitivity in the 1:4 dilution sample.

• Impact of Dilution (Low Input):

  • Performance degradation was most evident in the 1:4 dilution sample. While FreeBayes maintained high sensitivity, it likely at the cost of increased noise. Other callers struggled to detect variants with lower allele frequencies in the diluted sample.

• Filtering:

  • Filtering strategies varied significantly. MuTect2, Pisces, Platypus, and VarDict applied specific filters (e.g., strand bias, orientation bias) that flagged many variants as non-PASS. FreeBayes and VarScan, under default parameters, lacked specific filters, resulting in nearly all calls being marked "PASS," which contributes to their higher false-positive rates.

5. Significance and Conclusion

• Clinical Recommendation: The study concludes that FreeBayes, VarScan, MuTect2, and Pisces offer the best overall performance for HD789 data. However, relying on a single caller is insufficient.
• Strategy for Clinical Labs: To achieve the most reliable call sets in a clinical environment, the authors recommend:
  1. Consensus Calling: Employing multiple variant callers and focusing on variants called by at least two tools (concordance).
  2. Context-Aware Filtering: Applying careful filtering and flexible depth/VAF thresholds, particularly distinguishing between known, clinically actionable variants and novel variants.
  3. Quality Control: Acknowledging that high-depth data can reveal low-frequency artifacts; therefore, understanding input quantity/quality metrics is essential for interpreting results, especially in diluted or low-biomass samples.

This paper provides a critical roadmap for clinical laboratories transitioning to or optimizing tumour-only NGS workflows, emphasizing that while high sensitivity is achievable, it requires a multi-caller approach and rigorous filtering to ensure diagnostic accuracy.