modFDR: a rigorous method to evaluate the reliability of nanopore sequencing for detecting DNA modifications in real applications

This paper introduces modFDR, a rigorous framework utilizing negative controls to evaluate nanopore sequencing reliability, revealing that while the technology effectively detects abundant DNA modifications, it suffers from significant false-positive rates for rare modifications and false negatives for 5mCpG, necessitating its adoption in future studies and a shift toward prioritizing abundant targets in biomedical applications.

The Gist

The Big Picture: A High-Tech Detective with a "False Alarm" Problem

Imagine Nanopore Sequencing as a super-smart, high-tech detective. This detective can read the genetic code of life (DNA) directly, without needing to copy or chemically alter it first. It's famous for being able to spot tiny chemical "stickers" (modifications) on the DNA letters that act like switches, turning genes on or off. These stickers tell the story of how our cells work, how we age, and how diseases like cancer develop.

However, this paper reveals a critical flaw in how we trust this detective: It often sees things that aren't there.

The authors, led by Dr. Gang Fang, introduce a new rulebook called modFDR. Think of this as a "Reality Check" system. They argue that before we believe the detective's report, we must rigorously test it to see how many "false alarms" it generates, especially when the clues are rare.

---

The Core Problem: The "Crowded Room" vs. The "Empty Room"

To understand the problem, imagine the detective is trying to find a specific type of person in a crowd.

1. The High-Abundance Case (The Easy Job):
   Imagine looking for 5mC (a common DNA sticker) in a mammalian cell. It's like looking for people wearing red hats in a stadium full of 10,000 people, where 400 of them are actually wearing red hats.

   • Result: The detective is great at this. Even if they make a few mistakes, the sheer number of real red hats means the mistakes don't ruin the picture. The "False Discovery Rate" (FDR) is low.

2. The Low-Abundance Case (The Hard Job):
   Now, imagine looking for 6mA or 5hmC (rare DNA stickers) in a human cell. It's like looking for a specific type of rare blue hat in a stadium where almost no one is wearing it. Maybe only 1 or 2 people have it.

   • Result: The detective gets confused. Because the "blue hat" signal is so faint, the detective starts seeing blue hats on people who are actually wearing gray hats.
   • The Danger: If the detective says, "I found 50 blue hats!" but there were actually only 2 real ones, the other 48 are false alarms. If scientists trust this report, they might spend years studying a "rare disease" that doesn't actually exist, just because the detective was hallucinating.

The "ModFDR" Solution: The Negative Control Test

The paper proposes a new way to test the detective, called modFDR.

The Analogy: The "Blank Canvas" Test
Before the detective goes out to solve a crime, you give them a piece of paper that you know is completely blank (a negative control).

• If the detective says, "I see a drawing of a cat on this blank paper," you know immediately that the detective is prone to hallucinations.
• In the study, the authors used Whole Genome Amplified (WGA) DNA. This is DNA that has been copied so many times that all the original chemical stickers are wiped away. It is the "blank canvas."
• They found that even on this blank canvas, the detective still claimed to see stickers. This proved that the detective has a built-in "noise" problem.

The "Confounding" Confusion
The paper also found that the detective gets confused when two different types of stickers look similar.

• The Analogy: Imagine trying to tell the difference between a Red Hat (5mC) and a Pink Hat (5hmC).
• In a room full of Red Hats (like in human cells), the detective sometimes mistakes a Red Hat for a Pink Hat.
• The authors showed that if you don't account for this confusion, you might think you found a "Pink Hat" (a rare biological event), when it was actually just a "Red Hat" that the detective misidentified.

What They Found: A Reality Check on Recent Tech

The authors tested the very latest software updates (DORADO v5.2.0) from the company that makes the sequencing machines.

• Good News: The new software is slightly better at not seeing "ghosts" (false positives) in empty rooms.
• Bad News: The new software is now missing real clues (false negatives). It's so afraid of making mistakes that it sometimes ignores real stickers.
• The Verdict: The detective is still not ready to be trusted alone for finding the rarest, most elusive stickers (like 6mA in humans or 5hmC in blood cells).

The Takeaway: Don't Trust the Detective Blindly

The authors are not saying "Stop using Nanopore sequencing." They are saying: "Stop trusting the raw numbers without a reality check."

1. For Common Stickers (5mC in CpG sites): The technology is reliable. It's like a good weather forecast for rain; it works well.
2. For Rare Stickers (6mA, 5hmC in most cells): The technology is currently unreliable. It's like a weather forecast predicting a hurricane in a desert. It's likely a false alarm.

The Final Advice:
If you are a scientist or a doctor using this technology:

• Use the "modFDR" framework: Always run your "blank canvas" tests (negative controls) to see how many false alarms your specific sample generates.
• Be skeptical of rare findings: If a study claims to find a rare DNA modification in a human cell using only Nanopore data, ask: "Did they check for false alarms?"
• Wait for better tools: Until the software gets better at distinguishing between "Red Hats" and "Pink Hats" without hallucinating, we should prioritize using this technology for abundant modifications, not rare ones.

In short: Nanopore sequencing is a powerful tool, but without the "modFDR" safety net, it can lead us down a rabbit hole of fake discoveries.

Technical Summary

1. Problem Statement

Nanopore sequencing offers the unique ability to directly detect DNA modifications (e.g., 5mC, 5hmC, 6mA) without chemical conversion. However, the reliability of these calls is highly dependent on the abundance of the modification in the sample.

• The Core Issue: Existing computational tools are often trained on datasets with balanced ratios of modified/unmodified bases (e.g., 1:1), which do not reflect physiological reality where many modifications (like 5hmC in non-brain tissues or 6mA in mammals) are extremely rare.
• Consequence: In low-abundance contexts, standard detection methods produce a high rate of false positives (FPs). Relying on arbitrary probability thresholds (e.g., $P_{mod} > 0.5$) leads to spurious biological conclusions, as seen in past debates regarding 6mA in mammals and mtDNA methylation.
• Gap: There is a lack of a rigorous framework to estimate the False Discovery Rate (FDR) in real-world applications where modification abundance varies significantly across cell types and genomic contexts.

2. Methodology: The modFDR Framework

The authors propose modFDR, a framework designed to rigorously evaluate modification detection reliability by integrating rationally designed negative controls with FDR analysis.

• Negative Controls:
  • General Background: Whole Genome Amplification (WGA) samples, which are essentially modification-free, serve as the baseline for technical false positives.
  • Confounding Modifications: To address cases where abundant modifications mimic rare ones (e.g., abundant 5mC mimicking 5hmC), the authors used bacterial strains with well-characterized, near-100% 5mC methylation but zero 5hmC. This isolates false positives caused by cross-talk between modification types.
• FDR Calculation:
  • $FDR = N_{fp} / (N_{tp} + N_{fp})$
  • $N_{fp}$ (False Positives) is inferred from the negative control samples (WGA or bacteria).
  • $N_{tp}$ (True Positives) is derived from the native sample.
  • This allows for the estimation of reliability at specific abundance levels, rather than relying on a single global threshold.
• Benchmarking:
  • Data was generated using the latest Oxford Nanopore Technologies (ONT) R10.4.1 flow cells.
  • Multiple DORADO base-calling models were tested (v4.2.0, v4.3.0, and the latest v5.2.0 as of Dec 2025).
  • Orthogonal Validation: Results were compared against gold-standard enzymatic methods: EM-seq (detects 5mC + 5hmC) and ACE-seq (specifically detects 5hmC).

3. Key Results

A. Abundance Dictates Reliability

• High-Abundance Modifications: Nanopore sequencing is highly reliable for detecting 5mC at CpG sites in mammalian genomes (e.g., mouse prefrontal cortex, mPFC), where abundance is high (~4-5%). FDR remains low (<5%) even at moderate confidence thresholds.
• Low-Abundance Modifications:
  • 5mC at CpH sites: In most cell types (except specific brain regions), 5mCpH is rare. FDR analysis showed that calls are dominated by false positives (FDR > 90% in low-abundance simulations).
  • 5hmC: While reliable in brain tissue (high abundance), 5hmC detection in lymphoblastoid cells (hLCL) or blood (hPBMC) yielded FDRs approaching 1.0 (near-total false positives), even with high confidence thresholds.
  • 6mA: In mammalian cells, 6mA calls were essentially 100% false positives, confirming that current nanopore methods cannot reliably detect 6mA in mammalian genomes without rigorous FDR filtering.

B. The "Confounding" Effect

• The study revealed that abundant 5mC can be misclassified as 5hmC by the software.
• Using bacterial strains with 100% 5mC but no 5hmC, the authors found that DORADO miscalled ~0.11–3.94% of 5mC as 5hmC.
• In human hLCL cells, the estimated 5hmC levels based on WGA controls suggested 0.0046–0.21% 5hmC/C. However, when accounting for the confounding 5mC signal (using bacterial controls), the FDR for 5hmC in hLCL was revealed to be **1.0**, indicating that the detected 5hmC was almost entirely false positives derived from 5mC.

C. Evaluation of Newer Models (v5.2.0)

• The latest DORADO model (v5.2.0) showed improved performance in reducing intrinsic 5hmC bias in WGA samples (lower FDR in negative controls).
• However, new issues emerged:
  • Increased False Negatives: The v5.2. model exhibited a significant increase in false negatives for 5mCpG, underestimating global methylation levels compared to gold standards.
  • Persistent Systematic Errors: Strand-biased false positives (e.g., 5hmC calls at TGCGNN motifs) and confounding effects from high 5mC abundance persisted.
  • Context Failure: Both v4.3.0 and v5.2.0 models struggled with CpH contexts, showing high discordance with enzymatic methods.

D. Specific Artifacts

• Motif-Specific Errors: False positives for 5hmC frequently clustered at specific motifs (e.g., TGCGNN) and exhibited strand bias (5mC on one strand, 5hmC on the other), suggesting machine learning ambiguity in signal interpretation.
• True Positives in Clusters: Conversely, true positive 5mCpH events were found to cluster in specific repeat regions (e.g., (CA)n repeats in brain tissue), suggesting that local density can be a heuristic for distinguishing true signals from noise.

4. Key Contributions

1. modFDR Framework: Introduced a standardized, rigorous method to calculate FDR for DNA modification detection using tailored negative controls (WGA and confounding-modification-free bacteria).
2. Quantification of Abundance Bias: Demonstrated that detection reliability is not an intrinsic property of the sequencer but is strictly a function of the modification's biological abundance in the specific sample.
3. Re-evaluation of 6mA and 5hmC: Provided strong evidence that 6mA is undetectable in mammalian genomes via current nanopore methods and that 5hmC detection in low-abundance tissues is currently unreliable due to confounding 5mC signals.
4. Model Benchmarking: Critically assessed the latest DORADO models, highlighting that while newer models reduce some false positives, they introduce new false negatives and fail to resolve complex context-dependent errors.

5. Significance and Recommendations

• Paradigm Shift: The paper argues against the use of arbitrary probability cut-offs. Instead, researchers must apply modFDR analysis specific to their sample's expected modification abundance.
• Biological Implications: Many previous studies claiming the presence of rare modifications (like 6mA in humans or widespread mtDNA methylation) may have been misled by unquantified false positives.
• Future Directions:
  • Prioritize Abundant Targets: Nanopore sequencing should currently be prioritized for mapping abundant modifications (e.g., 5mCpG) rather than rare ones.
  • Model Training: Future algorithm development must use training datasets that reflect physiologically relevant modification ratios, not just synthetic 1:1 mixes.
  • Comprehensive Controls: Method development must include negative controls that account for confounding modifications (e.g., testing 5hmC detection in the presence of high 5mC).
  • Orthogonal Validation: For low-abundance modifications, nanopore data should be validated against orthogonal methods (EM-seq, ACE-seq, LC-MS/MS) or used in conjunction with enrichment strategies (e.g., antibody pull-down) to increase signal-to-noise ratios.

In conclusion, modFDR provides a critical "reality check" for the field, urging a move from optimistic detection claims to rigorous, context-aware statistical validation to prevent the propagation of erroneous biological interpretations.