Benchmarking circRNA Detection Tools from Long-Read Sequencing Using Data-Driven and Flexible Simulation Framework

This study introduces a novel, open-source simulation framework to generate realistic Oxford Nanopore long-read datasets and presents the first comprehensive benchmark comparing three circRNA detection tools (CIRI-long, IsoCIRC, and circNICK-Irs), revealing their distinct performance profiles and limited overlap to guide future tool selection and development.

The Gist

Imagine your cell's genetic library (DNA) is like a massive cookbook. Usually, when the cell wants to make a dish (a protein), it copies a recipe from the book, cuts out the boring parts (introns), and stitches the good parts (exons) together to make a straight, linear instruction manual. This is how most of our genes work.

But sometimes, the cell gets creative. Instead of making a straight line, it takes the ends of a recipe and glues them together to form a circle. These are called circRNAs. Think of them as a recipe looped back on itself, like a bracelet or a donut. Because they are closed loops, they are super tough and don't fall apart easily, making them very stable. Scientists think these "donuts" might hold clues to diseases like cancer, so finding them is a big deal.

The Problem: Finding the Donuts in the Haystack

To find these circular donuts, scientists use a high-tech scanner called Oxford Nanopore sequencing. Unlike older scanners that chop the DNA into tiny, unreadable crumbs, this new scanner can read the whole "donut" in one go. That's great!

But here's the catch: The software (the "detectives") trying to find these donuts in the scanner data isn't very good yet. There are three main detective tools available (CIRI-long, IsoCirc, and circNICK-LRS), but no one knew which one was the best, or if they were even looking in the right places.

The Experiment: The "Fake Donut" Factory

To test these detectives, the authors of this paper built a virtual simulation factory.

Imagine you want to test a metal detector, but you don't want to dig up a real park and risk missing a real treasure. Instead, you build a sandbox where you bury exactly 7,500 fake gold coins (circRNAs) and 117,000 fake rocks (linear RNA) in a specific pattern. You know exactly where every coin is. This is your "Ground Truth."

The authors created a computer program that:

1. Looked at real biological data to understand what these "donuts" actually look like (how long they are, what they are made of).
2. Used a tool called NanoSim to generate millions of fake DNA reads that looked and sounded exactly like real data from the Nanopore scanner.
3. Ran the three detective tools on this fake data to see who found the most coins and who made the most mistakes.

The Results: The Three Detectives

Here is how the three tools performed, using our "Donut Detective" analogy:

1. IsoCirc: The "Sniper"

• Style: Extremely precise but very picky.
• Performance: When it said, "I found a donut!" it was almost always right (High Precision). However, it missed a huge number of donuts that were actually there (Low Recall).
• Weakness: It has a built-in size limit. It refuses to look for "giant donuts" (long circRNAs) and mostly ignores the weird, rare types.
• Best for: Researchers who need 100% confidence in a few specific findings and have limited computer power.

2. CIRI-long: The "Heavy Lifter"

• Style: A balanced approach, but it's a bit clumsy and hungry.
• Performance: It found more donuts than the Sniper, but it still missed some. It was the only one brave enough to find a specific rare type of donut called a "ciRNA."
• Weakness: It is extremely hungry. It eats up so much computer memory (RAM) that it can crash your computer if you aren't careful.
• Best for: Researchers who need a middle-ground solution and have powerful computers.

3. circNICK-LRS: The "Net Catcher"

• Style: Casts a wide net and catches almost everything.
• Performance: It found the most donuts, including the giant ones and the hard-to-find ones (High Recall). It was the most sensitive tool.
• Weakness: Because it casts such a wide net, it sometimes catches rocks thinking they are donuts (Lower Precision). It also struggles to describe the exact shape of the donut (the internal structure). It is also very slow, like a turtle.
• Best for: Researchers who want to find every possible donut, even if they have to double-check the results later.

The Big Takeaway

The study revealed a shocking truth: The three detectives rarely agreed with each other.

• If you used only IsoCirc, you might miss 90% of the donuts.
• If you used only circNICK-LRS, you might have a lot of false alarms.
• The Solution: The authors suggest that to get the full picture, you shouldn't rely on just one tool. You should run all three and combine their results. It's like using a metal detector, a shovel, and a magnet together to find all the treasure.

Why This Matters

Before this study, scientists were guessing which software to use. Now, they have a clear map.

• If you want speed and accuracy for a few specific targets, use IsoCirc.
• If you want to find everything and don't mind a slow process, use circNICK-LRS.
• If you want a balance, use CIRI-long.

The authors also made their "Fake Donut Factory" (the simulation code) free for everyone to use. This means other scientists can now test their own new tools against this perfect standard, helping to build better software for understanding these mysterious genetic "donuts" that could one day help cure diseases.

Technical Summary

1. Problem Statement

Circular RNAs (circRNAs) are stable, covalently closed non-coding RNAs with potential roles as disease biomarkers. However, their detection is challenging due to their circular structure and diverse size range (100 bp to 100 kb).

• Sequencing Context: While Oxford Nanopore Technologies (ONT) long-read sequencing can capture full-length circRNA molecules without fragmentation, the bioinformatic tools required to analyze this data lack standardized evaluation.
• Evaluation Gap: Existing wet-lab datasets suffer from a lack of "ground truth" (unknown true positive/negative rates) and biological variability. There is no comprehensive benchmark comparing specialized circRNA detection tools (CIRI-long, IsoCirc, circNICK-LRS) specifically for ONT long-read data.
• Tool Limitations: Current tools utilize different experimental protocols (rolling circle reverse transcription vs. amplification vs. linearization) and computational pipelines, leading to unknown biases regarding sensitivity, precision, and the ability to detect specific circRNA subtypes (exonic, intronic, intergenic).

2. Methodology

The authors developed a novel, open-source, data-driven simulation framework to generate realistic ground-truth datasets for rigorous benchmarking.

A. Simulation Framework (The "Nano-Circ" Pipeline)

Instead of relying solely on biological samples, the authors created a computational pipeline to simulate ONT reads with known ground truth:

1. Feature Extraction: They analyzed mouse circRNA databases (circAtlas v3 and circBase) and real wet-lab ONT data (Zhang et al., 2021) to extract molecular features:
   • Subtypes: Exonic (ecircRNA), Exon-Intron (EIciRNA), Intronic (ciRNA), and Intergenic.
   • Structural Properties: Splice site motifs (GT-AG, etc.), exon counts, mature lengths, and rolling circle amplification patterns.
   • Error Profiles: Derived from real ONT data to model insertion/deletion/substitution rates.
2. In Silico Generation:
   • Generated 7,503 unique simulated circRNAs and 117,667 linear transcripts.
   • Implemented specific biological complexities, such as alternative splicing (exon skipping, intron retention) and rolling circle replication patterns.
3. Read Simulation: Used NanoSim to generate FASTQ reads mimicking ONT cDNA 1D sequencing (Guppy basecaller, Q20 quality) based on the generated sequences.
4. Dataset Configuration: Created four distinct datasets:
   • Balanced Modes: 50% circRNA / 50% linear RNA at low (100k), medium (200k), and high (300k) sequencing depths.
   • Realistic Mode: ~3% circRNA enrichment (12k circRNA / 380k linear), mimicking experimental conditions.

B. Benchmarking Tools

Three specialized tools were evaluated:

1. CIRI-long: Uses rolling circle reverse transcription (RCRT) to detect tandem repeats and generate consensus sequences.
2. IsoCirc: Uses rolling circle amplification (RCA) followed by nanopore sequencing; detects tandem repeats via Tandem Repeat Finder (TRF).
3. circNICK-LRS: Uses a linearization approach (nicking) to produce single-copy reads spanning back-splice junctions (BSJ), detected via split-read alignment.

C. Evaluation Metrics

Performance was assessed using Precision, Recall (Sensitivity), Specificity, Accuracy, and F1 Score.

• Evaluation Levels:
  • Transcript-level: Overlap of the entire circRNA locus.
  • Exon-level (Block-level): Strict requirement for correct internal exon structure and splicing patterns.
• Overlap Thresholds: Tested at 25%, 50%, 75%, and 95% reciprocal overlap.
• Additional Metrics: Computational efficiency (memory usage, processing time) and expression quantification accuracy (correlation with ground truth).

3. Key Results

A. Detection Performance & Trade-offs

• Precision vs. Recall:
  • IsoCirc: Highest Precision (98%) but extremely low Recall (2-5%). It is conservative, missing many true positives but rarely generating false positives.
  • circNICK-LRS: Highest Recall (Sensitivity) and best F1 Score, but lower precision (~77-90%). It detects the most unique circRNAs but introduces more false positives.
  • CIRI-long: Occupied an intermediate position with high precision (~94-98%) and moderate recall.
• Intersection: The overlap of detected circRNAs between tools was very low (only ~10% consensus among all three), indicating that relying on a single tool results in significant data loss.
• Subtype Bias:
  • Intergenic circRNAs: All three tools failed to detect intergenic circRNAs (0% detection), despite them comprising ~8-12% of the ground truth.
  • ciRNAs: Only CIRI-long detected ciRNAs (and only in balanced modes); all tools missed them in realistic modes.
  • EIciRNAs: circNICK-LRS showed a strong bias toward detecting Exon-Intron circRNAs.
  • ecircRNAs: CIRI-long and IsoCirc were relatively enriched for exonic circRNAs.
• Length Bias:
  • circNICK-LRS: Best at detecting long circRNAs (mean length >3000 bp).
  • IsoCirc: Severely restricted to short circRNAs (mean ~200 bp) due to a built-in 4,000 nt cutoff in its TRF module.
  • CIRI-long: Biased toward shorter molecules (mean ~300-500 bp).

B. Expression Quantification

• IsoCirc and CIRI-long showed strong correlation with ground-truth expression levels (Pearson $r \approx 0.7$), making them suitable for quantification.
• circNICK-LRS had weaker quantitative agreement ($r \approx 0.4-0.5$), prioritizing sensitivity over abundance accuracy.
• Sensitivity by Expression: All tools struggled with low-abundance and very high-abundance circRNAs, with peak sensitivity observed in medium-expression ranges.

C. Computational Efficiency

• IsoCirc: Most efficient. High throughput (6.2M reads/hour) and low memory usage (17 GB).
• CIRI-long: Extremely high memory consumption (~307 GB), posing a risk of Out-of-Memory (OOM) errors on standard clusters.
• circNICK-LRS: Slowest processing speed (726k reads/hour) due to single-threaded architecture, though memory efficient (3.5 GB).

D. Structural Reconstruction

When evaluated at the exon-level (requiring correct internal splicing), performance metrics dropped sharply for all tools. Precision fell from ~90% (transcript level) to ~40-60% (exon level), indicating that while tools can identify the locus of a circRNA, accurately reconstructing the full-length isoform structure remains a major challenge.

4. Key Contributions

1. First Comprehensive Benchmark: The first systematic comparison of CIRI-long, IsoCirc, and circNICK-LRS on ONT long-read data.
2. Novel Simulation Framework: Developed and released an open-source, flexible pipeline (nano-circ) that integrates real wet-lab error profiles and database-derived biological features to generate realistic ground-truth datasets. This addresses the lack of standardized evaluation frameworks in the field.
3. Containerization: The authors built Docker/Singularity containers for CIRI-long and IsoCirc to resolve installation difficulties, promoting accessibility for the community.
4. Strategic Recommendations: Provided clear guidelines for tool selection based on research goals (e.g., use IsoCirc for precision/quantification, circNICK-LRS for sensitivity/long reads, or a union of tools for maximum discovery).

5. Significance

This study highlights that no single tool is sufficient for comprehensive circRNA discovery using ONT data.

• Complementarity: The low overlap between tools suggests that combining multiple approaches is necessary to capture the full diversity of the circRNA landscape.
• Blind Spots: The failure to detect intergenic and intronic circRNAs reveals a critical gap in current methodologies, likely due to reliance on known gene annotations and canonical splice motifs.
• Future Directions: The authors emphasize the need for improved algorithms to handle complex internal splicing (isoform reconstruction) and the development of deep learning models to better detect non-canonical and intergenic circRNAs.
• Resource Availability: The release of the simulation framework, scripts, and containers provides a vital resource for optimizing detection strategies and developing next-generation bioinformatic tools.

Conclusion: The paper establishes that while long-read sequencing holds promise for circRNA analysis, current bioinformatic tools have distinct, non-overlapping strengths and weaknesses. Researchers must carefully select tools based on their specific biological questions (e.g., discovery vs. quantification, short vs. long RNAs) and consider using ensemble approaches to mitigate individual tool biases.