Circular RNA identification using a genomic language model and a small number of authenticated examples

The paper introduces circFormer, a novel genomic language model that leverages curriculum learning and fine-tuning on noisy candidates to accurately identify circular RNAs from limited validated examples, outperforming existing tools and achieving high experimental validation rates while offering interpretable mechanistic insights.

The Gist

The Big Problem: Finding Needles in a Haystack (That's on Fire)

Imagine you are trying to find a specific type of rare, magical needle (called circular RNA) hidden inside a massive, chaotic haystack.

• The Haystack: This is the data from modern DNA sequencing machines. It's huge (millions of pieces of data), but it's messy. It contains the real needles, but also a lot of broken twigs, plastic wrappers, and fake needles created by the machine itself (noise).
• The Real Needles: These are the circular RNAs. They are tiny, ring-shaped molecules that act like "sponges" or "switches" in our cells, controlling how genes work.
• The Problem: Scientists have a very small list of proven real needles (only about 939 examples). They have a massive list of suspected needles (2.3 million), but most of them are likely fakes.

If you try to teach a computer to find the needles using only the 939 real ones, the computer gets confused and memorizes the wrong things (it "overfits"). If you try to teach it using the 2.3 million suspects, it gets overwhelmed by the garbage and learns nothing useful.

The Solution: circFormer (The Smart Intern)

The authors built a new AI tool called circFormer. Think of it as a highly trained "Smart Intern" who uses a special learning strategy called Curriculum Learning.

Here is how the intern learns, step-by-step:

1. Phase 1: The Classroom (Small Group):
   First, the intern studies the 939 proven real needles in a quiet classroom. They learn the basic shape and texture of a real needle. At this stage, the intern is good, but not perfect.

2. Phase 2: The Sorting Hat (Scoring the Chaos):
   The intern is now handed the massive pile of 2.3 million suspects. Instead of trying to learn from all of them at once, the intern acts as a "Teacher." They look at every single suspect and give it a Confidence Score (e.g., "This one looks 95% real," or "This one looks like trash").

3. Phase 3: The Final Exam (Learning from the Crowd):
   Now, the intern goes back to the classroom, but this time they study the massive pile again. However, they don't treat every piece of trash equally.

   • If the intern gave a suspect a high confidence score in Phase 2, they study it closely.
   • If the score was low, they glance at it but don't waste much time.
   • The Magic: By weighting the "noisy" data based on their own confidence, the intern learns to ignore the garbage and spot the subtle patterns of the real needles that other tools miss.

The Results: Better Than the Experts

The authors tested this new intern against 16 other popular computer programs (the "old guard") that scientists usually use.

• The Benchmark Test: They asked the old programs to find needles in a known pile. The new intern's ranking of the old programs matched perfectly with what human scientists found in the lab. This proved the intern could tell "fake" from "real" just by looking at the data.
• The Lab Test (The Real Proof): The intern picked out 50 "suspects" that the other 16 programs had completely ignored (thinking they were fake). The scientists took these 50 to the lab and tested them physically.
  • The Result: 94% of the "ignored" suspects turned out to be real circular RNAs.
  • The Metaphor: It's like the other tools were looking for needles that were shiny and gold, but the new intern found needles that were dull and silver, which turned out to be the real treasure all along.

The "Black Box" Problem: How Does the Intern Think?

Usually, AI is a "Black Box." You put data in, and an answer comes out, but you have no idea why the AI made that decision. The authors wanted to open the box.

They used a technique called Sparse Autoencoders (think of it as a "Translator"). They asked the AI to explain its reasoning in human terms.

• The Discovery: The AI found two different "languages" for circular RNA:
  1. The Standard Language: Most circular RNAs follow the classic rules of biology (like a specific "AG/GT" code). The AI learned this perfectly.
  2. The Secret Language: The AI discovered a second type of circular RNA that doesn't follow the classic rules. It uses a different pattern (rich in Pyrimidines and Purines) that looks like it's connected to cell membranes and transcription factors.
  • Why this matters: Before this, scientists thought these "non-standard" RNAs were just mistakes. The AI suggested they might be a completely different, regulated biological process. The AI didn't just find the needles; it discovered a new type of needle we didn't know existed.

The Takeaway

circFormer is a breakthrough because it solves the "Data Scarcity" problem. It shows that you don't need millions of perfect examples to train a powerful AI. Instead, you can use a small number of perfect examples to teach the AI how to "grade" the messy, imperfect data, and then let the AI learn from its own grading.

It turns a noisy, confusing haystack into a clear map of where the real biological treasures are hiding, and it even tells us why they are there. This approach could be used for many other diseases and biological mysteries where we have lots of messy data but very few confirmed answers.

Technical Summary

1. Problem Statement

The field of functional genomics faces a critical bottleneck: the scarcity of experimentally verified ("gold-standard") training data contrasted with the abundance of noisy, unlabeled high-throughput data.

• The Challenge: Large Genomic Language Models (gLMs) require vast amounts of high-quality labeled data to avoid overfitting and ensure generalizability. However, for Circular RNAs (circRNAs), experimentally validated examples are limited (often <1,000), while computational predictions from RNA-seq data number in the millions but are rife with false positives caused by sequencing errors, mapping artifacts, and repetitive sequences.
• The Limitation of Current Methods: Traditional machine learning (SVM, CNN, LSTM) and existing deep learning pipelines struggle with this data imbalance. They either overfit on small datasets or fail when trained on large noisy datasets, lacking the ability to distinguish genuine biological signals from technical artifacts effectively.

2. Methodology: circFormer

The authors developed circFormer, the first gLM-driven framework for circRNA identification, which integrates Curriculum Learning with gLM fine-tuning to overcome data scarcity.

A. Core Architecture

• Backbone Model: Utilizes the Nucleotide Transformer (NT), a 500-million-parameter pre-trained genomic language model.
• Input: Genomic sequences centered on or adjacent to putative back-splicing junctions (tested window sizes from 50 to 800 nt; optimal was 100 nt "full window").

B. Three-Phase Curriculum Learning Strategy

1. Phase 1 (Teacher Training): The NT model is fine-tuned on a small, high-quality dataset of 939 experimentally validated circRNAs (plus matched negative controls). This creates a "teacher" model capable of recognizing intrinsic circRNA features.
2. Phase 2 (Scoring Noisy Data): The Phase 1 model scores a massive dataset of ~2.34 million noisy, unlabeled candidates aggregated from 13 public circRNA databases. Each candidate is assigned a confidence score (probability of being a true positive).
3. Phase 3 (Student Refinement): The model undergoes a second round of fine-tuning using both the gold-standard set and the massive noisy set. Crucially, the noisy samples are weighted in the loss function based on their Phase 2 confidence scores (e.g., high-confidence noisy samples get higher weights, low-confidence get lower weights). This allows the model to learn from the vast data volume without being corrupted by noise.

C. Explainable AI (xAI) Integration

To address the "black box" nature of deep learning, the authors employed a dual-level interpretability strategy:

• In Silico Mutagenesis (ISM): Quantifies the importance of individual nucleotides by mutating them and observing prediction changes.
• Sparse Autoencoders (SAEs): Decomposes the model's dense, polysemantic latent embeddings (768 dimensions) into a sparse, overcomplete dictionary (12,800 features) of "mono-semantic" biological concepts. This isolates specific sequence motifs and regulatory logic.

3. Key Results

A. Performance and Robustness

• Superior Accuracy: circFormer achieved an AUC of 0.923 and F1-score of 0.920 after the second round of fine-tuning, outperforming traditional SVM, CNN, and LSTM models which degraded when exposed to noisy data.
• Strand Specificity: The model correctly rejected 96.8% of "strand-contaminated decoys" (sequences from real circRNAs swapped to the opposite strand), proving it learned strand-specific biological signals rather than memorizing genomic coordinates.
• Correlation with Benchmarks: When ranking 12 existing circRNA detection pipelines, circFormer's "authenticity rate" rankings correlated strongly (Spearman's $\rho$ = 0.623) with experimental validation results from a recent large-scale study.

B. Filtering and Discovery

• Noise Reduction: Applied to 2.3 million candidates, circFormer filtered out >50% of entries as likely noise, retaining ~938,000 high-confidence candidates.
• Database Validation: Databases with manual curation (e.g., circRNADb) received higher validation scores from circFormer than comprehensive, automated databases (e.g., CSCD), validating the model's ability to assess data quality.

C. Experimental Validation (Wet-Lab)

• Novel Discovery: The authors selected 50 high-confidence circRNAs that were missed by at least 14 of 16 existing tools.
• Validation Rate: Using RNase R digestion and RT-qPCR, 94.1% (32/34) of the evaluable candidates were confirmed as genuine circRNAs.
  • High-expression cohort: 100% validation rate (28/28).
  • Low-expression cohort: 66.7% validation rate (4/6), attributed to technical limits of qPCR rather than model error.

D. Biological Insights (Mechanistic Discovery)

• AG/GT vs. Non-AG/GT: The model successfully distinguished between canonical (AG/GT) and non-canonical (non-AG/GT) back-splicing.
• Motif Discovery:
  • AG/GT circRNAs: Associated with motifs linked to ribosomal machinery and translational elongation, supporting the theory of co-transcriptional processing.
  • Non-AG/GT circRNAs: Revealed distinct motifs (pyrimidine-rich and purine-rich tracts) associated with sequence-specific DNA binding, transcription factor activity, and membrane components. This suggests non-canonical circRNAs may arise via alternative pathways distinct from the standard spliceosome.

4. Key Contributions

1. Methodological Innovation: Introduced a curriculum learning framework that enables gLMs to leverage massive noisy datasets effectively when only a small number of ground-truth labels are available.
2. Tool Development: Created circFormer, a state-of-the-art predictor, and circFormer-STAR, an integrated pipeline compatible with the industry-standard STAR aligner.
3. Interpretability: Demonstrated how Sparse Autoencoders can transform a "black box" gLM into a transparent tool that reveals specific biological mechanisms and generates novel hypotheses about RNA biogenesis.
4. Biological Expansion: Significantly expanded the known circRNA repertoire by identifying and validating high-confidence candidates that conventional heuristic-based tools systematically miss.

5. Significance

This work provides a scalable, interpretable, and generalizable framework for applying genomic language models in data-scarce biological settings. It solves the fundamental "data scarcity vs. noise" paradox in genomics, offering a practical path to convert noisy high-throughput sequencing data into reliable functional annotations. Furthermore, by uncovering distinct regulatory logics for atypical circRNAs, the study moves beyond mere prediction to mechanistic discovery, suggesting that a properly trained gLM can act as an "in-silico biologist" capable of generating testable hypotheses for unexplored genomic features.