Uncertainty-aware benchmarking reveals ambiguous transcripts in mRNA-lncRNA classification

This study introduces an uncertainty-aware benchmarking framework that combines controlled evaluation, inter-tool agreement analysis, and expanded feature profiling to identify sequence properties and patterns driving classification ambiguity between mRNAs and lncRNAs, thereby guiding the development of more robust classifiers.

The Gist

Imagine the human genome as a massive, bustling library containing millions of books (genes). For a long time, librarians (scientists) knew exactly how to sort these books into two main categories: "Instruction Manuals" (mRNAs, which tell the cell how to build proteins) and "Regulatory Notes" (lncRNAs, which don't build proteins but help manage the library's organization).

However, in recent years, the library has gotten messy. Many "Regulatory Notes" look suspiciously like "Instruction Manuals." They have similar cover designs, similar chapter lengths, and even similar sentence structures. This has made it very hard for the librarians to tell them apart.

This paper is like a quality control audit of the computer programs (classifiers) that the librarians use to sort these books. Here is the story of what they found, explained simply:

1. The Problem: The "Look-Alike" Books

The researchers realized that while the sorting computers are generally good at their job, they often disagree with each other. If you ask eight different experts to sort a specific pile of books, they might all agree on 55% of them. But for the other 45%, the experts are arguing! Some say, "This is an Instruction Manual!" while others shout, "No, it's a Regulatory Note!"

The authors asked: Why are they arguing? What makes these specific books so confusing?

2. The Experiment: A Strict "Taste Test"

To find the answer, the team set up a very strict, fair test:

• The Dataset: They gathered a huge list of books from the latest library catalog (GENCODE v47) but only kept the ones that were clearly labeled in both the old and new catalogs. This ensured they weren't testing on books that the library itself was still confused about.
• The Cleanup: They removed "duplicate" books (books that were 90% identical) so the computers couldn't just cheat by memorizing the answers.
• The Contest: They took eight different sorting algorithms (ranging from simple math tricks to complex AI) and forced them to re-learn how to sort these specific books from scratch.

3. The Discovery: The "Confusion Zone"

The results were surprising. Even though the computers were very accurate overall, they hit a wall with a specific group of books.

• The "Easy" Books: Some books were so clearly written that every computer agreed instantly.
• The "Ambiguous" Books: About 45% of the books fell into a "Confusion Zone." These books had a mix of features. They had some "Instruction Manual" traits (like long sentences that could code for proteins) but also "Regulatory Note" traits.

The researchers used a concept called "Entropy" (which is just a fancy word for "confusion").

• Low Entropy: The computers were confident. "I know this one!"
• High Entropy: The computers were sweating. "I'm not sure... maybe this, maybe that?"

4. The Detective Work: What Makes a Book Confusing?

The team didn't just stop at counting the arguments. They looked inside the confusing books to see what made them so hard to sort. They looked for clues that standard sorting programs usually ignore:

• The "Repetitive Text" Clue (Transposable Elements): They found that many "Regulatory Notes" were filled with repetitive paragraphs (like a sentence repeated 50 times). Standard programs often ignore this, but the researchers found that the amount and type of repetition helped distinguish the books. It's like realizing that "Regulatory Notes" often use a specific, repetitive font that "Instruction Manuals" don't.
• The "Strange Shapes" Clue (Non-B DNA): DNA usually looks like a twisted ladder (a helix). But sometimes, it folds into weird shapes like knots or loops (Non-B DNA). They found that "Instruction Manuals" often had these weird shapes built into their structure, while "Regulatory Notes" relied more on their text content.

5. The Big Lesson: It's a Spectrum, Not a Switch

The most important takeaway is that the line between "Instruction Manual" and "Regulatory Note" isn't a sharp wall; it's a foggy gray area.

• The "Imposter" Books: Some books that look like "Regulatory Notes" actually have strong "Instruction Manual" signals hidden inside them.
• The "Mimic" Books: Some "Instruction Manuals" look so much like "Regulatory Notes" that even the best computers get confused.

Why Does This Matter?

This study is like a manual for future librarians. It tells them:

1. Don't trust the computer blindly: If the computer is "confused" (high entropy), don't just guess. Flag that book for a human expert to double-check.
2. Look deeper: To sort these books better, we need to look at the "repetitive text" and the "weird shapes," not just the length of the sentences.
3. Biological Reality: These "confusing" books might actually be doing both jobs (building proteins AND regulating the library). Nature is messy, and our computers need to learn to embrace that messiness rather than trying to force a perfect binary choice.

In a nutshell: The researchers built a better way to test how well computers sort genetic books. They found that nearly half the books are in a "gray zone" where the computers argue. By looking at hidden clues like repetitive text and DNA shapes, they figured out why the computers are confused, helping us understand that the difference between coding and non-coding DNA is a spectrum, not a simple yes-or-no question.

Technical Summary

1. Problem Statement

Distinguishing long non-coding RNAs (lncRNAs) from protein-coding transcripts (mRNAs) remains a significant challenge in computational genomics. Despite the proliferation of classification tools, several critical issues persist:

• Annotation Instability: Transcripts often change classification status (e.g., from lncRNA to protein-coding) between genome annotation releases (e.g., GENCODE v46 to v47), introducing label noise.
• Data Leakage: Many benchmarks suffer from sequence redundancy (near-duplicate isoforms) between training and testing sets, leading to inflated performance metrics.
• Feature Limitations: Existing classifiers rely heavily on standard sequence features (k-mers, ORF length, hexamer scores) and often ignore biologically relevant signals like repetitive elements (Transposable Elements) and non-B DNA motifs.
• Lack of Uncertainty Quantification: High aggregate accuracy metrics often mask substantial inter-tool disagreement at the individual transcript level, leaving "difficult" or ambiguous cases uncharacterized.

2. Methodology

The authors developed a rigorous, uncertainty-aware benchmarking framework consisting of four main components:

A. Dataset Construction (Common-CDHIT)

• Stability Filtering: They constructed a "common" dataset using transcripts present in both GENCODE v46 and v47 that retained the same biotype label (protein-coding or lncRNA) across versions. This minimized label noise.
• Redundancy Reduction: To prevent data leakage, they applied CD-HIT clustering (90% identity threshold) to the common set, selecting the longest representative sequence per cluster.
• Final Set: The resulting "common-CDHIT" dataset contains ~112,000 transcripts (66,703 mRNAs, 45,561 lncRNAs), split into 5-fold cross-validation sets with balanced training classes.

B. Extended Feature Extraction

Beyond standard sequence features, the authors computed two novel feature sets:

1. Repeat-Derived Features: Using RepeatMasker, they extracted 170 features per transcript, including counts, coverage, and family-specific summaries of Transposable Elements (TEs), low-complexity regions, and tandem repeats.
2. Non-B DNA Motif Features: Using G4Discovery and non-B_gfa pipelines, they predicted 178 features across gene bodies, including G-quadruplexes, Z-DNA, triplexes, and various repeat motifs (A-phased, direct, inverted, mirror).

C. Model Benchmarking

• Tools: Eight representative classifiers were retrained from scratch on the common-CDHIT dataset: lncRNA-BERT, CPAT, mRNN, FEELnc, RNAsamba, LncFinder, LncDC, and PLncPRO.
• Protocol: All models were evaluated using identical 5-fold cross-validation splits to ensure fair comparison.

D. Uncertainty and Disagreement Analysis

• Entropy-Based Stratification: The authors calculated predictive entropy and a disagreement score (based on mutual information) across the ensemble of 8 tools.
• Grouping: Transcripts were stratified into Low-Entropy (high confidence/agreement) and High-Entropy (low confidence/disagreement) groups using the top and bottom 10th percentiles.
• Feature Attribution: They employed Random Forest (RF) models and SHAP (SHapley Additive exPlanations) values to quantify global feature importance and local contributions to specific misclassifications.

3. Key Results

Performance vs. Agreement

• High Aggregate Scores: The top-performing tools (lncRNA-BERT, mRNN, FEELnc) achieved >90% balanced accuracy, precision, and F1-scores.
• Substantial Discordance: Despite high aggregate scores, ~45% of transcripts showed inter-tool disagreement.
  • Only 54.5% of transcripts were unanimously classified correctly.
  • 0.9% were unanimously misclassified (all tools agreed on the wrong label).
  • Discordance was significantly higher for lncRNAs than for mRNAs.

Characterization of Uncertainty Groups

• Low-Entropy Transcripts: These clustered tightly in feature space and exhibited strong, classical coding signals (long ORFs, high hexamer scores) for mRNAs, or distinct TE profiles (LTR, LINE, SINE enrichment) for lncRNAs.
• High-Entropy Transcripts: These localized at the boundary between coding and non-coding classes in t-SNE visualizations. They displayed mixed signatures:
  • lncRNAs in this group often possessed coding-like features (long ORFs, high hexamer scores) that confused classifiers.
  • They exhibited conflicting signals where different features pushed the prediction in opposite directions.

Feature Importance Insights

• Classical Drivers: Sequence composition (12-mer k-mers, Fickett scores) and ORF metrics remained the strongest global predictors.
• Novel Drivers:
  • Repetitive Elements: TE-derived features (specifically LTR presence) emerged as consistent discriminators for lncRNAs across both low- and high-entropy groups.
  • Non-B DNA: Protein-coding genes were enriched for non-B DNA motifs (e.g., Z-DNA, G-quadruplexes) in their gene bodies, whereas lncRNAs showed different structural propensities.
• SHAP Analysis: In high-entropy cases, features like the 12-mer score might correctly predict the class, but other features (e.g., specific repeat content or gap statistics) actively counteracted this signal, reducing prediction confidence.

4. Key Contributions

1. Robust Benchmarking Framework: Introduced a "common-CDHIT" dataset that eliminates label instability and sequence redundancy, setting a new standard for reproducible benchmarking.
2. Uncertainty-Aware Stratification: Demonstrated that aggregate metrics hide significant biological ambiguity. The entropy-based stratification provides a practical method to identify "difficult" transcripts requiring manual review.
3. Expanded Feature Space: Validated that repetitive elements (particularly LTRs) and non-B DNA motifs are not just noise but carry significant discriminative power for distinguishing lncRNAs from mRNAs.
4. Biological Interpretability: Provided evidence that high-entropy transcripts often represent a biological spectrum where coding and non-coding signals overlap, rather than simple annotation errors.

5. Significance and Implications

• For Genome Annotation: The study suggests that inter-tool agreement and entropy scores should be used as quality control steps. Transcripts with high uncertainty should be flagged for orthogonal validation (e.g., ribosome profiling, proteomics) rather than being assigned definitive labels.
• For Model Development: Future classifiers should incorporate repeat-derived and structural DNA features to better resolve the "ambiguous zone" where current tools fail.
• For lncRNA Biology: The findings support the hypothesis that lncRNAs are evolutionarily younger and less constrained, often retaining TE fragments that serve functional roles (e.g., nuclear localization) while simultaneously obscuring their classification. The distinction between coding and non-coding is proposed as a spectrum rather than a binary boundary.

In conclusion, this paper moves beyond simple accuracy metrics to provide a nuanced, feature-rich understanding of why transcript classification fails, offering a roadmap for more robust and biologically interpretable genomic tools.