From Circles to Signals: Representation Learning on Ultra-Long Extrachromosomal Circular DNA

This paper introduces eccDNAMamba, a bidirectional state space model built on the Mamba-2 framework that overcomes the limitations of existing genomic foundation models by efficiently scaling to ultra-long sequences and preserving the intrinsic circular topology of extrachromosomal circular DNA (eccDNA) to achieve superior performance in cancer-related prediction tasks.

The Gist

The Big Picture: The "Loose Thread" in the Cell

Imagine your DNA as a massive, organized library of books (chromosomes) where every instruction for building a human is neatly shelved.

Now, imagine that in cancer cells, some pages get ripped out of these books, rolled up into tight little balls, and thrown onto the floor. These loose, circular balls of DNA are called eccDNA (extrachromosomal circular DNA).

These aren't just random scraps. They are often super-charged. They contain "evil" instructions (oncogenes) that tell the cancer cell to grow faster, resist medicine, and spread. The problem? These circular balls can be huge—sometimes millions of letters long—and they are circular.

The Problem: Why Old Tools Failed

Scientists wanted to use AI to read these circular DNA balls to predict if a patient has cancer or how aggressive the tumor is. But existing AI tools had two big problems:

1. The "Short Memory" Problem: Most DNA AI models are like students with very short attention spans. They can only read a few pages at a time. If you give them a 1-million-letter circle, they chop it into tiny, disconnected pieces. This destroys the story. It's like trying to understand a movie by watching 1-second clips of it; you miss the plot.
2. The "Straight Line" Problem: Most AI models read text from left to right, like a book. But eccDNA is a circle. The end of the sequence connects back to the beginning. If you cut the circle open to read it linearly, you break the connection between the start and the finish. It's like trying to understand a necklace by cutting the string; the beads fall apart, and you lose the shape.

The Solution: Introducing eccDNAMamba

The researchers built a new AI model called eccDNAMamba. Think of it as a super-smart, circular-reading robot designed specifically for these DNA balls.

Here is how it works, step-by-step:

1. The "Zipper" (Efficient Tokenization)

Imagine the DNA sequence is a long sentence with lots of repeated words.

• Old way: Reading every single letter (A, C, T, G) individually.
• eccDNAMamba way: It uses a "Zipper" (called Byte-Pair Encoding). It recognizes that "GCTGA" appears a lot, so it groups those letters into one single "token" or symbol.
• Analogy: Instead of reading "The quick brown fox jumps over the lazy dog" letter by letter, it reads "The-quick-brown-fox" as one block. This makes the long sequence much shorter and faster to process without losing meaning.

2. The "Seamless Loop" (Circular Augmentation)

This is the model's secret sauce. Since the DNA is a circle, the model needs to know that the end connects to the start.

• The Trick: The model takes the first few words of the story and pastes them onto the very end of the story.
• Analogy: Imagine reading a story on a scroll. To make sure you don't miss the connection between the last sentence and the first, you tape a copy of the first paragraph to the very end of the scroll. Now, even if you read straight through, you see the "wrap-around" connection. This teaches the AI that the DNA is a loop, not a straight line.

3. The "Two-Way Scanner" (Bidirectional Mamba-2)

Old models read left-to-right. This model reads both ways at the same time.

• Analogy: Imagine two detectives scanning a crime scene. One walks from the front door to the back, and the other walks from the back to the front. They meet in the middle and combine their notes. This ensures the AI understands the context from every angle, capturing long-range relationships that other models miss.

What Did They Discover?

The team tested eccDNAMamba on real cancer data and found it was much better than previous tools at two things:

1. Spotting Cancer: It could tell the difference between "healthy" DNA circles and "cancerous" ones with high accuracy, even when the circles were huge (ultra-long).
2. Counting Copies: It could guess how many copies of a dangerous gene were present just by looking at the sequence, which is usually a very difficult task.

The "Why" (Interpretability):
The researchers didn't just trust the AI; they asked it why it made its decisions. They found that the model was focusing on specific "regulatory switches" (like light switches for genes) and "jumping genes" (transposable elements) that are known to drive cancer.

• Analogy: It's like a detective who doesn't just guess who the killer is, but points to the specific fingerprints on the gun. The AI showed it was looking at the exact biological parts that make cancer cells dangerous.

Why Does This Matter?

• Speed & Memory: Because it uses this new "Mamba" technology, it doesn't need a supercomputer to run. It's fast and memory-efficient, like a sports car compared to a heavy truck.
• New Insights: It proves that the shape of the DNA (the circle) and the long-range connections matter. By respecting the circular topology, the AI can see patterns that were previously invisible.

Summary

eccDNAMamba is a new AI tool that finally learned how to read the "loose, circular balls" of DNA found in cancer. Instead of chopping them up or reading them straight like a book, it zips them up for speed, tapes the ends together to respect their shape, and scans them from both directions. This allows scientists to better understand how cancer evolves and potentially find new ways to fight it.

Technical Summary

1. Problem Statement

Extrachromosomal circular DNA (eccDNA) plays a critical role in cancer biology, often harboring oncogenes and regulatory sequences that drive tumor evolution and therapeutic resistance. However, modeling eccDNA presents unique computational challenges that existing genomic foundation models fail to address:

• Ultra-Long Sequences: eccDNA molecules can span tens of kilobases to several megabases. Standard Transformer-based models (e.g., DNABERT-2, Nucleotide Transformer) rely on attention mechanisms with quadratic complexity ($O(N^2)$), making them computationally infeasible for such long sequences.
• Circular Topology: eccDNA is a covalently closed circle with "wrap-around" dependencies between the head and tail. Existing efficient models (e.g., HyenaDNA, Caduceus) typically process sequences linearly or truncate them into fragments, thereby breaking the circular continuity and losing critical long-range contextual information.
• Tokenization Inefficiency: Standard base-level tokenization (per-nucleotide) drastically expands sequence length, exacerbating memory constraints even for linear-time models.

2. Methodology: eccDNAMamba

The authors propose eccDNAMamba, a bidirectional State Space Model (SSM) built on the Mamba-2 framework, specifically designed to handle ultra-long, circular DNA sequences efficiently. The pipeline consists of four key components:

A. Efficient Tokenization (Byte-Pair Encoding)

Instead of tokenizing at the single-nucleotide level, the model employs Byte-Pair Encoding (BPE).

• Mechanism: Frequent nucleotide patterns are merged into compact tokens based on co-occurrence frequency.
• Vocabulary: A vocabulary size of 4,096 is used.
• Benefit: This significantly reduces the effective sequence length while preserving biological meaning, enabling the processing of megabase-scale sequences without excessive memory overhead.

B. Circular Data Augmentation

To preserve the intrinsic circular topology, the model introduces a lightweight augmentation strategy.

• Mechanism: The first 64 tokens of a sequence are appended to the end of the sequence ($\tilde{x} = [x_1, \dots, x_L, x_1, \dots, x_{64}]$).
• Rationale: This explicitly exposes the head–tail junction to the model, allowing it to learn "wrap-around" dependencies without requiring complex circular convolution operations. The choice of 64 tokens (approx. 25% of typical pretraining sequences) balances context coverage with efficiency.

C. Bidirectional Mamba-2 Encoding

The core architecture utilizes two independent Mamba-2 encoders to process the augmented sequence.

• Forward Pass: Scans the sequence in natural order.
• Reverse Pass: Scans the reversed sequence.
• Fusion: The outputs of both passes are aligned (using a FLIP operator on the reverse output) and fused via a shared Multi-Layer Perceptron (MLP) to create a unified representation.
• Complexity: This approach maintains linear time complexity ($O(N)$) and a stable memory footprint, unlike the quadratic scaling of Transformers.

D. Pretraining Strategy

The model is pre-trained using a span-masked language modeling objective.

• Masking: Instead of masking isolated tokens, contiguous spans of roughly three tokens are masked (covering ~15% of the sequence).
• Objective: The model reconstructs the missing spans from the surrounding context. This encourages the learning of intra-span dependencies and long-range circular coherence.

3. Key Contributions

1. First Topology-Aware Foundation Model for eccDNA: eccDNAMamba is the first model to simultaneously achieve efficient linear-time scaling for ultra-long sequences while explicitly preserving the circular topology of eccDNA.
2. Novel Augmentation Strategy: The introduction of circular augmentation (appending head tokens to the tail) effectively bridges the gap between linear SSMs and circular biological structures.
3. Comprehensive Benchmarking: The authors established the EccDNA Multi-Task Benchmark, standardizing data from CircleBase and eccDNAdb to evaluate models on cancer discrimination and copy-number prediction.
4. Biological Interpretability: The model provides mechanistic insights through Integrated Gradients (IG), revealing that it focuses on biologically relevant regulatory elements and discovers novel cancer-associated motifs.

4. Experimental Results

The model was evaluated against state-of-the-art baselines (DNABERT-2, HyenaDNA, Caduceus) on two primary tasks:

A. Cancer vs. Healthy eccDNA Discrimination

• Short Sequences (<10k bp): eccDNAMamba achieved 59.0% MCC and 79.3% F1, outperforming all baselines.
• Ultra-Long Sequences (10k–200k bp): eccDNAMamba maintained robust performance (57.9% MCC, 82.1% F1).
• Baseline Failure: DNABERT-2 collapsed on ultra-long sequences (dropping to 10.9% MCC) due to truncation requirements. HyenaDNA and Caduceus showed significant performance degradation compared to eccDNAMamba.
• Ablation: Removing circular augmentation (eccDNAMamba-1M w/o CA) resulted in performance degradation, confirming the necessity of topology-aware modeling.

B. Copy-Number Level Prediction

• Task: Predicting low vs. high copy-number amplification from sequence alone.
• Performance: eccDNAMamba achieved 36.0% MCC (high threshold) and 28.7% MCC (low threshold), significantly outperforming Caduceus (20.4% and 2.5% respectively) and DNABERT-2.
• Significance: This demonstrates the first successful inference of copy-number variations directly from sequence data.

C. Efficiency

• Memory Footprint: eccDNAMamba uses 40% less GPU memory than HyenaDNA and Caduceus, and 50% less than DNABERT-2 during fine-tuning. Its memory usage remains near-constant regardless of sequence length, whereas attention-based models scale poorly.

5. Biological Interpretations & Significance

Using Integrated Gradients (IG), the authors analyzed what the model learned:

• Regulatory Focus: The model assigns high attribution to known regulatory elements (promoters, enhancers) and specific transposable elements (LINE-1, ERV), which are known to amplify oncogenic programs.
• Topological Awareness: IG analysis revealed a pronounced signal enrichment at the head–tail junction (breakpoints), validating that the circular augmentation successfully taught the model to recognize the circular structure.
• Motif Discovery: The model identified 23 motifs significantly enriched in cancer eccDNAs. While some matched known transcription factors (STAT, FOX, ARID families), 15/23 motifs had no database match, suggesting the discovery of novel sequence patterns specific to cancer eccDNA.

Conclusion

eccDNAMamba bridges a critical gap in genomic sequence analysis by providing a scalable, topology-aware framework for ultra-long circular DNA. It overcomes the computational bottlenecks of Transformers and the structural limitations of linear SSMs. The model not only achieves superior predictive performance in cancer biology tasks but also offers interpretable insights into the regulatory architecture and novel motifs driving oncogenesis, establishing a new standard for analyzing extrachromosomal DNA.