ARCH3D: A foundation model for global genome architecture

This paper introduces ARCH3D, a foundation model for global genome architecture that utilizes a novel masked locus modeling task to capture genome-wide spatial structures and enable the reconstruction of complex chromosomal interactions, ultimately serving as a structural basis for simulating genome behavior.

The Gist

Imagine your DNA isn't just a long, static string of letters (A, C, G, T) like a book sitting on a shelf. Instead, think of it as a giant, tangled ball of yarn inside a tiny room (the cell nucleus).

For a long time, scientists have been great at reading the letters on the yarn (the DNA sequence) and understanding the words (the genes). But they struggled to understand how the yarn is actually folded and tangled in 3D space. This folding is crucial because it determines which parts of the yarn touch each other, turning genes "on" or "off."

Enter ARCH3D. Think of it as a new, super-smart AI detective designed specifically to understand the "knots and tangles" of this DNA yarn ball.

Here is a simple breakdown of how it works and why it's a big deal:

1. The Old Way: Looking at a Single Patch

Previously, other AI models tried to understand DNA folding by looking at tiny, 2x2 inch patches of the yarn ball.

• The Problem: If you only look at a small patch, you can't see how that patch connects to the rest of the ball. You miss the big picture. It's like trying to understand a city's traffic by only looking at one street corner. You might see a red light, but you don't know if the traffic jam is actually happening three miles away.

2. The ARCH3D Way: The "Global Map" Approach

ARCH3D is different. Instead of looking at a small patch, it looks at entire strands of the yarn and asks: "If I pull this strand here, what happens to that strand over there?"

• The Analogy: Imagine you have a map of the entire world. Instead of zooming in on just New York, ARCH3D looks at New York, Tokyo, and London all at once to see how they are connected by flight paths.
• The Magic Trick: It uses a technique called "Masked Locus Modeling." Imagine the AI is playing a game of "Guess the Connection." It hides a piece of the yarn (masks it) and tries to guess how that hidden piece connects to every other piece in the room. By playing this game millions of times with data from 31 different human organs, it learns the "rules of the knot."

3. What ARCH3D Can Do (The Superpowers)

A. Seeing the Invisible (Filling in the Blanks)

Hi-C data (the technology used to map DNA folding) is often "noisy" or "sparse," meaning there are huge gaps in the map, especially between different chromosomes (different balls of yarn).

• The Result: Even if the data is 99% empty (like a map with only 1% of the roads drawn), ARCH3D can use its knowledge of the whole system to predict the missing roads. It can reconstruct the full 3D shape of the DNA even when the data is incredibly poor.

B. Predicting Group Hugs (Multi-way Interactions)

Most models only look at how two points touch (Pairwise). But in reality, three or four strands might come together in a "huddle" to control a gene.

• The Result: ARCH3D can predict these complex group huddles (called hyperedges) just by looking at the 2D map. It's like being able to predict that three people in a room are about to form a huddle just by looking at a photo of them standing apart.

C. Building a "Virtual Genome"

The ultimate goal is to create a Virtual Genome.

• The Vision: Imagine a video game simulation of a cell. You could tweak the DNA (like changing a character's stats) and watch the AI predict exactly how the cell would react, how the yarn would re-tangle, and what diseases might result. This could revolutionize how we design drugs and cure diseases, letting us run thousands of experiments on a computer before ever touching a real petri dish.

Why This Matters

Before ARCH3D, we had great models for the letters of DNA and the proteins they make, but we were blind to the architecture (the folding).

ARCH3D fills that gap. It's the missing piece of the puzzle that allows us to move from just "reading" the genome to truly "understanding" how it behaves as a 3D machine. It's the foundation for building a digital twin of human biology.

Technical Summary

1. Problem Statement

Biological foundation models have recently advanced significantly in representing DNA sequences, RNA, and proteins by leveraging transformer architectures and increasing input context windows. However, genome architecture (the 3D spatial organization of chromatin) remains underexplored.

• Limitations of Existing Methods: Current models for Hi-C data (chromosome conformation capture) generally fall into two categories:
  • Patch-based approaches (e.g., HiCFoundation): These use small submatrices (e.g., 224×224 pixels) as input. While effective for local, fine-scale analysis, they have extremely restricted context windows (approx. 1.12 Mb) and cannot capture interactions between distant loci or interchromosomal contacts.
  • Locus-based approaches (e.g., Hyper-SAGNN, MATCHA): These use rows from the Hi-C matrix but are severely limited by small context windows (often only 5 loci), low resolution (1 Mb), and a focus solely on intrachromosomal blocks.
• The Gap: There is no foundation model capable of generating global representations of genome architecture that capture long-range, interchromosomal, and higher-order interactions at high resolution (5 kb) across the entire genome.

2. Methodology

The authors introduce ARCH3D, a transformer-based foundation model designed to learn global genome structure through a novel pre-training strategy.

A. Data Curation and Tokenization

• Corpus: Pre-trained on 481 Hi-C experiments from 31 human tissues (4DNucleome and ENCODE consortia), normalized at 5 kb resolution.
• Locus-Level Tokenization: Unlike patch-based models, ARCH3D tokenizes the genome at the locus level.
  • A "token" is a row from the Hi-C matrix corresponding to a specific genomic locus.
  • Variable Length: Loci can vary in length (5 kb to 1 Mb). Longer loci are formed by averaging the rows of the constituent 5 kb bins.
  • Global Context: Input sequences consist of 1,024 randomly selected loci from across the entire genome (not just contiguous regions). This maximizes information density and reduces redundancy inherent in adjacent loci.

B. Architecture

• Backbone: A BERT-large style transformer encoder (24 layers, 1,024 hidden dimensions, 16 attention heads, 302M parameters).
• Positional Encoding: A novel, biologically-inspired encoding is used to preserve genomic position within the permutation-invariant transformer:
  • Chromosome Embedding: Learnable vectors for each chromosome.
  • Base Pair Embedding: Sinusoidal encoding (Vaswani et al.) applied to the start and end base pairs of the locus.
  • These are concatenated and added to the token embedding, allowing the model to understand the physical distance and chromosomal location of non-contiguous loci.

C. Pre-training Task: Masked Locus Modeling (MLM)

• Task: A variation of Masked Language Modeling adapted for 3D structure.
  • An input sequence of 1,024 loci is created.
  • 200 loci are masked (replaced with a learnable [MASK] vector).
  • Objective: The model must predict the contact frequency (pixel values) between all pairs of loci in the sequence (both masked and unmasked).
  • Loss Function: Mean Squared Error (MSE) between predicted and observed Hi-C pixel values.
• Significance: By predicting interactions between all pairs, the model learns to infer the global conformation of a locus based on its contacts with distant, random loci across the genome.

D. Downstream Tasks

1. Resolution Enhancement (Fine-tuning): The model is fine-tuned to reconstruct high-coverage Hi-C maps from extremely sparse inputs (downsampled to 1% or 0.42% non-zero pixels), specifically targeting interchromosomal regions where data is usually lost.
2. Higher-Order Interaction Prediction: The model is used to predict hyperedges (multi-way interactions involving 3+ loci) from pairwise Hi-C data, effectively simulating Pore-C data.

3. Key Contributions

1. First Global Genome Architecture Foundation Model: ARCH3D is the first model to utilize a locus-based approach with a genome-wide context window, moving beyond local submatrices.
2. Novel Masked Locus Modeling: Introduces a pre-training task that forces the model to learn global spatial relationships by predicting contacts between randomly sampled, non-contiguous loci.
3. Biologically-Informed Positional Encoding: Develops an encoding scheme that captures absolute genomic coordinates and chromosome identity, enabling the model to handle non-sequential input tokens.
4. Scalable Tokenization: Enables variable-length locus inputs, allowing alignment with other foundation models (e.g., DNA sequence, scRNA-seq) and increasing the training dataset size to over 2 billion tokens.

4. Results

• Preservation of Spatial Structure:
  • The embedding space naturally mirrors the physical nucleus. Loci from the same chromosome cluster closer together than those from different chromosomes (reflecting chromosome territories).
  • Embedding distances correlate with cell differentiation (e.g., H1-hESC vs. IMR-90), reflecting known changes in chromatin compaction.
  • Intrachromosomal compartmentalization patterns in the embedding space closely match Hi-C contact maps.
• Extreme Sparsity Resilience (Resolution Enhancement):
  • ARCH3D successfully reconstructed interchromosomal contacts from data with 99.58% sparsity (only 0.42% non-zero pixels).
  • While patch-based models (HiCFoundation) excel at fine-scale diagonal features, they fail completely in sparse interchromosomal regions. ARCH3D leverages global context to infer these missing interactions.
• Higher-Order Interaction Identification:
  • ARCH3D was tested on predicting 3-way, 4-way, and 5-way contacts (hyperedges) derived from Pore-C data.
  • Performance: Achieved an average AUROC of 0.930 across three cell lines.
  • Comparison: Significantly outperformed the state-of-the-art deep learning model MATCHA (AUROC ~0.55–0.78) and statistical multi-correlation methods.
  • Crucially, ARCH3D generalized well to real Hi-C data inputs, whereas MATCHA struggled without retraining.

5. Significance and Future Outlook

• Virtual Genome: ARCH3D provides a structural foundation for building a "virtual genome"—an AI system capable of simulating genome behavior and dynamics.
• Multi-Modal Integration: The locus-level embeddings can be directly aligned with embeddings from DNA sequence (e.g., Evo, Enformer) and transcriptomic (e.g., Geneformer) foundation models. This enables the modeling of complex biological processes that depend on the interplay between sequence, expression, and 3D structure.
• Accelerated Discovery: By enabling in silico perturbation experiments (e.g., predicting how a structural change affects gene regulation), ARCH3D could help identify optimal strategies for cell reprogramming and accelerate wet-lab experimentation.

In summary, ARCH3D represents a paradigm shift from local, patch-based analysis of genome architecture to a global, context-rich foundation model, unlocking the ability to infer complex, long-range, and higher-order genomic interactions from sparse data.