DNA-LLM2Vec: Architecture-Aware Task-Conditioned Adaptation of Decoder DNA Foundation Models

The human genome is often described as a vast instruction manual, written in a four-letter chemical alphabet that spells out the blueprint for life. For decades, scientists have tried to build computer models that can read this manual, hoping to understand how tiny changes in the code lead to disease or how specific sequences control the body's functions. A major challenge in this effort is that the instructions are not just a simple list; they are a complex web where a single gene's activity depends on signals coming from both before and after it in the sequence. Furthermore, because DNA is a double-stranded molecule, the same information can be read from either direction, meaning a computer model must understand that a sequence and its mirror image carry the same meaning. Until recently, the most powerful computer models for reading DNA were designed to read only in one direction, like a person reading a book from left to right, missing the crucial context that comes from the other side.

A team of researchers has now developed a new method to fix this limitation, turning these one-directional readers into powerful, two-way interpreters without having to rebuild them from scratch. They took two existing, highly advanced DNA models—one that reads in short chunks and another that handles very long stretches of genetic code—and taught them to look backward and forward simultaneously. By doing so, they created a system that can understand the full context of a genetic location, much like a reader who can glance at the paragraph before and after a sentence to grasp its true meaning. The researchers found that this conversion works exceptionally well for tasks involving long-range genetic interactions, such as how distant parts of the genome talk to each other to turn genes on or off. However, they also discovered that the best way to adapt these models depends heavily on the specific type of genetic task and the original design of the model itself; there is no single "perfect" version that works for everything.

The core of this work involves a step-by-step process to transform a "causal" DNA model, which predicts the next letter in a sequence based only on what came before, into a "bidirectional" model that sees the whole picture at once. The researchers first opened the model's attention mechanism so it could look at the entire sequence, not just the past. This created a mismatch, however, because the model had never been trained to handle the concept of "missing" information, which is necessary for this new way of reading. To repair this, they taught the model to predict a hidden letter based on the surrounding context, a technique that allowed it to learn how to use its new two-way vision effectively. Finally, they added a training step that taught the model to recognize that a DNA sequence and its reverse mirror image are fundamentally the same, ensuring the model respects the natural symmetry of the double helix.

When the team tested these newly adapted models on a wide variety of biological tasks, the results were striking but nuanced. For tasks that require understanding long distances, such as how an enhancer (a genetic switch) interacts with a promoter (the start of a gene) hundreds of thousands of letters away, the new method provided a massive boost in accuracy. In these long-range scenarios, the adapted models improved their performance by up to 27 percentage points compared to the original, one-way versions. This suggests that the ability to see the full context is critical for understanding how the genome regulates itself over large distances. One specific training technique, which involved showing the model slightly different crops of the same long sequence, proved to be the most effective for these extended tasks.

However, the story changes when looking at shorter, more local genetic tasks. For problems involving the precise recognition of splice sites (where DNA is cut and joined) or the binding of proteins to specific short patterns, the benefits were not uniform. In some cases, the new two-way models performed worse than the original one-way versions, particularly when the model's original design used a specific type of tokenization that grouped letters together in ways that didn't fit the short-range task. This finding is crucial because it shows that simply making a model bidirectional is not a magic bullet; the success of the adaptation depends on matching the method to the specific structure of the problem and the underlying architecture of the model.

The researchers also looked inside the models to understand what was actually happening during this transformation. They used a diagnostic tool that simulated tiny mutations in the DNA sequence to see how the model's internal understanding of the genetic code changed. They found that the training process did indeed organize the model's internal structure to better recognize known regulatory patterns, and the training that taught the model to respect the double-stranded nature of DNA successfully made its understanding consistent regardless of which strand was presented. Yet, they also found that these internal improvements did not always predict which version of the model would perform best on a specific external task. For instance, the version that best understood the symmetry of the DNA strands was not necessarily the one that performed best at predicting long-range gene interactions.

This disconnect highlights a key insight from the study: the way a model organizes its internal knowledge does not always translate directly into better performance on every downstream application. The researchers demonstrated that the value of their method lies in creating a frozen, reusable representation of DNA that captures bidirectional context and strand symmetry. While this representation is highly effective for certain types of tasks, particularly those involving long-range interactions, it requires careful selection of the adaptation strategy based on the specific goal. The study concludes that there is no single best way to convert these powerful DNA models; instead, scientists must choose the adaptation approach that aligns with both the biological question they are asking and the specific design of the model they are using.

The implications of this work extend beyond just improving one specific algorithm. It suggests that the future of genomic AI lies not in building a single, universal model that does everything perfectly, but in developing flexible frameworks that can be tuned to the specific needs of different biological problems. By showing that existing models can be successfully repurposed to handle the complex, bidirectional nature of DNA, the researchers have provided a new tool for scientists to explore the genome. This approach allows them to leverage the massive amounts of data already used to train these models while adding the critical ability to see the full context of the genetic code, opening new doors for understanding how life is regulated at the molecular level.