Technical Summary: TadA-Bench

Problem Statement

The field of AI for scientific discovery is transitioning toward "agentic" workflows, where models actively participate in iterative decision-making rather than passively fitting static datasets. In protein engineering, this implies systems that can read wet-lab histories, analyze data, and prioritize candidates for future experiments. However, existing benchmarks often fail to evaluate a model's ability to generalize from past experimental rounds to future ones. Standard benchmarks typically rely on random data splits, which test interpolation (predicting within known distributions) rather than the critical task of future-round discovery (ranking variants that emerge only in later experimental rounds). There is a lack of reproducible, large-scale benchmarks that preserve the chronological structure of real directed-evolution campaigns to test whether models can prioritize future wet-lab outcomes based on recorded past evidence.

Methodology

1. Benchmark Construction: TadA-Bench

The authors introduce TadA-Bench, a fixed-data, wet-lab replay benchmark derived from 31 rounds of TadA (adenosine deaminase) directed evolution.

• Scale: The dataset covers approximately one million variants observed via Next-Generation Sequencing (NGS).
• Views: It provides aligned DNA, RNA, and protein sequence views, allowing evaluation of genomic, transcript, and protein language models on the same underlying campaign.
• Protocol: The benchmark defines a "future-round replay" task. Models are trained on variants and measurements from earlier rounds ($D_{\le k}$) and evaluated on their ability to rank variants that appear only in later rounds ($D_{> k}$). In the primary instance, models train on rounds 1–27, validate on round 28, and test on rounds 29–31. The splits are sequence-disjoint to prevent memorization.

2. Data Integration: Seq2Graph

A core methodological contribution is Seq2Graph, a graph-based pipeline designed to unify noisy, multi-round enrichment measurements into consistent activity labels.

• Challenge: Raw selection data contains batch effects, noisy enrichment estimates, and local pairwise inconsistencies across 31 rounds.
• Mechanism:
  • Graph Construction: Within each round, variants are ranked by read-count enrichment. A directed graph is built where edges connect adjacent variants in the sorted list, weighted by local enrichment ratios.
  • Anchoring: Variants appearing in multiple rounds (sequence overlaps) act as "anchors" to connect local ranking graphs across the campaign.
  • Inconsistency Elimination: The pipeline formulates the removal of contradictory cycles (e.g., $A > B > C > A$) as a Weighted Feedback Arc Set problem. A greedy heuristic is used to remove low-confidence edges, transforming the graph into a Directed Acyclic Graph (DAG).
  • Activity Assignment: An activity scale is anchored at a reference variant (TadA8e = 1.0). Scores are propagated in the log domain (multiplicative ratios) from the reference to all other nodes via the fewest-edge path.
• Output: This produces sequence-defined relative activity labels for DNA, RNA, and protein views without inferring biological ancestry or lineage trees.

3. Evaluation Framework

The evaluation treats biological language models as a "ranking module" for an agentic loop.

• Models: The study evaluates DNA models (Evo 2, Nucleotide Transformer), RNA models (OmniGenome), and protein models (ESM2, ProtTrans, ESMC).
• Protocols:
  • Frozen Encoder: Primary evaluation freezes the pre-trained encoder and trains a downstream regression head.
  • Adaptation Checks: Includes full fine-tuning and prompt tuning to rule out protocol artifacts.
  • Metrics: Spearman's rank correlation ($\rho$), Recall@10%, and nDCG@10%.
  • Discovery Mode: Evaluates candidate prioritization under finite budgets (96 and 384 variants, simulating standard plate sizes).

Key Results

1. The Future-Round Gap

Representative biological language models exhibit a stark performance gap between interpolation and future-round discovery:

• Interpolation (Random Split): Under an 8:1:1 random split, protein models achieve strong Spearman correlations ($\rho \approx 0.8$), indicating the label space is learnable.
• Future-Round Discovery: Under the fixed chronological split, performance degrades sharply. Spearman correlations drop to $\rho \approx 0.03$–$0.05$ across DNA, RNA, and protein views.
• Adaptation: Full fine-tuning and prompt tuning fail to close this gap (best observed $\rho \approx 0.06$), suggesting the difficulty lies in the representations' ability to prioritize future evidence, not just the regression head.

2. Discovery-Mode Prioritization

In a "discovery mode" setting where only a limited number of candidates (e.g., 96) can be tested:

• Models fail to enrich for true high-activity future variants.
• Hit rates for top-5% candidates are close to random chance (approx. 3–5% vs. 5% expected), indicating that current models cannot effectively identify promising variants for the next experimental round.

3. Evolutionary Coverage vs. Local Density

The authors investigated whether training data composition affects future-round performance:

• Density: Simply increasing the number of samples from the same evolutionary rounds (random subsampling) yields diminishing returns.
• Coverage: Subsets that prioritize evolutionary coverage (selecting variants from diverse rounds or maximizing similarity to the target validation set) outperform larger, denser subsets from limited rounds.
• Conclusion: On TadA-Bench, covering broad evolutionary or functional regions is more informative for future-round discovery than increasing local data density.

4. Robustness

• Biological Validation: Orthogonal low-throughput GFP reporter assays on key variants showed strong agreement ($\rho > 0.99$) with the high-throughput NGS-derived labels.
• Computational Robustness: Bootstrapping analyses (subsampled rounds or sequences) confirmed that the Seq2Graph pipeline produces stable scores ($\rho > 0.90$), validating the label consistency.

Significance and Claims

The paper positions TadA-Bench as a reproducible substrate for testing the "ranking" component of future agentic protein engineering systems before they are deployed in costly closed-loop wet-lab campaigns.

• Exposing a Bottleneck: The benchmark reveals that current biological foundation models, while successful at interpolation, struggle significantly with the specific task of ranking variants that emerge in future experimental rounds. This is identified as a concrete bottleneck for iterative protein engineering workflows.
• Benchmark Design: It argues that benchmarks for protein discovery must preserve campaign structure (chronology and cross-round consistency) rather than treating all variants as independent data points. The distinction between "local density" and "evolutionary coverage" is highlighted as a critical design principle.
• Scope: The authors explicitly state that TadA-Bench isolates the ranking problem. It does not evaluate planning policies, tool use, or autonomous execution, but provides a stable, fixed-data test for whether representations can utilize recorded experimental history to prioritize future candidates.

The authors release the data, code, and baseline protocols on Hugging Face and GitHub to enable reproducible comparisons of models on this specific, recorded experimental trajectory.