CRC-Screen: Certified DNA-Synthesis Hazard Screening Under Taxonomic Shift

1. Problem Statement

DNA synthesis providers act as the final enforcement point against the creation of hazardous biological agents. Current screening relies on sequence-similarity searches (e.g., BLAST, DIAMOND) against curated lists of known toxins. This approach assumes that hazardous orders will resemble previously seen hazardous sequences.

However, this baseline fails under taxonomic shift (the "out-of-family" scenario), which is becoming more prevalent due to:

• Generative Protein Design: AI models create functional variants with drifted primary sequences.
• Fragmented Databases: Hazardous proteins from understudied taxonomic families are sparsely represented in reference lists.

The Failure Mode: When a hazardous sequence belongs to a taxonomic family absent from the reference set, sequence similarity scores drop significantly. To satisfy a safety constraint (Certified Miss Rate), Conformal Risk Control (CRC) is forced to lower the detection threshold so drastically that it flags every benign order, resulting in a 100% False Positive Rate (FPR).

2. Methodology: CRC-Screen

The authors propose CRC-Screen, a system that moves beyond pure sequence matching by fusing multiple signals derived from the synthesis order's public annotation (protein name, organism, keywords, function description) and calibrating the decision threshold using Conformal Risk Control (CRC).

A. Three Input Signals

The system derives three distinct signals for each order:

1. Homology Signal ($s_{hom}$): A k-mer Jaccard similarity (k=5) between the query sequence and known hazards. This serves as the baseline but is acknowledged to be weak under taxonomic shift.
2. LLM Panel Score ($s_{LLM}$): A panel of five Large Language Models (Claude, GPT, Llama, Qwen, GLM) reads the annotation text and outputs a hazard probability. Scores are aggregated via a trimmed mean (dropping the highest and lowest) to mitigate individual model biases.
3. Embedding Distance ($s_{emb}$): The text annotation is encoded using a large text-embedding model. The score is the maximum cosine similarity between the query embedding and the centroids of clusters formed by known hazard embeddings.

B. Monotone Logistic Aggregator

The three signals are fused using a logistic regression model with a non-negativity constraint on weights ($w_k \geq 0$).

• Rationale: This ensures the aggregator is monotone; increasing any single signal score can only increase (or maintain) the final hazard probability, never decrease it. This prevents the system from "canceling out" a strong hazard signal with a weak benign signal.
• Training: Weights are learned on the training fold; if a weight is negative, that signal is dropped, and the model is refitted.

C. Calibration via Conformal Risk Control (CRC)

Instead of a fixed threshold, the system uses CRC to set a threshold ($\hat{\tau}$) that guarantees an upper bound on the Expected False Negative Rate (FNR).

• Constraint: $E[FNR] \leq \alpha$.
• Mechanism: The threshold is chosen based on the empirical loss distribution of a calibration set.
• Handling Distribution Shift: Since the test distribution (held-out taxonomic family) differs from the calibration distribution, the authors incorporate a Total Variation (TV) distance term into the bound to account for non-exchangeability:
  $$E[FNR] \leq \alpha + TV(\text{cal}, \text{test})$$
• Leak Control: To prevent data leakage, the homology and embedding reference sets are recomputed for each fold using only the training hazards, ensuring the test family does not influence its own scoring.

3. Key Contributions

1. Demonstration of Baseline Collapse: The paper empirically proves that sequence-similarity-only screening incurs a 100% False Positive Rate under taxonomic-family holdout when trying to maintain a non-trivial safety guarantee.
2. Leak-Controlled Protocol: A rigorous evaluation protocol where reference sets are rebuilt per-fold to prevent the test family's hazards from contaminating the scoring of their own fold.
3. Signal Composition Strategy: Demonstrates that fusing LLM analysis and text-embedding proximity (without relying on sequence homology) achieves a 0% False Negative Rate and near-0% False Positive Rate.
4. Data vs. Algorithm Insight: Identifies that the primary bottleneck for achieving procurement-grade safety (e.g., $\alpha = 10^{-3}$) is not the algorithm, but the size of the calibration dataset.

4. Experimental Results

The system was evaluated on UniProt KW-0800 (Toxin) using a Leave-One-Taxonomic-Family-Out (LOTO) cross-validation across 10 families (e.g., Crotalinae, Conus, Lycosidae).

• Performance at $\alpha = 0.05$:

  • False Negative Rate (FNR): 0% on all 10 folds (0 missed hazards).
  • False Positive Rate (FPR): 0% on 9 out of 10 folds; 5% (1 benign flagged) on the Actiniidae fold.
  • Bound Validity: The empirical FNR remained well within the certified bound ($\alpha + TV$), which ranged from 24.3% to 49.4% due to distribution shift.

• Ablation Studies:

  • Homology Only: 100% FPR (fails completely).
  • LLM Only: 0% FNR, but 1.75% FPR.
  • Embedding Only: 0.45% FNR, 6.85% FPR.
  • LLM + Embedding: 0% FNR, 0% FPR (The optimal operating point).
  • LLM + Embedding + Homology: Adding homology slightly increased FPR (to 0.5%) without improving recall, as the homology signal acted as noise under taxonomic shift.

• Scalability Analysis:

  • Current subsample ($N_{cal} \approx 55$) limits the certifiable $\alpha$ to 1.77% due to the finite-sample slack term $1/(N_{cal} + 1)$.
  • To reach procurement-grade $\alpha = 10^{-3}$, a calibration set of $\approx 999$ hazards is required (an 18x increase). The full UniProt KW-0800 corpus (~6,000 toxins) is sufficient to close this gap.

5. Significance and Implications

• Paradigm Shift: The paper argues that DNA synthesis screening must evolve from a "sequence matching" problem to a "calibrated decision" problem. Sequence similarity alone is insufficient for novel or drifted threats.
• Operational Recommendation: Providers should integrate annotation-derived signals (LLM judges and semantic embeddings) into their screening stacks alongside sequence homology.
• Resource Allocation: The path to ultra-low miss rates (e.g., 0.1%) lies in curating larger, high-quality calibration datasets, not in developing more complex algorithms. The mathematical tools (CRC) already exist to certify these rates; the limiting factor is data volume.
• Safety: The system is designed purely for defensive screening (flagging orders for human review) and does not generate or enhance biological sequences.

In conclusion, CRC-Screen provides a mathematically certified framework for biosecurity screening that remains robust even when encountering entirely new families of toxins, provided sufficient calibration data is available.