Technical Summary: LOGICA (Logit-space Contrastive Alignment)

Problem Statement

Pretrained biological language models (BioFMs) provide normalized token probability distributions essential for sequence design, variant scoring, and mechanistic interpretation. However, these distributions are learned from broad, unlabeled corpora and lack conditioning on specific biological contexts such as interaction partners (e.g., ligands, peptides), cellular environments, or therapeutic interventions.

Existing methods for contextual matching generally fall into two categories, both of which compromise the native utility of BioFMs:

1. Latent-space approaches: These methods discard the token-level interface, learning scalar compatibility scores from pooled embeddings or contrastive latent spaces. While effective for retrieval, they lose the position-specific likelihoods required for mutation-local analysis and likelihood-based generation.
2. Conditional Masked Language Modeling (MLM): These methods preserve the token interface by fine-tuning with conditional objectives. However, standard reconstruction losses encourage token recovery but fail to explicitly separate matched (compatible) from mismatched (incompatible) contexts, particularly when paired biological data is sparse or noisy.

The core challenge is to align distinct biological modalities (e.g., protein and drug, TCR and peptide) across different vocabularies without destroying the native, token-level probabilistic interface that makes these models useful for local variant ranking and generation.

Methodology: LOGICA

The authors introduce LOGICA (Logit-space Contrastive Alignment), a framework that performs contrastive learning directly in the output-logit space of pretrained models rather than in a shared latent embedding space.

Core Architecture

• Native-Head-Preserving Adapters: LOGICA couples two pretrained biological language models (e.g., ESM-2 for proteins, SELFormer for ligands) using gated cross-modal adapters. These adapters inject contextual information into the hidden states of each model but are designed to return updates that remain compatible with each model's native token head.
• Logit-Space Scoring: Instead of pooling hidden states, LOGICA uses the resulting contextualized token log-likelihoods as the compatibility scores. For a sequence $x$ and context $y$, the score is defined as the site-averaged conditional log-likelihood:
  $$s_{\text{LOGICA}}(x, y; A) = \frac{1}{|A|} \sum_{i \in A} \log \pi_\theta(x_i | x_{\setminus i}, y)$$
  where $A$ is the set of evaluated positions (e.g., mutation sites).
• Bidirectional Alignment: The framework supports bidirectional scoring, combining likelihoods from both directions ($x|y$ and $y|x$) via a learned convex mixture to handle differences in sequence length and vocabulary entropy.
• Contrastive Training: The model is trained using an InfoNCE-style objective (or Bradley-Terry loss for pairs) where "positives" are matched biological pairs and "negatives" are constructed by holding one modality fixed and corrupting the other (e.g., via random point mutations or partner swapping).

Theoretical Advantage: Mutation-Local Cancellation

A key theoretical contribution is the mutation-local cancellation property. When comparing two variants ($x_A, x_B$) of the same wild-type sequence ($x_{wt}$) under the same context, and they share the same set of mutated positions $M$, the wild-type anchor terms in the likelihood ratio cancel out exactly:
$$s_M(x_A, y; x_{wt}) - s_M(x_B, y; x_{wt}) = \frac{1}{|M|} \sum_{i \in M} \left( \log \pi_\theta(x^A_i | \dots) - \log \pi_\theta(x^B_i | \dots) \right)$$
This reduces the ranking objective to a direct comparison of context-conditioned mutant-token likelihoods at the perturbed sites. This exact reduction is not generally available in latent-space models, which compare global sequence representations where shared terms do not algebraically cancel.

Key Contributions

1. Logit-Space Contrastive Alignment: A novel framework that aligns distinct modalities directly in the output-logit space, preserving the native token probability interface of pretrained models.
2. Gated Cross-Modal Adapters: A mechanism that contextualizes pretrained encoders without replacing their native heads, enabling the use of sparse paired data across different vocabularies without a shared tokenizer or decoder.
3. Mutation-Local Variant Ranking: A demonstration that preserving the token-level interface allows for exact cancellation of wild-type terms, enabling precise ranking of variants based solely on the likelihood of the mutated tokens in the presence of a biological context.
4. Interpretability and Generation: By retaining the native token head, LOGICA supports token-level dependency mapping (identifying which context residues influence specific sequence positions) and context-conditioned generation via Gibbs sampling.

Experimental Results

The authors evaluated LOGICA on three distinct biological tasks:

1. Protein–Ligand Binding and Resistance

• Setup: Coupled ESM-2 (protein) and SELFormer (ligand) on BindingDB data.
• Performance: LOGICA outperformed latent-contrastive baselines (LatentCA) and conditional-MLM baselines (LOGIMLM) on standard drug-target interaction benchmarks (DAVIS, BindingDB, BioSNAP).
• Drug Resistance: In drug-conditioned variant ranking (predicting resistance mutations for oncogenes like KRAS, BRAF, EGFR), LOGICA significantly improved AUC (from ~0.55 for latent baselines to ~0.65) and Spearman correlation. Latent-space methods failed to improve over random despite fine-tuning, highlighting the necessity of the mutation-local likelihood interface.
• Scaling: Performance scaled with backbone size (up to 650M parameters) and showed robustness in few-shot regimes (10–15% labeled data).

2. TCR–Peptide Recognition

• Setup: Coupled TCRLang (TCR) and ESM-2 (peptide) on IEDB data.
• Zero-Shot Ranking: LOGICA achieved state-of-the-art zero-shot performance in ranking peptide and TCR variants against experimental deep mutational scanning (DMS) data (ePytope, BATCAVE, ATLAS).
• Comparison: It significantly outperformed latent-contrastive and classifier-based baselines. Notably, while conditional MLM (LOGIMLM) performed near-randomly, LOGICA's contrastive alignment effectively separated functional near-neighbors from disrupted pairs.
• Dependency Mapping: The model successfully identified cross-modal residue dependencies that overlapped with structural contacts (AUC ~0.74 for peptide–CDR3$\beta$ contacts), demonstrating that the learned logits encode biologically meaningful structural signals.

3. Case Study: Peptide Optimization

• In a case study optimizing the NLVPMVATV peptide for TCR recognition, LOGICA ranked experimentally validated activity-enhancing variants (e.g., L2I, L2V) highly (3rd and 8th out of 171), demonstrating its utility for generative design.

Significance and Claims

The paper claims that LOGICA offers a principled alternative to latent-space alignment for biological foundation models. Its primary significance lies in preserving the probabilistic semantics of pretrained models while making them context-sensitive.

• Modularity: The framework allows external biological context to reshape token probabilities without replacing the model's learned vocabulary or architecture, making it applicable across diverse backbones (proteins, immune receptors, small molecules, DNA/RNA).
• Local vs. Global: The authors emphasize that many biological questions are inherently token-local (e.g., how a ligand changes the plausibility of a specific residue substitution). LOGICA addresses this by keeping the scoring in the output space where these local semantics are encoded, whereas latent methods dilute this signal into global embeddings.
• Trade-off: The paper modestly acknowledges that LOGICA is less computationally efficient than dual-encoder models for massive-scale retrieval (requiring $O(NM)$ pair-specific evaluations vs. $O(N+M)$ for dual encoders). Consequently, it is positioned as a reranking, variant-scoring, or generative interaction model rather than a first-stage retrieval engine.

In summary, LOGICA demonstrates that contrastive supervision can be effectively applied to sparse, cross-modal biological data by aligning models in logit space, thereby retaining the interpretability and generative capabilities of the original pretrained language models.