🧬 biology

Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction?

Despite their synthetic causal pretraining, tabular in-context learners like TabPFN3 and TabICL prove to be competitive, data-efficient predictors for biomolecular property tasks, though their effectiveness is heavily contingent on the quality of the underlying molecular or protein representations.

Original authors: Davy Guan, Lu Zhang, Asiri Wijesinghe, Allen Zhu, He Zhao, Helen Power, F. Hafna Ahmed, Andrew Warden, Cheng Soon Ong, Daniel M. Steinberg

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Davy Guan, Lu Zhang, Asiri Wijesinghe, Allen Zhu, He Zhao, Helen Power, F. Hafna Ahmed, Andrew Warden, Cheng Soon Ong, Daniel M. Steinberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Question: Can a "Table Expert" Understand Biology?

Imagine you have a brilliant new AI assistant. This assistant is a master at reading spreadsheets (tables of numbers). It has been trained on millions of fake spreadsheets to learn how to spot patterns, predict outcomes, and make smart guesses based on very few examples. This is what the paper calls a "Tabular In-Context Learner" (specifically models named TabPFN3 and TabICL).

The scientists asked a weird question: If we feed this spreadsheet expert data about proteins and molecules, will it work?

Usually, these AI models are trained on synthetic data (made-up numbers) that follow simple cause-and-effect rules. Proteins and molecules, however, are messy, complex biological things. It seemed unlikely that a model trained on "fake math tables" could understand "real biology."

The Setup: Translating Biology into a Spreadsheet

To test this, the researchers had to translate biology into a language the spreadsheet expert could understand.

The Protein Translator (ESMC): Think of a protein as a long, complicated sentence made of amino acids. The researchers used a pre-trained "translator" (a model called ESMC) to turn that long sentence into a single, fixed-length list of numbers (a vector). It's like taking a 500-page novel and summarizing it into a single 960-digit phone number that captures the story's essence.
The Molecule Translator: For small molecules (like drugs), they used standard chemical "fingerprints" (ECFP) and physical property lists (RDKit). These are also just lists of numbers.

Once translated, the protein or molecule became just another row in a spreadsheet. The AI's job was to look at a few rows with known answers (e.g., "This protein works well") and guess the answer for a new row (e.g., "Will this new protein work well?").

The Results: How Did They Do?

The researchers tested this on two different types of biological puzzles.

1. The Protein Puzzle (ProteinGym & PpEST)

The Task: Predicting how well a protein works (its "fitness") based on very few examples.
The Analogy: Imagine you are trying to guess the score of a new sports team based on the stats of just 8 to 64 previous games.
The Outcome: It worked surprisingly well. The spreadsheet experts (TabPFN3 and TabICL) were just as good as, or even better than, traditional methods that are specifically designed for biology.
- TabPFN3 was the slight winner overall.
- They didn't need to retrain the model; they just looked at the examples provided in the "context" (the few known data points) and made a prediction.
- Key Insight: As long as the "translator" (ESMC) did a good job turning the protein into numbers, the spreadsheet expert could figure out the rest.

2. The Molecule Puzzle (Drug Discovery)

The Task: Predicting if a small molecule (a drug candidate) has certain properties, like being toxic or effective.
The Analogy: Trying to guess if a new recipe will taste good based on a few ingredients.
The Outcome: It was a mixed bag. The spreadsheet experts were competitive, but they didn't win every time.
- Sometimes the spreadsheet expert won; sometimes a model that looks at the molecule's 3D shape (like a graph) won.
- The Twist: The result depended heavily on how the molecule was translated into numbers. If you used one type of "fingerprint" (ECFP), the spreadsheet expert might win. If you used a different one (RDKit), a different model might win.
- Key Insight: For molecules, the "translator" matters just as much as the "predictor." There is no single "best" combination for all drug types.

The "Gotchas" (Limitations)

The paper is very honest about where the method struggles:

The "Hard" Tests: When the scientists made the test harder (by splitting the data in tricky ways, like grouping similar proteins together), the spreadsheet experts got worse. This means they are great at spotting patterns in random data, but they can get confused if the test data is too similar to the training data in specific ways.
The "Black Box" Problem: You can't just throw raw biology at these models. You must use a specific translator first. If you pick the wrong translator, the model fails.
Out-of-Distribution: When testing on completely new types of molecules (ones the model has never seen before), the spreadsheet experts didn't always generalize well. They are good at interpolation (guessing between known points) but struggle with extrapolation (guessing outside the known range).

The Bottom Line

The paper concludes that Tabular In-Context Learners are a powerful new tool for biology, but they aren't magic.

For Proteins: They are excellent. If you have a good translator (ESMC) and a few data points, these models can predict protein fitness very accurately.
For Molecules: They are useful, but you have to be careful. You need to pick the right "fingerprint" for the specific drug you are studying.

The Main Takeaway: Don't treat these AI models as magic black boxes that understand biology on their own. They are like specialized calculators. They are incredibly fast and accurate at solving math problems, but only if you first translate the biology problem into the right kind of math problem for them to solve.

Technical Summary: Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction?

Problem Statement

Predicting biomolecular properties from limited labeled data is a critical bottleneck in protein engineering and small-molecule design. While large biological foundation models (e.g., protein language models) have successfully generated rich, fixed-length sequence representations, the challenge has shifted to designing data-efficient predictors capable of exploiting these embeddings in few-shot regimes (often with only $O(10)$ labeled examples).

The paper investigates whether Tabular Foundation Models—specifically TabPFN3 and TabICL—can serve as effective predictors for biomolecular tasks. These models are in-context learners pretrained on synthetic tables derived from random causal graphs. The central hypothesis is counter-intuitive: despite having no inherent prior knowledge of protein sequences or molecular graphs, these tabular models might generalize effectively if paired with appropriate domain-specific feature representations.

Methodology

The authors evaluate tabular in-context learning (ICL) across two distinct domains: protein fitness regression and small-molecule property classification. The experimental design treats the system as a predictor–representation pair, keeping feature pipelines fixed to isolate the performance of the predictor.

1. Protein Fitness Regression

Representation: Sequences are encoded using ESMC (a 300M-parameter protein language model), producing fixed-length 960-dimensional embeddings.
Predictors: TabICL and TabPFN3 are compared against supervised baselines (Ridge Regression, RBF Sampler, HistGradientBoostingRegressor, and fine-tuned ESMC with an MLP head).
Datasets:
- ProteinGym: 217 Deep Mutational Scanning (DMS) assays. Evaluations include random 5-fold splits and official "harder" holdout schemes (modulo and contiguous).
- PpEST: A diverse esterase family dataset with catalytic rates and thermostability measurements.
Metrics: Mean Squared Error (MSE) and Spearman rank correlation. Few-shot performance is evaluated across support sizes of 8, 16, 32, and 64.

2. Small-Molecule Property Classification

Representations: Fixed molecular descriptors including ECFP (Morgan fingerprints), RDKit physicochemical descriptors, and their concatenation.
Predictors: TabICL, TabPFN3, and XGBoost (all using fixed descriptors) are compared against graph-based baselines: ChemProp (trained from scratch) and ChemProp+CheMeleon (foundation-model initialized).
Datasets:
- TDC ADMET: 13 endpoints for absorption, distribution, metabolism, excretion, and toxicity.
- MoleculeNet: 806 registered endpoints (excluding PCBA for TabPFN3 due to resource constraints).
- FS-Mol: 157 few-shot molecular property tasks.
- DrugOOD: 3 tasks specifically designed to test Out-of-Distribution (OOD) generalization across assay, scaffold, and molecular-size shifts.
Metrics: ROC-AUC. For OOD tasks, the gap between In-Distribution (ID) and OOD performance is analyzed.

Key Results

Protein Prediction

Performance: TabPFN3 and TabICL are consistently competitive, often outperforming classical baselines and fine-tuned ESMC.
- On ProteinGym (random 5-fold), TabPFN3 achieved the highest mean Spearman correlation (0.767) and lowest MSE (0.351), closely followed by TabICL (0.753).
- On PpEST, both tabular models provided the strongest overall results, with TabICL showing the lowest MSE on three of five endpoints and TabPFN3 on the other two.
Few-Shot Capability: Both models demonstrated strong rank preservation and error reduction as support set sizes increased from 8 to 64, effectively exploiting local structure in the ESMC embedding space without gradient updates.
Generalization: Performance dropped significantly on the "harder" modulo and contiguous holdout schemes compared to random splits, indicating that split choice materially affects generalization estimates.

Small-Molecule Prediction

No Single Dominant Pair: Unlike the protein domain, no single learner–representation pair dominated across all molecular benchmarks.
- TDC ADMET: TabPFN3 with RDKit descriptors was strongest.
- FS-Mol: TabPFN3 with ECFP+RDKit was strongest.
- MoleculeNet: TabPFN3 with RDKit led (on the available subset).
- DrugOOD (OOD): ChemProp+CheMeleon (a graph-based foundation model) outperformed all tabular pairs on OOD ROC-AUC.
Representation Sensitivity: The choice of molecular descriptor (ECFP vs. RDKit) was often as critical as the choice of the predictor. TabPFN3 and TabICL did not uniformly dominate each other; their relative performance flipped depending on the benchmark and representation.
OOD Generalization: Tabular models showed positive ID/OOD gaps on DrugOOD, indicating that in-distribution performance does not guarantee robustness under scaffold or assay shifts.

Key Contributions

Empirical Benchmark: The paper provides the first systematic audit of tabular foundation models (TabPFN3, TabICL) on biomolecular prediction tasks, covering both protein regression and small-molecule classification.
Learner–Representation Pairing: It establishes that these models should be evaluated not as standalone architectures but as part of a specific pairing with domain representations (e.g., ESMC for proteins, RDKit/ECFP for molecules).
Few-Shot Validation: The study demonstrates that tabular in-context learners can effectively leverage pretrained biological embeddings in low-data regimes, offering a practical alternative to task-specific fine-tuning.
Rigorous Evaluation Protocols: The authors separate few-shot from full-training conclusions, explicitly test OOD generalization, and analyze support-set sensitivity to avoid over-optimistic claims.

Significance and Claims

The paper concludes that tabular foundation models are viable and strong performers for biomolecular property prediction, but their success is contingent on the quality of the input representation.

For Proteins: When paired with high-quality embeddings like ESMC, tabular ICL models are highly competitive, often surpassing traditional supervised baselines and fine-tuned models in few-shot settings.
For Small Molecules: The results are more nuanced. While tabular models are competitive, they do not universally dominate graph-based foundation models, particularly in Out-of-Distribution scenarios. The choice of molecular descriptor is a primary determinant of success.

The authors emphasize that these models are not "representation-free black boxes." Their utility lies in their ability to act as efficient, data-scarce predictors when the input features already encode the necessary structural and functional constraints of the biomolecules. The paper advocates for a cautious interpretation of results, noting that performance on random splits does not guarantee robustness on harder, structurally distinct holdouts.