🤖 machine learning

PerturbPFN: Probing the Limits of Synthetic Priors in Drug Perturbation Modelling

PerturbPFN is a novel amortized model that predicts cellular responses to unseen chemical perturbations by inferring latent system graphs and intervention targets from synthetic priors, achieving competitive accuracy and interpretability without requiring test-time gradient updates.

Original authors: Yuche Gao, José Miguel Hernández-Lobato, Siyuan Guo

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuche Gao, José Miguel Hernández-Lobato, Siyuan Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out why a city's traffic suddenly grinds to a halt. You can see the cars stopped (the result), but you can't see the accident, the broken traffic light, or the spilled coffee that caused it (the cause). In the world of biology, scientists face a similar mystery every day. They want to know how a new drug will change the behavior of a cell. Cells are like tiny, bustling cities with thousands of genes acting as workers. When a drug (a chemical perturbation) enters, it might trip up a specific worker, causing a chain reaction that changes how the whole city operates. The problem is that there are millions of possible drugs, and testing them all in a real lab is too slow, too expensive, and often impossible because we don't know exactly which "worker" the drug will target or how strong that hit will be.

To solve this, scientists have built computer models that try to predict these outcomes. Some models are like black boxes: they look at past data and guess the future result without explaining how or why the change happened. Others try to be more like mechanics, looking for the specific broken part, but they often need to be re-tuned from scratch for every new drug or cell type, which is time-consuming. The big question is: Can we build a model that learns the general "rules of the road" for how cells react to drugs, so it can instantly guess what happens with a brand-new drug it has never seen before, while also telling us exactly which genes it thinks the drug is hitting?

This is where a new tool called PerturbPFN comes in. Think of PerturbPFN as a super-smart apprentice mechanic who has spent years watching millions of simulated traffic jams in a video game. Instead of learning from real-world accidents (which are rare and messy), this apprentice learned from a computer program that generated billions of fake but realistic scenarios. In these simulations, the program knew exactly which "traffic light" was broken and how hard it was broken. By studying these synthetic examples, PerturbPFN learned to recognize patterns: "Oh, when the traffic slows down in this specific pattern, it usually means the red light at Main Street was hit with medium force."

When the researchers tested this apprentice on real-world data—actual experiments where cells were treated with real drugs—they found something impressive. PerturbPFN didn't just guess the final traffic jam; it successfully guessed the hidden cause. It could predict how a cell would respond to a new drug, identify which specific genes the drug likely targeted, and even estimate how strong that drug's effect would be. It did all this in a single, lightning-fast pass, without needing to stop and re-learn anything for the new drug.

The paper shows that PerturbPFN is a strong competitor to existing methods. While some other models are slightly better at pure prediction in certain cases, PerturbPFN offers a unique trade-off: it is incredibly fast and gives scientists a "mechanistic explanation" (a guess at the target and strength) that other fast models usually can't provide. However, the authors are careful to note that because the model was trained on synthetic data, its "guesses" about the internal structure are based on the rules of that simulation. It suggests a powerful new way to learn biology, but it doesn't claim to have solved the mystery of every drug interaction perfectly. Instead, it offers a promising, fast, and interpretable way to explore the unknown corners of drug discovery, acting as a bridge between raw data and biological understanding.

Technical Summary: PerturbPFN

Problem Statement

Predicting cellular responses to unseen chemical perturbations is a central challenge in drug discovery. Unlike discrete genetic edits (e.g., CRISPR), small-molecule drugs introduce complex interventions where the target, pathway, and mechanism are often unknown, context-dependent, or only partially captured by perturbation covariates. Current predictive models face three primary hurdles:

Unknown Mechanisms: The need to infer latent atomic targets and mechanisms from observed response data rather than relying on pre-defined targets.
Generalization: The theoretical design space of small molecules vastly exceeds physical screening capabilities, requiring models to generalize to unseen perturbations rather than merely interpolating between known drugs.
Limitations of Existing Paradigms:
- Unstructured black-box models learn statistical shifts but lack interpretability.
- Structured mechanistic models offer interpretability but often rely on instance-specific optimization or dataset-specific fitting, limiting their ability to amortize knowledge across heterogeneous contexts (e.g., different cell lines or batches).
- Foundational models pre-trained on massive real-world data (e.g., scGPT, GenePT) learn robust representations but may not explicitly model the causal propagation of interventions.

Methodology

The authors propose PerturbPFN, a Prior-Data Fitted Network (PFN) style framework designed for amortized inference of perturbation effects under a hierarchical synthetic Structural Causal Model (SCM) prior.

1. Theoretical Framework

The approach models the system as an SCM $\mathcal{M} = \{G, \theta\}$ , where $G$ is a Directed Acyclic Graph (DAG) representing gene regulatory relationships, and $\theta$ parameterizes local mechanisms.

Shift Interventions: Instead of hard interventions (removing parent dependence), PerturbPFN models perturbations as additive shifts to local mechanisms: $x_i = f_i(x_{G_i}; \theta_i) + \psi_i + \epsilon_i$ , where $\psi_i$ is the intervention strength for targeted nodes $I$ .
Episodic Prediction: The task is defined as predicting the posterior predictive distribution $p(x_j | \gamma_j, D_{ctx})$ for a query regime $\gamma_j$ given context observations $D_{ctx}$ . The model approximates this by inferring point estimates of the latent graph $G$ , intervention targets $I$ , and strengths $\psi$ .

2. Hierarchical Synthetic Prior

PerturbPFN is trained entirely on prior-predictive synthetic episodes, avoiding the need for real perturbation responses during training.

Graph Generator: Samples gene-regulatory networks (GRNs) with module structures to induce heterogeneous, scale-free topologies.
Perturbation Generator: Descriptors $\gamma$ generate sparse atomic targets from a truncated geometric distribution. Intervention strengths follow Hill-type dose-response functions.
Observation Model: Synthetic data is sampled from a zero-inflated log-normal expression model with SERGIO noise injection.
Counterfactual Supervision: Exogenous noise is reused across factual and intervened rollouts to ensure paired differences reflect intervention-induced changes.

3. Model Architecture

The model utilizes a transformer-based amortized predictor with a structured bottleneck:

Context Intervention Memory: A shared encoder processes context pairs $(\gamma_i, x_i)$ to produce node representations and regime summaries. Target and strength heads predict $\hat{I}_i$ and $\hat{\psi}_i$ for context regimes, which are encoded into intervention-memory tokens.
Structure Branch: A set-to-graph encoder combines context response statistics with inferred interventions to estimate the system graph $\hat{G}$ . It employs a NOTEARS-style penalty ( $L_{dag}$ ) to enforce acyclicity and a greedy projection to calibrate edge density.
Query Inference: The query regime is conditioned on the context memory. The model predicts query-specific targets $\hat{I}_j$ and strengths $\hat{\psi}_j$ .
SCM Decoder: An intervened SCM decoder rolls out the effects. It uses a hard projected graph $\hat{G}$ and the predicted interventions to propagate shifts through the latent system, outputting a zero-inflated log-normal predictive distribution.

4. Training and Inference

Training Objective: A multi-task loss combines graph loss ( $L_G$ ), target loss ( $L_I$ ), strength loss ( $L_\psi$ ), observation likelihood ( $L_{obs}$ ), and an acyclicity penalty ( $A_{dag}$ ).
Inference: At test time, PerturbPFN performs a single forward pass without gradient updates. It converts continuous edge scores into a hard DAG and rolls out the decoder to predict effects.

Key Contributions

Framework: Introduction of PerturbPFN, a PFN-style framework that amortizes inference for perturbation effects using a hierarchical synthetic SCM prior. It explicitly predicts latent graphs, sparse targets, and strengths before decoding responses, rather than regressing high-dimensional outputs directly.
Efficiency and Generalization: Demonstrates competitive effect prediction on real single-cell data through a single in-context forward pass, eliminating the need for dataset-specific optimization at test time.
Interpretability: Provides structured intermediate estimates (targets, strengths, system structure) that offer mechanistic insights, contrasting with black-box latent space predictions.
Evaluation: Comprehensive assessment across in-prior latent recovery, real-world effect prediction (Sci-Plex dataset), and regulatory structure discovery (GeneRNIB benchmark).

Results

In-Prior Latent Recovery

On 1,024 synthetic episodes, PerturbPFN successfully recovered latent variables without collapse:

Graph Recovery: AUROC of 0.8133 and AUPRC of 0.2980 (significantly above the 0.0363 edge prevalence).
Target Identification: AUROC of 0.9330 and F1 of 0.7649.
Strength Estimation: Mean absolute strength ratio of 0.9386.
Response Prediction: High cosine similarity (0.7437) between predicted and true intervention directions.

Perturbation Effect Prediction (Sci-Plex)

Evaluated on the 12-protocol highest-dosage holdout benchmark:

Performance: PerturbPFN achieved the best normalized Wasserstein-2 ( $W_2$ ) score (0.9183) and ranked second overall (0.8394 average score), behind only the RF X-Learner.
Efficiency: Once pretrained, it requires 1.11 seconds per protocol for inference. This is significantly faster than GIM (4495s) and CondOT (7266s). Even with amortized pretraining costs, it remains substantially faster than complex baselines.
Trade-off: It offers a favorable balance between predictive performance, structured interpretability, and low inference cost.

GRN Structure Discovery (GeneRNIB)

Evaluated on the 300BCG benchmark for transferring learned structure to real biological data:

Performance: PerturbPFN achieved the best GS-F1 (0.6410), indicating strong alignment with active pathways, and competitive TFB-F1 (0.1146).
Limitations: It did not outperform dedicated GRN baselines (e.g., GRNBoost2, PORTIA) on R-Precision, R-Recall, or Virtual Cell (VC) metrics. The authors note this is expected, as PerturbPFN is optimized for causal propagation under a synthetic prior, not for reproducing static curated GRN annotations.

Significance and Claims

The paper positions PerturbPFN as a step toward structured foundation models for biology. Its primary significance lies in demonstrating that:

Synthetic Pretraining is Viable: A model can be pretrained on large-scale synthetic tasks to amortize the inference of complex latent variables (graphs, targets, mechanisms) without requiring real labeled perturbation data for training.
Amortized Inverse Problems: The transformer effectively amortizes an inverse problem—inferring which latent graph and mechanism changes explain observed contexts—and applies this procedure to new tasks without gradient-based adaptation.
Structured Bottleneck: The structured bottleneck serves a dual purpose: it acts as an architectural constraint for efficiency and exposes inspectable hypotheses about how perturbations act and propagate.

The authors maintain a modest stance, acknowledging that the meaning of latent variables is tied to the synthetic prior and that recovering them in-prior does not guarantee unique identifiability of biological mechanisms. They suggest future work should focus on richer simulators, posterior uncertainty quantification, and scaling to full transcriptomes.