🤖 AI

TxBench-PP: Analyzing AI Agent Performance on Small-Molecule Preclinical Pharmacology

This paper introduces TxBench-PP, a verifiable benchmark for evaluating AI agents on small-molecule preclinical pharmacology tasks using real-world assay data, revealing that even the most advanced current models fail to reliably recover accurate preclinical decisions.

Original authors: Hannah Le, Ramesh Ramasamy, Alex Urrutia, Mahsa Yazdani, Tim Proctor, Kenny Workman

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hannah Le, Ramesh Ramasamy, Alex Urrutia, Mahsa Yazdani, Tim Proctor, Kenny Workman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of super-smart, AI-powered research assistants to help you decide which new medicines to build. You have a massive pile of messy, real-world lab data—charts, microscope photos, and chemical test results—and you need these assistants to look at the evidence and tell you: "This drug looks promising, let's keep it," or "This one is dangerous or useless, let's throw it out."

This paper, TxBench-PP, is a report card on how well these AI assistants actually performed that specific job.

The Big Idea: A "Driver's Test" for AI

The authors created a special test called TxBench-PP. Think of it like a driving test, but instead of a car, the AI is driving a drug discovery program.

The Trap: They didn't just ask the AI to recite facts from a textbook (like "What does aspirin do?"). That's easy for AI because they memorized the internet.
The Real Challenge: They gave the AI a fresh, messy folder of new data from a real lab experiment and asked, "Based only on these specific numbers and images, what should we do?"
The Goal: To see if the AI can act like a scientist who looks at the evidence, or if it just guesses based on what it remembers from its training.

The Results: The AI is Still a Rookie

The researchers tested 16 different combinations of AI models (the "brains") and software tools (the "hands"). They ran the test 4,800 times in total.

Here is the verdict: The AI is not ready to drive alone yet.

The Best Score: The top-performing AI system got about 59% of the answers right. In a classroom, that's a failing grade. In a real drug company, getting it wrong 4 out of 10 times is dangerous because it could mean wasting millions of dollars or, worse, moving a dangerous drug forward.
The Reality Check: Even the "smartest" AI failed to get the right answer on the same task three times in a row. It was inconsistent.

Where Did the AI Go Wrong?

The authors looked closely at the mistakes, and they found the AI wasn't just "forgetting" things. It was making specific types of scientific errors:

The "Textbook" Trap: Sometimes the AI saw a picture of a cell dying and ignored the actual data in front of it. Instead, it remembered a famous story from a textbook about a different drug and applied that story here. It ignored the evidence to fit a pre-written narrative.
Bad Math and Bad Filters: The AI often messed up the "quality control." Imagine a scientist looking at a graph and saying, "Oh, that weird spike is just a glitch, I'll ignore it," when actually, that spike was the most important part of the data. The AI frequently threw away important data or kept garbage data.
The "All-or-Nothing" Problem: When asked to pick the best drug from a list of 10, the AI often picked the one that looked "okay" but missed the one that was truly great, or it picked a dangerous one because it looked active. It struggled to make the hard choice of "kill this project" vs. "advance this project."

The "Tool" Matters

One interesting finding was that the software the AI used to do the work mattered just as much as the AI itself.

Think of the AI as a brilliant chef.
Harness A gave the chef a sharp knife and a clean cutting board.
Harness B gave the same chef a dull butter knife and a dirty board.
The chef with the good tools (called "Pi" in the paper) cooked much better meals than the same chef with the bad tools, even though the "chef" (the AI model) was identical.

The Bottom Line

This paper is a reality check. While AI promises to speed up drug discovery, it cannot yet be trusted to make the critical "go/no-go" decisions on its own.

The current AI agents are like interns who are very good at reading the manual but still need a senior scientist to look over their shoulder, check their math, and make sure they aren't ignoring the messy reality of the lab data. They are helpful, but they are not yet reliable enough to run the show.

Technical Summary: TxBench-PP

Problem Statement

Artificial intelligence (AI) agents hold the promise of accelerating drug discovery by compressing interpretation and decision-making loops. However, the practical deployment of these agents in industrial workflows is hindered by a lack of trusted, standardized evaluation methods. Existing biology-agent benchmarks often emphasize verifiable analysis tasks but struggle to capture the complexity of the drug discovery ecosystem, which spans diverse assay categories, development stages, and therapeutic modalities.

Specifically, preclinical small-molecule pharmacology involves critical decisions regarding potency, target engagement, mechanism of action (MoA), pharmacodynamics (PD), safety, and in vivo efficacy. These decisions determine whether a molecular candidate is advanced, held, repeated, or killed. Such decisions are often upstream of costly experiments and rely on fragmented expert judgment. A significant challenge is distinguishing whether an agent can derive accurate conclusions from supplied real-world data versus relying on memorized literature facts or well-understood mechanisms. Current systems lack reliable verification on realistic program decisions derived from raw assay data.

Methodology

The authors introduce TxBench-PP (TherapeuticsBench Preclinical Pharmacology), a verifiable benchmark designed to evaluate AI agents on small-molecule preclinical pharmacology. It serves as a focused slice of the broader TherapeuticsBench roadmap.

Benchmark Construction

Scope: The benchmark consists of 100 evaluations indexed by program stage, assay type, and task structure. It covers eight program stages: disease/model context, screening/hit confirmation, drug response/pharmacogenomics, causal target validation, MoA/PD, compound-target characterization, developability/safety/PK, and translational efficacy. (Human genetic target support is reserved for future work).
Data & Context: Each evaluation provides realistic workflow artifacts (e.g., dose-response screens, proteomics, imaging, DMPK data) from a specific local decision point. Agents operate in a coding environment to inspect files.
Task Design: Tasks are intentionally designed to penalize memorized solutions. The intended answer must be supported strictly by the supplied data artifacts, not by external knowledge. Many tasks include "decision-blocking gaps" (e.g., tox-species mismatch, insufficient free-drug coverage) that are hidden from the agent but used for post-hoc failure analysis.
Grading: Answers are structured and graded deterministically. Graders check for label sets, numerical tolerances, rank-order relationships, and categorical choices. A trajectory passes only if the final structured answer satisfies all task-specific criteria.

Experimental Setup

Models & Harnesses: The study evaluated 16 model–harness configurations, comprising 11 distinct models (including Claude Opus 4.6–4.8, GPT-5.4–5.5, Gemini 3.1 Pro–3.5 Flash, Kimi K2P6, Grok 4.20–4.3, and Sonnet 4.6) across three agent harnesses: Pi, Claude Code, and OpenAI Codex.
Execution: Each configuration was run on all 100 tasks with three independent attempts, generating 4,800 agent trajectories.
Analysis: The authors performed manual trajectory reviews on 1,834 failing trajectories to categorize errors into scientific judgment gaps (method, calibration, perception) and behavioral issues (overfitting, under-investment).

Key Results

Overall Performance

No system reliably recovered preclinical pharmacology decisions. Even the strongest configuration, Claude Opus 4.8 / Pi, achieved an endpoint pass rate of 59.3% (178/300; 95% CI, 51.1–67.6). The second-best, GPT-5.5 / Pi, passed 55.3% (166/300).

Reliability: The top configuration failed 122 of 300 attempts and passed only 41 of 100 tasks in all three replicates.
Confidence Intervals: Confidence intervals for the top models overlap, indicating a "leading group" rather than a single clear winner.

Error Analysis

Manual review of failures revealed that most errors were legitimate gaps in scientific judgment rather than simple data retrieval failures:

Scientific Errors (71% of assignable failures): Dominated by method errors (skipped QC, incorrect statistical choices, wrong data layers) and calibration errors (inappropriate cutoffs).
Behavioral Patterns: 67% of failures involved agents that engaged with data and performed plausible analyses but reached incorrect conclusions. Only 12% abandoned correct answers, and 9% relied on memorized literature.
Step Count: Increased effort (step count) did not reliably correlate with accuracy. For instance, Gemini 3.5 Flash took the most steps (55.0) but achieved a lower pass rate (51%) than models with fewer steps.

Performance by Task and Stage

Program Stages: Pass rates varied significantly by stage, ranging from 27% in screening/hit prioritization to 55% in drug response.
Task Types:
- High Performance: Exposure/PK and biomarker discovery tasks averaged ~57% pass rates. These often required computed values or binary calls.
- Low Performance: Hit prioritization and cheminformatics tasks averaged ~24%. These required selecting a defensible subset from many plausible candidates. No model exceeded 45% on these tasks.
Program Decisions: On complex "advance/hold" decisions requiring synthesis of multiple endpoints, models passed only 35% of attempts. Notably, overall benchmark scores did not predict performance on these specific program decisions (Spearman $\rho$ = 0.08).

Harness Impact

The choice of agent harness significantly influenced performance. Within the same model family, the Pi harness consistently outperformed Claude Code and OpenAI Codex (e.g., +4.4 to +5.5 percentage points). This suggests that reported scores are properties of the model–harness configuration, not the model alone.

Significance and Claims

The paper claims that TxBench-PP provides a necessary "measuring stick" for evaluating agentic systems on local preclinical pharmacology decisions.

Current State: While agents can complete some tasks correctly, they are not yet reliable as scientific assistants for preclinical pharmacology. They frequently fail to distinguish technical artifacts from biological signals and struggle with calibrated decision-making in complex, multi-evidence scenarios.
Scope Limitation: The authors explicitly state that TxBench-PP is a focused benchmark for small-molecule preclinical pharmacology. It should not be used to make general claims about other areas such as human genetic target support, full disease-model selection, clinical translation, or non-small-molecule modalities (e.g., biologics, oligonucleotides).
Future Direction: The work establishes a foundation for the broader TherapeuticsBench roadmap, with future efforts planned to extend downstream to clinical tasks, vertically to pharmacogenomics, and across modalities to antibodies, cell therapies, and gene therapies.

In summary, the paper concludes that substantial work remains before AI agents can be integrated reliably and reproducibly into preclinical biology workflows, as current systems exhibit significant gaps in scientific judgment and decision-making robustness.