🤖 machine learning

ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation

The paper introduces ChemDIRT, a comprehensive benchmark designed to evaluate the robustness of chemistry-focused large language models by systematically measuring their performance and consistency across diverse instructions, molecular representations, and eight distinct task categories.

Original authors: Eric Inae, Tim Gunn, Chris Bond, Meng Jiang

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eric Inae, Tim Gunn, Chris Bond, Meng Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Science has long relied on computers to help solve problems, but recently, a new kind of computer program has emerged that can read and write human language with surprising fluency. These programs, known as large language models, have been trained on vast amounts of text and can answer questions, write stories, and even explain complex ideas. Because they are so good at handling words, scientists have begun to ask if they can also understand the language of chemistry. This field involves understanding how tiny particles called atoms stick together to form molecules, and how those molecules change during reactions. If a computer can truly understand chemistry, it could help design new medicines or discover new materials. However, there is a catch: just because a computer gives a correct answer to one question does not mean it understands the underlying science. It might simply be guessing based on the specific way the question was asked.

A team of researchers at the University of Notre Dame decided to test whether these computer programs actually possess a deep understanding of chemistry or if they are just good at playing a word game. They built a new testing ground called ChemDIRT, which is designed to see if the models can handle the same problem when it is presented in different ways. In the real world, a chemist might describe a molecule using a string of letters, a chemical name, or even a drawing. A truly intelligent system should give the same correct answer regardless of which format is used. The researchers also wanted to see if the models could handle different types of tasks, ranging from simple counting of atoms to predicting how a chemical reaction will unfold. They gathered thousands of examples covering eight different categories of chemical reasoning and tested twenty-three different computer models, including both open-source programs and those developed by major technology companies.

The results revealed a startling lack of consistency. When the researchers asked the same chemistry question using different wording, the computer programs often gave completely different answers. For one specific test involving drug toxicity, changing the phrasing of the question caused the accuracy of some models to swing wildly, sometimes by more than eighty percentage points. A model that appeared to be a genius on one version of the test would fail miserably on a version that meant exactly the same thing. This suggests that many of these programs are not reasoning through the chemistry at all; instead, they are reacting to the specific shape of the prompt. It is as if a student could solve a math problem perfectly if the numbers were written in blue ink, but would fail if the same numbers were written in red ink, even though the math was identical.

The study also found that the way a molecule is represented matters just as much as the words used to ask the question. Chemists can describe the same molecule using a string of letters, a systematic name, or a visual diagram. The researchers tested the models using these different formats and found that the programs were often confused by the change. A model that performed well when given a letter-based description might struggle significantly when shown the same molecule as a visual diagram or a formal name. Some models were excellent at translating between these formats, while others failed entirely. This inconsistency means that a model's high score on a standard test might not reflect a true understanding of chemical principles, but rather a lucky match between the test format and the model's training data.

Beyond the issue of wording and format, the researchers discovered that the models were highly specialized and often failed at tasks that required basic numerical reasoning. While some programs could generate chemical structures or predict reaction outcomes with impressive accuracy, they frequently stumbled when asked to calculate simple properties like molecular weight or to balance chemical equations. In several cases, the models produced numbers that were wildly incorrect, sometimes off by factors of billions or more. This indicates that the ability to produce fluent text or valid chemical strings does not guarantee that the computer can perform the actual calculations required in a laboratory. The strongest performers were often those that had been specifically trained for chemistry, yet even these specialized systems showed significant weaknesses when the task or the input format changed slightly.

The researchers concluded that current computer models do not yet demonstrate a reliable, general ability to reason about chemistry. Their performance is too dependent on the specific instructions they receive and the format in which the information is presented. A model that looks powerful on a single test might be fragile in a real-world setting where questions are asked in many different ways. The study suggests that to truly trust these tools, scientists need to evaluate them across a wide variety of tasks and formats, rather than relying on a single, fixed test. Until these models can handle the same problem consistently, regardless of how it is asked or how the data is shown, they remain more like sophisticated pattern matchers than true scientific reasoning engines. The path forward involves building systems that are robust to these variations, ensuring that when a computer says it understands chemistry, it truly does.

Technical Summary: ChemDIRT

Problem Statement

The rapid integration of Large Language Models (LLMs) into scientific domains, particularly chemistry, has outpaced the development of rigorous evaluation frameworks. Existing chemistry benchmarks (e.g., ChemLLMBench, ChemBench4K, Mol-Instructions) suffer from three primary limitations that lead to an overestimation of model capabilities:

Narrow Task Coverage: They focus on limited subsets of reasoning tasks, failing to span the full spectrum of chemical reasoning (e.g., design, regression, mechanistic, relational).
Instruction Sensitivity: Benchmarks often employ a single fixed prompt template per task. However, LLMs are known to be highly sensitive to semantic variations in wording, where equivalent prompts can yield drastically different performance, conflating prompt engineering with genuine reasoning ability.
Representation Dependence: Most evaluations rely exclusively on SMILES strings. Recent evidence suggests models exhibit low consistency across different molecular representations (e.g., IUPAC, InChI, SELFIES), and performance may depend on tokenization characteristics or pretraining corpus prevalence rather than robust chemical understanding.

Consequently, reported performance on single-format, single-prompt benchmarks may not reflect a model's true ability to reason consistently across realistic, variable settings.

Methodology: The ChemDIRT Framework

To address these gaps, the authors introduce ChemDIRT (Diversified Instruction, Representation, and Task Benchmark), a comprehensive evaluation framework designed to measure robustness.

1. Taxonomy of Chemical Reasoning

ChemDIRT organizes 33 task settings into eight distinct reasoning categories, defined by the minimum chemical reasoning required rather than output format:

Description: Intramolecular structural reasoning (e.g., carbon count, ring count, functional group presence).
Classification: Empirical property inference (e.g., toxicity, bioactivity).
Design: Generating molecular structures from specifications (e.g., IUPAC-to-SMILES, image-to-structure).
Analytical: Open-form language reasoning requiring causal or deductive inference (e.g., literature-grounded QA).
Regression: Quantitative property prediction (e.g., solubility, lipophilicity).
Computation: Symbolic and numerical reasoning (e.g., stoichiometry, unit conversion).
Mechanistic: Reasoning over reaction pathways (e.g., forward prediction, retrosynthesis).
Relational: Reasoning over relationships between multiple entities (e.g., representation equivalence, similarity comparison).

2. Diversified Evaluation Design

Instruction Variation: For applicable tasks, the benchmark employs multiple semantically equivalent prompt templates (generated via LLM and manually verified) rather than a single fixed instruction. This allows for the measurement of prompt sensitivity.
Representation Variation: Tasks are evaluated across multiple molecular representations, primarily comparing SMILES and IUPAC names, and including image-based inputs (molecular graphs) for vision-capable models.
Dataset Construction: The benchmark comprises 4,484 examples. To manage evaluation costs while maintaining statistical representativeness, the authors employ a K-means clustering strategy on reference model accuracy vectors to select representative subsets from larger source datasets (e.g., MoleculeNet, QM9, ZINC).

3. Experimental Setup

The framework was used to evaluate 23 open- and closed-source models (including specialized chemistry models like ChemDFM and general-purpose LLMs like Llama3, Qwen, and GPT variants) in a zero-shot setting. Evaluation metrics varied by task type, including accuracy, exact match, fingerprint similarity (MACCS, Morgan, RDK), and regression errors (Log RMSE, MAE).

Key Results

The evaluation of 23 models reveals substantial instability and specialization, challenging the notion of "general" chemical reasoning in current LLMs:

Prompt Sensitivity: Performance varies wildly across semantically equivalent prompts. For instance, on the ClinTox classification task, Llama3's accuracy ranged from 0.095 to 0.905 depending on the prompt template. Similar large variances were observed for Falcon and GPT-5.6-Luna.
Representation Dependence: Models are not representation-invariant. Some models (e.g., Kale-Chem) performed significantly better with IUPAC inputs for representation equivalence tasks, while others (e.g., Llama2) showed advantages with SMILES for chemical similarity. Success in IUPAC-to-SMILES translation did not guarantee comparable performance in the reverse direction.
Task Specialization: No single model excelled across all categories.
- Specialized Models: Models like ChemDFM-R and ChemDFM-v2 showed strength in translation, classification, and mechanistic tasks but failed on open-response QA and true/false questions.
- General Models: Strong language models often struggled with structure generation and mechanistic reasoning.
- Numerical Failure: Numerical tasks (regression and computation) exposed a critical failure mode. Several models produced errors orders of magnitude larger than peers (e.g., $10^{125}$ magnitude errors in lipophilicity prediction), indicating a lack of reliable numerical reasoning despite fluent text generation.
Image Capabilities: Vision-capable models (e.g., Claude-Sonnet-5) dominated image-to-structure tasks but remained weaker than specialized chemistry models on formula-based mechanistic tasks.

Significance and Claims

The paper claims that ChemDIRT provides a more reliable assessment of model reasoning capabilities than conventional single-format benchmarks by explicitly accounting for instruction and representation diversity.

Robustness as a Requirement: The authors argue that robustness to prompt variations and representation changes is a fundamental requirement for reliable deployment in scientific workflows, not an optional feature.
Re-evaluation of Capability: The results suggest that current models, including those specialized for chemistry, do not yet demonstrate robust general chemical reasoning under zero-shot evaluation. High performance on narrow benchmarks often masks severe weaknesses in other task families or under different formulations.
Reproducibility: By releasing the dataset, verified prompt templates, and evaluation code, ChemDIRT aims to make benchmark design choices auditable and support reproducible scientific-model comparison.

The paper concludes that future improvements in robustness may require tool augmentation, explicit molecular representations, constrained numerical execution, and training across equivalent instructions, rather than relying solely on scaling or standard fine-tuning.