🤖 machine learning

Rethinking Molecular Text Representations for LLMs: An Empirical Study

This paper presents a systematic benchmark of 16 LLMs across nine molecular representations and eight chemical tasks, revealing that performance is highly representation-dependent with no single format dominating all tasks, thereby arguing against representation-invariant evaluation and advocating for task-aware representation routing.

Original authors: Arun Raja, Garrett M. Morris, Kian Ming A. Chai

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arun Raja, Garrett M. Morris, Kian Ming A. Chai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, well-read robot (a Large Language Model or LLM) how to understand chemistry. The robot is brilliant at reading stories and writing essays, but chemistry is a bit like a secret code. To talk to the robot about a molecule (like aspirin or caffeine), you have to translate that molecule into text.

The problem is, there are many different "dialects" or ways to write this chemical code. Some look like short, cryptic strings of letters (SMILES), some look like long, formal names (IUPAC), and some look like detailed blueprints written in a structured format (CML or MolJSON).

This paper is like a massive "taste test" where the researchers tried feeding the robot the same molecule written in nine different dialects and asked it to perform eight different chemistry tasks. They wanted to find out: Which dialect does the robot understand best?

Here is what they discovered, broken down with simple analogies:

1. The "One Size Fits All" Myth is False

You might think that if a robot is good at chemistry, it should be good at chemistry no matter how you write the molecule down. The paper says this is wrong.

The Analogy: Imagine you are asking a chef to chop vegetables. If you give them a knife made of wood, a knife made of glass, or a knife made of steel, they will chop differently. The chef's skill isn't just about being a chef; it's about how well they can handle the specific tool you handed them.
The Finding: The robot's performance changed drastically depending on which "dialect" you used. There was no single "best" way to write a molecule that worked for every task.

2. The "Blueprint" vs. The "Recipe"

The researchers found that different dialects are better for different jobs.

The "Blueprints" (CML and MolJSON): These are like detailed architectural drawings where every atom and bond is explicitly listed in a structured list.
- Best for: Tasks that require counting parts or seeing the exact shape, like "How many carbon atoms are here?" or "Is this molecule a mirror image of that one?"
- Why: The robot can easily scan the list like a spreadsheet.
The "Formal Names" (IUPAC): These are long, human-readable names like "2-acetyloxybenzoic acid."
- Best for: Tasks that involve matching descriptions, like "Find the molecule that matches this story."
- Why: Since these names are written in natural language, the robot (which is trained on human text) understands the meaning of the words better than the cryptic codes.
The "Short Codes" (SMILES): This is the most common way chemists write molecules today. It's a compact string of characters.
- The Surprise: Even though the robot was likely trained mostly on these short codes, they were rarely the best choice. In fact, for many tasks, the robot struggled with them compared to the detailed blueprints.

3. The "Specialist" Trap

The researchers also tested robots that were specifically trained only on chemistry (Chemistry-specialized models).

The Analogy: Imagine a chef who only ever practiced chopping carrots. If you give them a carrot, they are amazing. But if you hand them a potato or a tomato, they might freeze up or break the knife because they never practiced on those.
The Finding: These chemistry-specialized robots were great at the short codes (SMILES) because that's what they were trained on. However, when you gave them the "blueprints" (CML/MolJSON), they performed terribly. They had become so specialized in one dialect that they forgot how to read the others. The general-purpose robots were actually more flexible.

4. The "Hallucination" Problem

When the robot tried to create a new molecule based on a description, it often made mistakes.

The Finding: Using the "Formal Names" (IUPAC) resulted in the fewest errors. When the robot tried to generate molecules using the short codes or the blueprints, it was more likely to "hallucinate" (make up fake molecules or break the rules of chemistry).
The Mechanism: The researchers looked inside the robot's "brain" (its attention mechanisms) and found that when reading the detailed blueprints, the robot had to pay attention to every single part of the molecule to get it right. When reading the short codes, it often skipped over important details.

5. The Main Takeaway

The paper concludes that we cannot just say "LLMs are good at chemistry." We have to ask, "Good at chemistry using which text format?"

If you want the robot to count atoms or spot structural differences, give it the detailed blueprints (CML/MolJSON).
If you want the robot to find a molecule based on a description, give it the formal name (IUPAC).
If you want the robot to generate a new molecule, the formal name is currently the safest bet to avoid errors.

In short: The way you ask the question matters just as much as the intelligence of the person (or robot) answering it. To get the best results from AI in chemistry, we need to stop using just one "default" language and start choosing the right language for the specific job we need done.

Technical Summary: Rethinking Molecular Text Representations for LLMs

Problem Statement

The application of Large Language Models (LLMs) to chemistry is growing, yet the field lacks a systematic understanding of which molecular text representation best supports different tasks. While Simplified Molecular Input Line Entry System (SMILES) has become the de facto standard due to its prevalence in pretraining corpora and literature, it is unclear if this representation is optimal for all LLM-based chemical reasoning tasks. Prior work often focuses on training models from scratch on molecular strings or fine-tuning for specific tasks, predominantly using SMILES. However, chemistry offers diverse machine-readable formats (e.g., IUPAC names, InChI, SELFIES, structured XML/JSON) that differ in compactness, invertibility, and semantic structure. The central question addressed is whether LLM performance is representation-invariant or if specific tasks require specific representations to achieve optimal competence.

Methodology

The authors introduce MolRepBench, a systematic benchmark designed to evaluate LLM molecular competence across nine distinct representations and eight chemical tasks.

Representations Evaluated:
The study covers nine formats, ranging from compact strings to explicit graph structures:

SMILES Variants: Canonical, Isomeric, and Randomized SMILES.
Generative-Optimized Strings: DeepSMILES and SELFIES.
Nomenclature & Identifiers: IUPAC names and International Chemical Identifier (InChI).
Structured Text: Chemical Markup Language (CML) and MolJSON (a recent JSON-based graph representation).

Models Evaluated:
The benchmark tests 16 LLMs across five families:

Open-weight general models: Qwen3 (4B/30B), Phi-4 (14B), and OLMo-3 (32B), including both reasoning and non-reasoning variants.
Domain-specialized models: ChemDFM (v2.0 and R variants) and Ether0, alongside their respective base models (Qwen2.5-14B and Mistral-Small-24B).
Closed frontier models: GPT-5.4-mini and Claude-Haiku-4.5.

Tasks:
Eight tasks span comprehension, discrimination, and generation:

Atom Counting: Parsing strings to count specific elements.
Functional Group Identification: Recognizing substructures (e.g., esters, amines).
Molecular Property Estimation: Predicting computable properties (LogP, TPSA, H-bond donors/acceptors) derived from RDKit.
Molecule Retrieval: Matching natural language descriptions to molecular structures.
Isomer Discrimination: Distinguishing molecules with subtle structural or stereochemical differences.
Tautomer Recognition: Identifying dynamic structural equivalences.
Protonation State Recognition: Identifying charge-state variants of the same molecule.
Caption-to-Molecule Generation: Translating natural language descriptions into valid molecular strings.

Evaluation Metrics:
The study employs exact-match accuracy, macro-F1, Spearman correlation, and Tanimoto similarity for structural comparisons. Crucially, for generation tasks, the authors utilize an LLM-as-a-judge (Gemini 3 Flash) to qualitatively assess error modes (e.g., hallucinations, syntax errors, wrong scaffolds) and score generations on faithfulness, chemical reasoning, and overall quality.

Mechanistic Analysis:
To explain performance differences, the authors conduct:

Tokenization Audits: Measuring token counts and reasoning token lengths.
Linear Probing: Testing hidden states for linearly accessible chemical information.
Attention Analysis: Examining attention weights between the final token and molecule tokens, and within-molecule self-attention.

Key Results

1. Performance is Representation-Dependent
No single representation is optimal across all tasks. The "best" representation varies significantly by task type:

Structured Representations (CML, MolJSON): Dominate tasks requiring explicit graph understanding, such as Atom Counting, Isomer Discrimination, and Tautomer Recognition. CML achieved the highest aggregate score (81 wins across 128 configurations), followed by MolJSON (75 wins).
IUPAC Names: Excel in semantic tasks, specifically Molecule Retrieval (winning for all 16 LLMs) and Caption-to-Molecule Generation. Their morphemic structure aligns well with natural language descriptions.
SMILES Variants: Despite their dominance in pretraining, SMILES variants are rarely optimal. They perform well in property regression (LogP, TPSA) but often underperform in structural disambiguation and generation compared to structured formats.
SELFIES: Consistently performed the worst across all 16 LLMs, particularly in atom counting and generation.

2. Specialization vs. Generalization
Chemistry-specialized models (e.g., ChemDFM) show a trade-off. While they significantly outperform their base models on SMILES-based tasks (e.g., ChemDFM-R-14B achieved high exact-match scores in generation with SMILES), they suffer catastrophic forgetting or severe degradation when presented with structured text representations (CML, MolJSON, InChI). For instance, ChemDFM-R-14B produced near-zero valid generations with CML and InChI, whereas its base model (Qwen2.5-14B) maintained robust performance. This suggests that SMILES-only fine-tuning rewards specialization that does not generalize to other formats.

3. Generation Quality and Error Modes

Validity $\neq$ Correctness: High validity rates (e.g., 97.6% for ChemDFM-R-14B with SMILES) do not guarantee structural correctness (Tanimoto similarity was only 0.68).
LLM-as-a-Judge: IUPAC names yielded the highest fraction of correct generations (50%) when judged by Gemini 3 Flash. Canonical SMILES had high representation faithfulness but low overall quality due to scaffold and reasoning errors.
Reasoning Models: Reasoning-enabled variants (e.g., Phi-4-Reasoning, OLMo-Think) generally reduced hallucination rates compared to their non-reasoning counterparts, though they did not uniformly improve performance across all tasks.

4. Mechanistic Insights

Tokenization: Structured representations (CML, MolJSON) are significantly longer (higher token counts) than SMILES. However, they result in shorter reasoning token lengths, suggesting the explicit structure reduces the cognitive load for the model.
Attention: Structured representations require the model to attend more heavily across the entire molecular span (higher "last-token to molecule" attention), whereas compact SMILES variants encourage richer token-to-token interaction within the string.
Internal Encoding: Linear probing revealed that while SMILES variants are well-encoded (likely due to pretraining), MolJSON (a post-training representation) often yielded better linear probe performance, indicating the model can leverage the explicit graph structure effectively even without prior exposure.

Significance and Claims

The paper argues against the notion of representation-invariant evaluation in chemistry. The authors claim that:

Representation Choice is Critical: The molecular text format is a key factor in LLM chemical competence; using a single representation (like SMILES) provides a biased and incomplete view of a model's capabilities.
Task-Aware Routing is Necessary: Different tasks benefit from different representations. The authors propose task-aware representation routing, where the input format is selected based on the specific chemical task (e.g., using IUPAC for retrieval, CML/MolJSON for structural parsing).
Evaluation Standards Must Evolve: Benchmarks should report performance across multiple representations to surface trade-offs, particularly the loss of generalization in domain-specialized models.
Qualitative Judgment: Rigid metrics (validity rate, exact match) are insufficient for generation tasks; LLM-as-a-judge is essential for analyzing error modes like hallucinations and scaffold errors.

The study concludes that current LLMs do not yet possess robust, representation-invariant molecular capabilities, and that the field must move beyond SMILES-centric evaluation to develop more robust, generalizable chemistry AI systems.