🤖 machine learning

Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language

The paper introduces CheMatE, a bi-semantic embedding model built on ModernBERT that utilizes a two-stage training process involving continued masked language modeling on a massive SMILES-annotated scientific corpus and subsequent contrastive learning to jointly represent molecular structures and natural language, achieving robust performance across both chemical property prediction and general language understanding tasks.

Original authors: David Ming Segura, Jeremy Goumaz, Joshua W. Sin, Bojana Ranković, Philippe Schwaller

Published 2026-08-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: David Ming Segura, Jeremy Goumaz, Joshua W. Sin, Bojana Ranković, Philippe Schwaller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of chemistry as a massive, bustling library. Inside, there are two very different ways to describe the same thing: a molecule. On one shelf, you have the "scientist's shorthand," a compact code called SMILES that looks like a strange string of letters and numbers (like C1=CC=CC=C1). It's efficient for computers but reads like a secret cipher to humans. On the other shelf, you have the "storyteller's description," long paragraphs of natural language explaining how a molecule behaves, where it's found, or how it reacts. For a long time, computer programs trying to understand chemistry had to choose a side. They could be great at reading the secret code but terrible at understanding the story, or vice versa. They often forgot how to speak "human" just to become experts in "chemical code." This paper asks a simple but tricky question: Can we build a single brain that is fluent in both the secret code and the story at the same time, without losing its mind?

The researchers behind this study, led by David Ming Segura and Philippe Schwaller, say yes. They built a new AI model called CheMatE (Chemical Embedder with Matryoshka Embedding) that acts like a super-bilingual translator. Instead of forcing the AI to choose between being a chemist or a linguist, they taught it to see the chemical code and the scientific story as two sides of the same coin.

Here's how they did it, and why it works.

The Problem: The "Specialist" Trap

Think of previous AI models like a student who studies only for a math test. They might get a perfect score on calculus, but if you ask them to write a poem, they might forget how to use adjectives. In the world of chemistry AI, models trained heavily on SMILES strings (the code) became so good at recognizing patterns in the code that they "forgot" how to understand the natural language sentences surrounding them. They became specialists who couldn't talk to the rest of the library.

The Solution: A "Bilingual" Library

The team decided to stop treating the code and the story as separate things. Instead, they created a training method where the two are woven together.

1. The Injection Pipeline: Spicing Up the Text
Imagine you have a stack of 14.4 million scientific articles and educational texts. These are the "stories." The researchers built a robot pipeline that reads these stories, finds every mention of a chemical (like "glucose" or "aspirin"), and secretly inserts the chemical's SMILES code right next to it.

The Result: A sentence like "Glucose is a simple sugar" becomes "Glucose is a simple sugar 0=С[С@H](0)...."
The Scale: They did this for over 14 million documents, creating a massive training set with 21.9 billion "words" (tokens). This means the AI sees the code and the story happening at the exact same time, over and over again.

2. The "Smart Batching" Trick: Fitting the Puzzle
Training an AI on such a massive library is like trying to fit puzzle pieces of wildly different sizes into a box. Some documents are short; others are huge (up to 8,192 tokens long). If you just throw them in randomly, some computer processors (GPUs) would sit idle waiting for the slow ones to finish, wasting time and money.

The Fix: The team invented a "Cost-Aware Batch Sampler." Think of it as a smart librarian who doesn't just count how many books are in a pile, but calculates how heavy and awkward each book is. They arrange the piles so that every computer processor gets an equal amount of "work" to do, regardless of how long the documents are. This made the training 30% more efficient and allowed them to handle those long, complex documents without crashing.

3. The Two-Stage Training: Learning to Read, Then to Connect
The AI didn't just read the mixed-up text once; it learned in two distinct phases, like a student first learning vocabulary and then learning how to write essays.

Stage 1 (The Vocabulary): The model used a technique called "Masked Language Modeling." Imagine reading a sentence where some words are covered by black boxes. The AI had to guess the missing words. But here, the missing words could be a normal word or a chunk of SMILES code. This forced the AI to learn that the code and the text belong in the same context.
Stage 2 (The Connection): Once the AI knew the vocabulary, they taught it to understand relationships. They created a game where the AI had to match a specific chemical code with the correct paragraph of text, while ignoring paragraphs about totally different chemicals. They used a method called "Multiple Negative Ranking Loss" with a "Matryoshka" twist (named after Russian nesting dolls). This means the AI learned to understand the chemical at different levels of detail, from a big picture view down to the tiniest details, ensuring it didn't lose important information even if the view was zoomed out.

The Results: The Best of Both Worlds

When they tested CheMatE, the results were impressive. They compared it against 11 other models, including those that were experts only in code and others that were experts only in text.

The Score: CheMatE was the only model that ranked in the top group for both tasks simultaneously. It scored a "Bi-semantic Score" of 86.7%, meaning it was among the best performers on nearly 88% of the 48 different tests they ran.
The Comparison: Models that were great at reading SMILES code (like MoLFormer or ChemBERTa) were terrible at understanding natural language. Models that were great at language were okay at code, but not the best. CheMatE broke this trade-off. It proved that you don't have to sacrifice one skill to gain the other.

Why It Matters

This isn't just about getting a higher score on a test. By proving that a single model can understand both the rigid structure of chemical formulas and the fluid context of scientific writing, CheMatE opens the door for smarter tools in drug discovery and materials science. It suggests that in the future, we won't need separate tools to read a chemical diagram and a research paper; one unified brain could do it all, helping scientists find new medicines or materials faster by understanding the full story behind the science.

The authors are careful to note that while this is a huge step forward, the system isn't perfect. It relies on existing tools to find chemicals in text, which can sometimes miss tricky or brand-new compounds. However, the core idea—that mixing the code and the story creates a smarter, more versatile AI—has been successfully demonstrated.

Technical Summary: CheMatE – Bi-semantic Chemical Embedder for Joint Representation Learning

Problem Statement

Transformer models have successfully extended natural language processing (NLP) architectures to chemistry via SMILES strings. However, existing domain-adaptive models face two critical limitations:

Domain Over-specialization: Models pre-trained intensively on SMILES often suffer from catastrophic forgetting of foundational semantic capabilities, struggling to generalize to diverse scientific natural language tasks. Conversely, models adapted from general language models degrade in chemical understanding during domain-specific pre-training.
Contextual Constraints: Most molecular embedding models are optimized for short, isolated SMILES strings, lacking the architectural capacity to process long-form scientific documents where chemical entities appear within broader contexts.

Current approaches typically force a trade-off, excelling in either chemical syntax or general scientific language, but rarely both simultaneously.

Methodology

The authors introduce CheMatE (Chemical Embedder with Matryoshka Embedding), a long-context bi-semantic encoder built on the ModernBERT backbone. The model is designed to capture molecular structure and domain-specific natural language within a single representation space through a novel two-stage training pipeline.

1. Data Construction: Large-Scale SMILES Injection

To create a training corpus that intertwines structural notation with scientific context, the authors developed an automated pipeline processing 14.4 million documents (11.45B tokens from ChemPile papers and 10.41B tokens from FineWeb-Education/Web after filtering).

Filtering: Texts were filtered using a Text Chemistry Score (TCS) based on word frequency analysis to retain chemistry-relevant content.
Annotation Pipeline: A four-stage process annotates chemical entities:
1. NER: Detection of chemical entities using Chemical Data Extractor 2 (CDE2).
2. Conversion: Deterministic IUPAC-to-SMILES conversion via OPSIN.
3. Lookup: Fallback to PubChem API for unresolved names.
4. Validation: Canonicalization and valence checking via RDKit.
Injection: Valid SMILES strings are inserted immediately after their corresponding chemical entity mentions in the text, creating a mixed-modality corpus where SMILES tokens appear naturally within long-context scientific narratives.

2. Training Strategy

The training proceeds in two sequential stages:

Stage I: Continued Masked Language Modeling (MLM)

Objective: The model undergoes continued pre-training on the SMILES-injected corpus using a standard MLM objective (15% masking rate).
Efficiency Innovation: To handle the extreme variability in sequence lengths (up to 8,192 tokens) across a multi-GPU setup, the authors developed the BalancedTokenBatchSampler. Unlike standard padding, which causes load imbalance, this sampler constructs batches based on three constraints: a raw-token budget, a padded-token memory cap, and a quadratic cost ceiling ( $\sum L_i^2$ ). This ensures balanced computational loads across GPUs, reducing padding overhead by up to 50% and eliminating out-of-memory errors.

Stage II: Contrastive Learning with Matryoshka Embeddings

Objective: To refine bi-semantic understanding, the model is fine-tuned using Multiple Negative Ranking Loss (MNRL) with a Matryoshka wrapper.
Synthetic Pair Generation: An algorithmic pipeline generates anchor-positive-negative triples directly from the annotated corpus. An anchor SMILES is selected, and text segments are scored based on Tanimoto similarity of Morgan fingerprints against the anchor. Segments with high similarity form the positive set; low similarity forms the negative set.
Matryoshka Loss: The model is optimized to produce informative embeddings even when truncated to sub-dimensions (768, 512, 256, 128, 64), enhancing flexibility for downstream applications.

Key Contributions

Large-Scale SMILES Injection Pipeline: A multi-stage automated system that integrates chemical entities into scientific text, producing a 21.9 billion token corpus optimized for bi-semantic learning.
Cost-Budget Distributed Batch Sampler: A novel sampler (BalancedTokenBatchSampler) that enables efficient training on heterogeneous, long-context data across multi-node GPU clusters by balancing quadratic computational costs rather than just sample counts.
Bi-semantic Sequential Training: A pipeline that treats SMILES not as a separate modality but as integrated tokens within natural language, followed by contrastive refinement. This allows a single encoder to learn robust representations for both structural chemistry and scientific NLP.
Unified Benchmarking: Evaluation across 48 diverse datasets (26 classification, 22 regression) covering molecular property prediction, materials science, and scientific language understanding.

Results

CheMatE was evaluated against 11 baselines, including general-purpose models (e.g., SciBERT, ModernBERT) and chemistry-specialized encoders (e.g., ChemBERTa, MoLFormer, BERT-SMILES).

Dual-Domain Performance: CheMatE achieved the lowest mean rank on both SMILES tasks (1.9) and NLP tasks (3.2). It is the only chemistry-specialized model to rank in the top Nemenyi equivalence group for both modalities simultaneously.
Ablation Study: Removing the contrastive training stage (CheMatE-MLM) resulted in a significant performance drop, with the bi-semantic score falling from 86.7% to 55.5%. This confirms that the sequential contrastive stage is critical for refining cross-modal performance.
Comparison to Baselines:
- Chemistry-specialized models (e.g., MoLFormer, ChemBERTa) performed well on SMILES but ranked at the bottom of the NLP leaderboard.
- General language models performed reasonably on NLP but struggled significantly on SMILES tasks.
- CheMatE successfully avoided this modality trade-off, outperforming Morgan fingerprints on SMILES tasks and matching dedicated scientific language models on NLP tasks.

Significance and Claims

The paper claims that CheMatE demonstrates that strong SMILES understanding and scientific natural language comprehension are not mutually exclusive in encoder-only models. By unifying structural and contextual signals within a single text-based framework, the model achieves competitive performance across specialized chemistry tasks and general-purpose language benchmarks without the typical specialization trade-off.

The authors position CheMatE as a versatile foundation for:

Molecule and Natural Language Embedding: Providing high-quality representations for both modalities.
Retrieval Applications: Serving as a backbone for retrieval in mixed-semantic environments.
Future Generative Models: Potentially acting as a foundation for Retrieval-Augmented Generation (RAG) pipelines where molecular structures are grounded in rich scientific contexts.

The work also highlights the value of the SMILES injection pipeline and the BalancedTokenBatchSampler as reusable tools for training large-scale, heterogeneous scientific corpora. The authors acknowledge limitations regarding the dependency of the annotation pipeline on existing NER and database tools (which may fail on novel compounds) and the modest size of the contrastive dataset, but assert that the approach validates the route of merging molecular notation directly into scientific text for general-purpose chemistry representation learning.