🤖 AI

Benchmark for Assessing Olfactory Perception of Large Language Models

This paper introduces the Olfactory Perception (OP) benchmark, a dataset of 1,010 questions designed to evaluate large language models' ability to reason about smell, revealing that current models rely primarily on lexical associations rather than structural molecular reasoning and achieve a maximum accuracy of 64.4% while showing improved performance through multilingual aggregation.

Original authors: Eftychia Makri, Nikolaos Nakis, Laura Sisson, Gigi Minsky, Leandros Tassiulas, Vahid Satarifard, Nicholas A. Christakis

Published 2026-04-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eftychia Makri, Nikolaos Nakis, Laura Sisson, Gigi Minsky, Leandros Tassiulas, Vahid Satarifard, Nicholas A. Christakis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that has read almost every book, website, and article ever written. It can write poetry, solve complex math problems, and even write computer code. But there's one thing it's never really "experienced": smell.

You can't put a smell into a book. You can't describe the scent of a fresh strawberry or a wet dog with perfect accuracy using just words. This paper introduces a new test, called the Olfactory Perception (OP) Benchmark, to see if these super-smart robots (Large Language Models, or LLMs) can actually "guess" what things smell like, or if they are just guessing based on the words they've seen before.

Here is the breakdown of what they did and what they found, using some simple analogies.

1. The Test: A "Smell Quiz" for Robots

The researchers created a massive quiz with 1,010 questions. Think of it like a final exam for a smell school. The questions cover different levels of difficulty:

The Easy Stuff: "Does this molecule smell, or is it odorless?" (Like asking, "Is this rock wet?")
The Middle Stuff: "Is this smell more like a rose or a lemon?" or "Is this smell strong or weak?"
The Hard Stuff: "Here is a mix of 10 chemicals. Does it smell like a strawberry or a fish?" or "Which specific biological sensors in a human nose would this molecule trigger?"

2. The Twist: The "Name" vs. The "Blueprint"

This is the most clever part of the study. The researchers asked the robots the same questions in two different ways:

Method A (The Name): They gave the robot the common name, like "Vanilla."
Method B (The Blueprint): They gave the robot the chemical formula (a string of letters and numbers called SMILES), which is like the molecule's DNA or architectural blueprint.

The Analogy: Imagine you are trying to guess what a cake tastes like.

Method A is telling you, "This is a Chocolate Cake."
Method B is handing you a list of ingredients and chemical structures without saying what the cake is.

3. The Results: Cheating vs. Understanding

The results were surprising and told a clear story:

The Robots are "Word Cheaters": When the robots were given the names (e.g., "Vanilla"), they did pretty well. They knew that "Vanilla" usually smells sweet.
The Robots are "Blind to Blueprints": When the robots were given the chemical blueprints (the SMILES strings), their performance dropped significantly.

What this means: The robots aren't actually "understanding" how molecules work. They aren't looking at the shape of the molecule and thinking, "Ah, this shape fits into a nose sensor that smells like mint." Instead, they are just memorizing that the word "Mint" is often associated with certain smells. They are relying on lexical associations (word connections) rather than structural reasoning (understanding the physical object).

It's like a student who memorized the answer key for a test but doesn't actually understand the math. If you change the numbers but keep the question the same, they fail.

4. The "Mixing Bowl" Problem

The hardest part of the test was Mixture Similarity.

The Task: "Here is a bowl of chemicals A, B, and C. Here is a bowl of chemicals X, Y, and Z. Do they smell the same?"
The Reality: In the real world, two completely different bowls of chemicals can smell identical (like two different recipes for the same soup).
The Robot Failure: The robots failed miserably here. They tried to count the ingredients. If the bowls didn't have the exact same ingredients, the robots said, "These smell totally different!" They couldn't grasp the concept that the whole can smell different from the sum of its parts.

5. The Language Barrier

The researchers also tested the robots in 21 different languages.

The Finding: Smell is tricky in some languages. Some languages have very specific words for smells, while others (like English) are a bit vague.
The Good News: When they combined the answers from all 21 languages, the robots got smarter! It's like having a team of experts where one speaks French, one speaks Japanese, and one speaks German. By pooling their knowledge, they could guess the smell better than any single language could.

6. The Safety Glitch

One funny (but serious) side note: Some of the smartest robots refused to answer questions about dangerous chemicals (like nerve agents) because their safety filters kicked in. They wouldn't say, "Yes, this smells bad," because they were programmed to be helpful and harmless. This shows that even in a science test, the robot's "conscience" can get in the way of the answer.

The Big Takeaway

Currently, these AI models are great at remembering words but terrible at understanding the physical world of smell.

They know that "Coffee" smells like coffee because they've read the word "coffee" a million times.
They don't know why a specific chemical structure smells like coffee.

The Future:
The authors say this is just the beginning. To make AI truly "smell," we need to teach them to look at the molecular blueprints, not just the names. We need them to move from being word-memorizers to molecular-understanders. Until then, if you ask an AI what a new, weird chemical smells like, it's just guessing based on the words it's seen, not the science it understands.

1. Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable capabilities in visual, auditory, and even chemical reasoning (e.g., predicting reaction products or interpreting SMILES strings), their ability to reason about olfaction (the sense of smell) remains largely unexplored.

The Gap: Unlike vision (wavelengths $\to$ colors) or audition (frequencies $\to$ pitch), olfaction relies on a complex, non-linear relationship between molecular structure and receptor biology that is not fully understood.
The Question: Do current LLMs possess factual knowledge about how molecules smell, or can they bridge the gap between molecular structure and sensory perception?
Limitations of Prior Work: Previous studies focused on subjective description-to-scent mapping (low success rates) or semantic similarity of odor words. None systematically evaluated LLMs on discrete factual knowledge regarding odor classification, intensity, pleasantness, and biological mechanisms using objective ground-truth data.

2. Methodology: The Olfactory Perception (OP) Benchmark

The authors introduce the OP Benchmark, a comprehensive evaluation framework comprising 1,010 questions across eight task categories.

A. Task Categories

The benchmark spans the full complexity of olfactory perception:

Odor Classification (OC): Binary classification (Odorous vs. Odorless).
Odor Primary Descriptor (OPD): Selecting the single primary odor category from a list.
Odor Intensity (OIn) & Pleasantness (OPl): Pairwise comparison of molecules to determine which is more intense or pleasant.
Rate-All-That-Apply (RATA): Multi-label semantic profiling (selecting all applicable descriptors from a 138-word lexicon).
Odor Similarity of Mixtures (OS): Predicting the perceptual similarity between two complex mixtures (2–10 molecules each).
Olfactory Receptor Activation (ORA): Identifying which specific human olfactory receptors (ORs) are activated by a molecule.
Smell Identification Test (SIT): Identifying a real-world food/object source based on a mixture of volatile organic compounds.

B. Experimental Design

Dual-Prompting Strategy: To distinguish between lexical association (knowing the name) and structural reasoning (understanding the molecule), every question is presented in two formats:
1. Compound Name: e.g., "Benzyl Alcohol."
2. Isomeric SMILES: The specific molecular string representation (e.g., C1=CC=C(C=C1)CO).
- Significance: This tests if models rely on memorized name-descriptor pairs or can infer properties from chemical structure.
Model Evaluation: 21 model configurations were tested across 6 major providers (OpenAI, Google, Anthropic, xAI, DeepSeek, Meta), including both reasoning-enabled (e.g., o3, Opus 4.6) and non-reasoning models.
Multilingual Evaluation: A subset of tasks (RATA) was translated into 21 languages to test if olfactory knowledge is language-specific or robust across linguistic boundaries.

3. Key Results

A. Overall Performance

Best Performance: The top-performing model, Claude Opus 4.6 (max), achieved an overall accuracy of 64.4% using compound name prompts.
Performance Hierarchy: Frontier models (Anthropic, OpenAI) significantly outperformed open-source models (Llama 3.3 70B scored 52.7%).
Task Difficulty:
- Simple Tasks: Odor Classification and Primary Descriptor identification showed high accuracy (up to 92%).
- Hard Tasks: Odor Similarity of Mixtures and Receptor Activation remained challenging, with accuracies often near or below chance levels for complex sub-tasks.

B. The "Name vs. Structure" Gap (Critical Finding)

Lexical Dominance: Compound name prompts consistently outperformed Isomeric SMILES prompts across all 21 models.
Magnitude: The performance gain ranged from +2.4 to +18.9 percentage points (mean $\approx$ +7 points).
Implication: Current LLMs access olfactory knowledge primarily through lexical associations (e.g., "Vanilla" $\to$ "Sweet") rather than structural molecular reasoning. Models struggle to deduce smell from raw chemical strings, especially for open-source models where the gap was largest (Llama 3.3: 33.8% with SMILES vs. 52.7% with names).

C. Reasoning Budgets

Increasing the reasoning token budget (e.g., from 8K to 32K tokens) yielded only modest gains (typically <2 percentage points).
This contrasts with chemistry benchmarks where extended reasoning significantly improves performance, suggesting that olfactory knowledge in LLMs is limited by data scarcity rather than reasoning depth.

D. Multilingual Findings

Language Diversity: English performed best, followed by Romance and Germanic languages. Non-Indo-European languages (e.g., Korean, Chinese) showed lower individual performance.
Ensemble Gain: Aggregating predictions across all 21 languages significantly improved performance, achieving an AUROC of 0.86 for the best language ensemble. This suggests that different languages provide complementary descriptor inventories for specific odors.

E. Systematic Failure Modes

Mixture Similarity: Models failed to predict perceptual similarity when mixtures shared few molecules. They relied on a "molecular overlap heuristic" (assuming low overlap = dissimilar), whereas human perception often finds similar smells in chemically distinct mixtures.
Safety Filters: Safety alignment mechanisms in some models (e.g., Claude Opus 4.6) refused to answer questions about hazardous compounds (e.g., nerve agents), artificially lowering benchmark scores for legitimate toxicological queries.
Receptor Knowledge: Models exhibited specific factual gaps regarding receptor mutations (e.g., D296N), often failing to self-correct even when reasoning traces were generated.

4. Key Contributions

First Comprehensive Olfactory Benchmark: Introduction of the OP benchmark with 1,010 questions grounded in peer-reviewed olfactory science, covering classification, perception, and biology.
Dual-Representation Analysis: Establishment of the "Name vs. SMILES" gap, providing evidence that current LLMs rely on lexical priors rather than structural reasoning for sensory tasks.
Multilingual Insight: Demonstration that olfactory knowledge is not monolithic; aggregating multilingual predictions significantly enhances accuracy, highlighting the role of language in encoding sensory concepts.
Baseline Establishment: Comprehensive performance baselines for 21 model configurations, identifying that while frontier models show emerging capabilities, substantial gaps remain in genuine olfactory reasoning.

5. Significance and Future Directions

Scientific Impact: This work positions olfaction as a critical, under-explored modality for evaluating AI. It reveals that LLMs have not yet learned the "structure-odor" relationship, unlike their mastery of visual or auditory patterns.
Practical Applications: Improving olfactory reasoning could accelerate fragrance design, flavor development, and the detection of hazardous contaminants in consumer products.
Future Work: The authors suggest future research should focus on:
- Hybrid systems combining LLMs with cheminformatics tools.
- Incorporating multimodal inputs (e.g., images of facial expressions during smelling).
- Developing training methods that explicitly encourage structure-based reasoning over lexical memorization.
- Expanding datasets to include concentration effects and temporal dynamics.

In conclusion, the paper establishes that while LLMs possess a "dictionary" of smells, they lack the "physics" of olfaction. Bridging this gap requires moving beyond lexical associations to true structural understanding.