💻 computer science

See & Sniff: Learning Visuo-Olfactory Representations

This paper introduces SmellNet-V, a scalable visuo-olfactory dataset created by synthetically pairing odor samples with semantically aligned images, and proposes the "See & Sniff" self-supervised framework to learn joint representations that enable odor classification, spatial localization, and cross-modal retrieval.

Original authors: Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a superpower: you can look at a picture of an apple and instantly know exactly what it smells like. Or, conversely, if someone hands you a vial of "apple scent," you can point to exactly where the apple is in a crowded photo.

For a long time, computers were great at seeing and great at hearing, but they were completely "nose-blind." They couldn't connect a smell to a picture because no one had ever taught them how. This paper, titled "See & Sniff," is like teaching a computer its first lesson in smelling.

Here is the simple breakdown of what the researchers did, using some everyday analogies.

1. The Problem: The "Missing Link"

Think of the computer's brain as a student. It has a library of books on how to see (images) and a library of books on how to hear (audio). But its library on smell is empty.

Usually, to teach a computer to link a smell to a picture, you'd need a robot to go out, sniff a strawberry, take a photo of that exact strawberry at that exact moment, and save them together. Doing this for thousands of items is incredibly expensive and slow. It's like trying to build a dictionary by hiring a translator to sit in a room and translate every single word as it's spoken, one by one.

2. The Clever Shortcut: The "Magic Pairing"

The researchers realized something smart about how smells work: A smell doesn't change just because the object looks different.

The Analogy: Imagine a red apple and a green apple. They look different, but they both smell like "apple." A small apple and a giant apple smell the same. A fresh apple and a slightly bruised apple smell mostly the same.
The Solution: Instead of taking new photos and sniffing them, the team took an existing database of only smells (called SmellNet) and "married" them to random, real-world photos of those same ingredients found on the internet.
They created a new dataset called SmellNet-V. It's like taking a playlist of songs (the smells) and randomly pairing them with music videos (the images) that feature the same artist. Even if the video isn't the exact one recorded with the song, the "vibe" (the semantic category) matches.

3. The Brain: "See & Sniff"

Once they had this new dataset, they built a computer model named See & Sniff. Think of this model as a detective with two magnifying glasses: one for eyes and one for a nose.

The Training: The model looks at a picture of a cinnamon stick and smells a "cinnamon" signal. It tries to find the connection.
The Secret Sauce (Dense Local Alignment): Most AI models just look at the "whole picture" and say, "This looks like cinnamon." But See & Sniff is more like a detective looking for clues. It scans the image pixel-by-pixel to find exactly where the cinnamon is. It learns that the smell signal matches the specific texture and color of the cinnamon stick, not the wooden table it's sitting on.

4. What Can It Do? (The Three Tricks)

The paper shows that this model learned three cool tricks:

Trick 1: The Smell Detective (Smell Localization)
If you give the computer a "smell" (like "pineapple"), it can look at a messy kitchen photo and draw a box around the pineapple. It ignores the knife, the cutting board, or the other fruits. It knows exactly where the smell is coming from.
- Analogy: It's like closing your eyes, smelling a pizza, and being able to point your finger at the exact slice on the table without looking.
Trick 2: The Translator (Cross-Modal Retrieval)
If you show it a picture of a mango, it can find the "mango smell" in a database. If you give it a "mango smell," it can find the picture of a mango. It's learning that the visual shape and the chemical scent are two sides of the same coin.
Trick 3: The Better Smeller (Smell Classification)
Here is the surprising part: By learning to look at pictures while smelling, the computer actually got better at smelling than if it had only smelled.
- Analogy: It's like learning to identify a song by listening to it while watching the music video. Even if you only listen to the song later, your brain remembers the visual cues, helping you identify it faster and more accurately. The paper claims this improved their accuracy by 7% compared to models that only used smell.

5. Why This Matters (According to the Paper)

The researchers didn't just build a cool toy; they built the first "gym" for smell AI.

They created the first dataset (SmellNet-V) that links smells to images.
They created the first test for finding smells in pictures (Smell Localization).
They proved that you don't need expensive, synchronized robots to teach a computer to smell; you can use "smart pairing" of existing data.

In a nutshell: This paper is about teaching a computer to connect its eyes to its nose. They did it by cleverly matching existing smell data with internet photos, creating a model that can not only identify smells but also point to exactly where they are in a picture, all while getting smarter at smelling just by looking.

Technical Summary: See & Sniff: Learning Visuo-Olfactory Representations

Problem Statement

Despite the integration of vision with language, audio, and touch in modern multimodal AI, olfaction remains a largely unexplored modality. This gap stems primarily from the scarcity of paired visuo-olfactory data; unlike cameras or microphones, portable olfactory sensors have historically been confined to laboratories, and large-scale public datasets lack corresponding visual information. While recent efforts like SmellNet have released large-scale smell datasets, they remain unimodal (olfactory-only), preventing the learning of joint representations between visual appearance and odor identity. The core challenge addressed by this work is how to enable self-supervised learning of visuo-olfactory representations without the availability of naturally paired, co-collected data.

Methodology

1. Dataset Construction: SmellNet-V

The authors introduce SmellNet-V, a scalable visuo-olfactory dataset constructed by synthetically pairing existing smell-only data with in-the-wild web images. This approach relies on the key insight that odor identity is largely invariant to visual transformations (e.g., lighting, scale, or minor color variations) within a semantic category. For instance, a small red apple and a large red apple emit nearly identical volatile organic compounds (VOCs) and belong to the same odor category.

The construction pipeline involves:

Sniffing Units: Extending the SmellNet dataset by segmenting long olfactory recordings into fixed-length temporal windows (simulating biological "sniffs") to increase data granularity.
Image Collection: Using a Large Language Model (LLM) to generate context-rich search queries for each ingredient (e.g., "whole apple on wooden table," "apples in grocery produce section") to retrieve diverse web images.
Filtering: A three-stage pipeline using CLIP for prompt-based verification (ensuring category consistency, photorealism, and valid object state), watermark detection, and human refinement.
Pairing: Randomly matching each sniff unit with an image from the same ingredient category to form the training pairs.

2. Framework: See & Sniff

See & Sniff is a self-supervised framework designed to learn joint visuo-olfactory representations via dense local alignment.

Architecture: The model employs a two-stream transformer architecture.
- Visual Encoder: Uses a frozen DINOv3-Small backbone with a lightweight aligner (LayerNorm + 1x1 convolution).
- Olfactory Encoder: Follows the ScentFormer architecture (4-layer transformer) trained from scratch, followed by a two-layer MLP aligner.
Training Objective: The framework utilizes contrastive learning. Unlike standard global alignment methods that pool features, See & Sniff computes a similarity map between the spatial visual feature map and the aggregated sniff feature.
- The sniff feature is average-pooled over time.
- A similarity map $M \in \mathbb{R}^{H \times W}$ is generated by computing the inner product between the aggregated sniff vector and every spatial location in the visual feature map.
- The final similarity score is derived via max-pooling of this map.
- The loss function attracts positive pairs (matching smell and image) while repelling negative pairs (mismatched images) within the batch.

3. New Tasks and Benchmarks

The paper introduces and evaluates three downstream tasks:

Smell Classification: Predicting ingredient categories from olfactory inputs alone.
Cross-Modal Retrieval: Bidirectional retrieval between smell and vision (e.g., retrieving an image given a smell, or vice versa).
Smell Localization: A novel task where the model must identify the spatial location of a smell source within a visual scene. This is evaluated using a newly constructed benchmark, SmellNet-V-Source, which includes pixel-level segmentation masks for ingredients.

Key Contributions

SmellNet-V Dataset: The construction of the first large-scale visuo-olfactory dataset by leveraging odor invariance to synthetically pair smell-only data with diverse web images, bypassing the need for costly co-collection.
See & Sniff Framework: A self-supervised model that learns joint representations through dense local alignment, naturally producing smell saliency maps for spatial grounding.
Smell Localization Task: The introduction of smell localization as a new downstream task, accompanied by the first pixel-level smell localization dataset and benchmark.
Performance Gains: Demonstration that visuo-olfactory training enhances intrinsic olfactory representation learning, surpassing smell-only baselines.

Experimental Results

Smell Classification

Evaluated on the SmellNet-Test set (smell-only inference), the See & Sniff model achieved a 7% improvement in accuracy over the smell-only ScentFormer baseline (57.71% vs. 50.6%). This gain was consistent across all ingredient families (fruits, herbs, nuts, spices, vegetables), with notable improvements in categories with distinct visual structures (e.g., fruits, nuts).

Cross-Modal Retrieval

On the SmellNet-V-Test set, See & Sniff achieved the best performance in 4 out of 6 metrics (Recall@1, @5, @10 for both directions). Notably, it outperformed global alignment baselines in the more challenging Recall@1 setting, indicating superior fine-grained matching capabilities.

Smell Localization

The model significantly outperformed all baselines on the SmellNet-V-Source benchmark:

mAP: 0.8362 (See & Sniff) vs. 0.6676 (Global alignment) and 0.1736 (Global-CLIP).
mIoU: 0.6456 (See & Sniff) vs. 0.5016 (Global alignment).
Key Finding: Dense local alignment was essential for spatial grounding; global alignment methods failed to model spatial cross-modal interactions. The unified end-to-end model also outperformed cascaded pipelines (SmellNet + SAM3), demonstrating the necessity of joint feature interaction.
Interactive Localization: In scenarios with multiple ingredients in a single image, See & Sniff successfully shifted its predicted region based on the specific smell query, whereas global methods and visual-only baselines failed.

Significance and Claims

The paper positions See & Sniff as a foundational step toward integrating olfaction into multimodal perception systems. The authors claim that:

Visual supervision enhances olfactory learning: Contrary to the assumption that vision is merely auxiliary, visual data significantly strengthens the intrinsic representation of smell, improving classification performance even when vision is not present at inference.
Synthetic pairing is viable: The strategy of pairing smell-only data with in-the-wild images based on semantic invariance is an effective, scalable alternative to costly real-world data collection.
Spatial grounding is possible: By learning dense local alignments, models can move beyond simple classification to spatially localize odor sources, a capability previously unexplored in machine olfaction.

The work establishes new benchmarks for classification, retrieval, and spatial grounding, moving machine olfaction beyond unimodal modeling. The authors modestly note that future work could extend these capabilities to degraded states (spoilage) and complex mixture odors, but the current contribution focuses on establishing the feasibility of visuo-olfactory learning in normal, non-degraded ingredient states.