🤖 machine learning

What Images Cannot Say: Language-Guided Olfactory Representation Learning

This paper introduces SCENT, a multimodal framework that leverages language-guided semantic descriptors from Vision-Language Models to bridge the gap between visual scenes and electronic-nose signals, achieving state-of-the-art crossmodal retrieval and interpretable olfactory representation learning on the New York Smells dataset.

Original authors: Eleftherios Tsonis, Xi Wang, Vicky Kalogeiton

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eleftherios Tsonis, Xi Wang, Vicky Kalogeiton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blind" Camera

Imagine you are looking at a photo of a busy city street. Your eyes tell you exactly what the scene looks like: cars, people, concrete, and a blue sky. But your nose knows something the camera doesn't: the smell of exhaust fumes, the scent of hot asphalt, or the faint odor of a nearby bakery.

The authors of this paper point out a major flaw in how computers currently "see" the world. Cameras are great at capturing visuals, but they are "smell-blind." Even if a computer has a photo of a scene and a sensor reading of the smell in that same spot, it struggles to connect the two. Why? Because the smell often comes from things outside the frame of the photo.

The Analogy: Imagine you are in a room with a closed door. You can see the inside of the room (the photo), but you can smell the pizza baking in the kitchen through the crack in the door (the smell sensor). A computer looking only at the photo of the room has no idea the pizza exists. It's like trying to guess the flavor of a soup just by looking at a picture of the spoon.

The Solution: SCENT (The "Detective" Framework)

To fix this, the researchers built a system called SCENT (Semantic Context-aware e-Nose Transformer). Instead of just forcing the computer to match a picture to a smell, they introduced a third helper: Language.

Think of the computer as a detective trying to solve a mystery.

The Visual Clue: The photo (what is visible).
The Sensory Clue: The electronic nose reading (the chemical mix in the air).
The Language Clue: A smart AI assistant that acts as a "world expert."

How It Works: The "Smart Assistant"

The researchers use a powerful AI (called a Vision-Language Model or VLM) that has read millions of books and seen millions of images. This AI knows the "rules of the world."

Looking at the Photo: The AI looks at the image of a kitchen.
Thinking Beyond the Pixels: It doesn't just say, "I see a coffee maker." It uses its world knowledge to say, "I see a coffee maker on a counter. Therefore, it is highly likely there is a smell of coffee residue, plastic from the electronics, and dust in the air, even if those specific smells aren't visible in the picture."
Bridging the Gap: The system takes these text descriptions ("coffee," "plastic," "dust") and uses them as a semantic bridge. It teaches the computer: "When you smell this chemical mix, it matches the idea of a coffee kitchen, not just the picture of the coffee maker."

The "De-mixer" (Untangling the Smell)

Smells in the real world are messy. A single sniff might contain the smell of a specific object (like a banana) mixed with the background (like the smell of a park).

The paper introduces a clever trick called Latent Decomposition.

The Analogy: Imagine a smoothie made of strawberries and spinach. If you just taste the smoothie, it's hard to know exactly how much strawberry vs. spinach is in there.
The SCENT Method: The system learns to "deconstruct" the smell. It separates the "strawberry" part (the object-specific smell) from the "spinach" part (the background environment). It does this by using the text descriptions to tell it what to look for. This allows the computer to understand that the "coffee" smell belongs to the machine, while the "dust" smell belongs to the room.

The Results: Smelling Better Than Seeing

The team tested this on the "New York Smells" dataset, which contains thousands of pairs of city photos and smell readings.

The Old Way: Previous systems tried to match the smell directly to the photo. They were often confused because the photo didn't show the source of the smell.
The SCENT Way: By using the language "hints" (the AI's educated guesses about what should smell), the system became much better at finding the right match.
- If you gave the system a smell of "rain on hot pavement," it could find the right photo of a city street, even if the photo didn't show the rain itself, because the language bridge understood the context.
- It also got better at matching smells to text descriptions (e.g., "smell of a library") without needing a photo at all.

The "Magic Trick" Test

To prove the AI wasn't just guessing or "hallucinating" (making up fake smells), the researchers did a special test.

They showed the AI View 1 (a photo of a desk with a computer) and asked it to guess the smells.
The AI guessed things like "paper dust" and "plastic."
Then, they showed them View 2 (a different angle of the same room that the AI hadn't seen yet).
The Result: View 2 actually showed a stack of papers and a plastic chair—exactly what the AI had guessed! This proved the AI was using real-world logic to infer hidden details, not just making random guesses.

Summary

In short, this paper teaches computers to "smell" by giving them a language guide. Instead of just staring at a picture and a sensor reading, the computer asks a smart AI, "What should this place smell like based on what I see?" This extra layer of common sense helps the computer connect the invisible world of smells with the visible world of images, making it much better at understanding our environment.

Technical Summary: What Images Cannot Say: Language-Guided Olfactory Representation Learning

Problem Statement
The paper addresses the fundamental challenge of aligning high-dimensional electronic-nose (e-nose) signals with visual data. While recent datasets (such as New York Smells, NYS) pair visual scenes with olfactory measurements, aligning these modalities remains difficult. The authors identify a critical limitation: images capture only a partial view of a scene, whereas olfactory signals often originate from sources outside the camera's field of view (e.g., ventilation systems, distant traffic, or hidden objects). Consequently, visual supervision alone is insufficient to fully account for the measured smell signals. Furthermore, smell recordings represent complex mixtures of object-specific odors and contextual environmental factors, making it difficult to disentangle individual semantic contributions using vision-only approaches.

Methodology: SCENT Framework
The authors propose SCENT (Semantic Context-aware e-Nose Transformer), a multimodal framework that utilizes language guidance as a semantic bridge between vision and olfaction. The framework operates through three primary stages:

VLM-based Semantic Scene Augmentation:
The system leverages pre-trained Vision-Language Models (VLMs), specifically Qwen3-VL-30B, to generate structured textual descriptors from input images. Unlike standard captioning, the VLM is prompted to infer plausible ambient smell cues based on the visual context, objects, and environmental background. This produces a set of descriptors: object ( $t_{obj}$ ), context ( $t_{ctx}$ ), and inferred smells ( $t_{smell}$ ). These descriptors are encoded using a frozen CLIP text encoder to create a textual embedding ( $z_T$ ) that serves as a semantic anchor, compensating for the partial observability of the visual input.
Multimodal Olfactory Representation Learning:
A Transformer-based smell encoder ( $f_S$ ) processes the e-nose signals (temporal concatenation of baseline and sample recordings) to generate a global olfactory embedding ( $z_S$ ). Two projection heads map this embedding into the shared 512-dimensional spaces of CLIP's visual ( $z_I$ ) and textual ( $z_T$ ) encoders. The model is trained using symmetric InfoNCE contrastive losses to align the smell representations with both the visual scene and the augmented textual descriptors. This creates a joint embedding space where olfactory signals are grounded in both visual and semantic world knowledge.
Latent Smell Disentanglement:
To address the mixture nature of olfactory signals, the framework introduces a latent decomposition mechanism. The olfactory embedding is split into two components: an object-specific odor vector ( $z^S_{obj}$ ) and a contextual environmental vector ( $z^S_{ctx}$ ). These components are aligned with their respective textual descriptors ( $t_{obj}$ and $t_{ctx}$ ) via separate contrastive losses. A reconstruction constraint ensures that the sum of these disentangled components preserves the information of the original sensor signal, preventing representation collapse.

Key Contributions

Language-Guided Semantic Bridge: The paper introduces the concept of using VLM-generated language to infer olfactory cues that are not directly visible, effectively bridging the gap between partial visual observations and holistic smell measurements.
SCENT Framework: A novel architecture that aligns e-nose, visual, and textual embeddings, enabling the learning of olfactory representations that go beyond direct visual supervision.
Language-Guided Disentanglement: A method to separate object-specific odors from environmental context within the latent space, facilitating the interpretation of complex smell mixtures.

Experimental Results
The framework was evaluated on the New York Smells (NYS) dataset, comprising 7,000 image-olfactory pairs across diverse urban environments.

Retrieval Performance: SCENT achieves state-of-the-art performance on cross-modal retrieval tasks. In Smell-to-Image (S2I) retrieval, it improves Recall@5 from 20.0% (reproduced NYS baseline) to 23.0%. In Smell-to-Text (S2T) retrieval, it achieves 11.9% Recall@5, a 47% relative improvement over the best adapted baseline. Joint-modality retrieval (S2IT) also shows significant gains.
Ablation Studies: The results demonstrate that performance scales with the semantic granularity of the language guidance. Using only object labels yields poor results, while incorporating environmental context and VLM-inferred smells significantly boosts alignment.
Disentanglement Validation: In a zero-shot recombination retrieval task, the model successfully synthesized new smell fingerprints by combining object and context factors from different training samples, outperforming raw signal recombination baselines.
VLM Sensitivity: The study found that retrieval quality correlates more strongly with the VLM's reasoning capability (e.g., Qwen3-VL-30B) than with parameter count alone, highlighting the importance of inferring plausible, non-visible cues.

Significance and Claims
The authors claim that SCENT reveals the critical importance of contextual semantic information for grounding olfactory perception in multimodal learning. By using language to infer "what images cannot say," the framework enables the interpretation of smell measurements through world knowledge, allowing the model to generalize to out-of-distribution samples where visual cues are insufficient. The work establishes a new paradigm for olfactory representation learning, moving beyond constrained laboratory settings to "in-the-wild" scenarios by leveraging the extensive world knowledge embedded in modern VLMs. The paper concludes that this approach paves the way for future research in disentanglement and cross-modal alignment for non-visual sensory modalities.