🧬 biology

EmoMind: Decoding Affective Captions from Human Brain fMRI

EmoMind is a novel end-to-end pipeline that decodes affective captions directly from fMRI signals by combining semantically grounded scene descriptions with continuous 34-dimensional emotion vectors, significantly outperforming discrete label-based baselines in generating individualized, person-specific emotional narratives.

Original authors: Bilal A. Mohammed, Lin Gu, Ruogo Fang

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bilal A. Mohammed, Lin Gu, Ruogo Fang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are watching a movie, and your brain is lighting up with a complex mix of feelings: a little fear, a lot of awe, and a touch of nostalgia. For a long time, scientists could read your brain waves to tell what you were seeing (e.g., "a dog chasing a ball"), but they couldn't tell how you felt about it. They could guess you were "happy" or "sad" based on broad categories, but they missed the unique, messy, personal flavor of your emotion.

EmoMind is a new system that changes the game. It doesn't just read your brain to see the scene; it reads your brain to understand your specific emotional reaction to that scene and writes a caption that captures that feeling.

Here is how it works, using a simple analogy:

The Two-Step Recipe

Think of the system as a two-step cooking process:

Step 1: The "What" (The Neutral Base)
First, the system looks at your brain activity and asks, "What is on the screen?" It retrieves a plain, neutral description of the scene, like a dry news report.

Analogy: Imagine a robot looking at a photo of a storm and saying, "There are dark clouds and rain falling." This is the factual base.

Step 2: The "How" (The Emotional Seasoning)
Next, the system looks at your brain activity again, but this time it ignores the picture and focuses entirely on your feelings. Instead of picking a single label like "Scared," it extracts a 34-dimensional emotion vector.

Analogy: Think of this as a complex spice rack with 34 different spices (Joy, Fear, Awe, Nostalgia, etc.). The system measures exactly how much of each spice is in your brain right now. Maybe you have a little bit of "Awe," a lot of "Anxiety," and a dash of "Sadness."

The Magic Rewriter
Finally, the system takes the neutral description from Step 1 and the 34-spice mix from Step 2 and feeds them into a "Rewriter."

Analogy: This Rewriter is like a chef who takes the plain "rain" description and seasons it perfectly with your specific spice mix.
- If your brain says "Anxiety," the chef might rewrite the sentence to: "The dark clouds loom threateningly, and the rain feels like a cold, heavy weight."
- If your brain says "Awe," the chef might rewrite it to: "The sky is a dramatic canvas of swirling clouds, and the rain falls with a majestic rhythm."

The result is a caption that describes the same scene but carries the exact emotional tone of the person who saw it.

Why This is a Big Deal

The paper compares EmoMind to a "smart" AI (GPT-4) that is just told, "This scene is scary."

The Old Way (Labels): If you tell the AI "Scary," it gives a generic scary description. It's like ordering a "Spicy" pizza; you get the same generic hot sauce on top, regardless of whether you like it mild or extra hot.
The EmoMind Way (Continuous Vector): EmoMind reads your brain's unique "spice profile." It knows your specific version of fear.

The researchers tested this in two ways:

Uniqueness: When six different people watched the same clip, EmoMind wrote six different captions that matched each person's unique brain pattern. The old label-based AI wrote the same generic caption for everyone.
Control: The researchers tried to trick the system by swapping the emotions. If they told the system to use the emotions from a different video, EmoMind immediately followed the new instructions. The old AI kept leaking hints of the original video's emotion, proving it wasn't truly listening to the new emotional signal.

The "Synthetic Brain" Test

To see if this works even if we don't have real brain scans, the researchers tried using a computer simulation of a brain (a "synthetic brain") instead of real fMRI data.

The Result: The system still worked well enough to capture the feeling of a single clip, but it struggled to capture the relationship between different clips. It's like the synthetic brain could tell you "this is sad," but it couldn't tell you how this sadness was different from that sadness. This suggests that real human brains hold a complex map of emotions that current simulations haven't fully captured yet.

The Bottom Line

EmoMind proves that we can decode a person's continuous, personal emotional state directly from their brain waves and use it to write text that feels like them. It moves us from asking "What is this person feeling?" (with a simple label) to "How does this person feel?" (with a rich, detailed description).

The paper concludes that this continuous signal is a powerful tool for understanding how individual brains organize emotions, offering a way to see the world through someone else's emotional eyes, one caption at a time.

Technical Summary: EmoMind – Decoding Affective Captions from Human Brain fMRI

Problem Statement

Current brain-to-text decoding systems have made significant strides in recovering semantic content from fMRI signals, such as reconstructing visual scenes or speech. However, these systems largely discard the affective dimension of perception. Existing approaches either generate neutral descriptions devoid of emotion or condition Large Language Models (LLMs) on coarse, categorical emotion labels (e.g., "happy," "sad"). These categorical labels fail to capture the rich, continuous, and highly variable nature of individual emotional experiences, collapsing inter-subject variability into discrete bins. Furthermore, while LLMs can generate emotional text when prompted with labels, they cannot read out a specific subject's continuous, person-specific affective state directly from neural activity.

Methodology: The EmoMind Pipeline

EmoMind is the first end-to-end pipeline designed to decode affective captions directly from fMRI signals by utilizing a continuous, high-dimensional emotion vector as a control signal. The architecture operates in two primary stages:

1. Dual-Stream Brain Decoding (Stage 1)

Given an fMRI voxel response vector $v$ from a subject viewing a video clip, the system decodes two complementary representations using per-subject linear models:

Semantic Retrieval ( $x$ ): A neutral scene description is retrieved from a pre-indexed training corpus. This is achieved by mapping fMRI signals to a multi-layer feature stack of a pretrained DeBERTa-large model (24 layers) and performing a nearest-neighbor cosine search. This replaces iterative token-by-token generation with a single-pass retrieval, significantly reducing inference costs.
Continuous Emotion Decoding ( $e$ ): A subject-specific ridge regression model maps fMRI voxels to a 34-dimensional Cowen & Keltner (C&K) emotion vector. Unlike categorical labels, this vector represents fine-grained emotional states (e.g., Joy, Fear, Awe, Nostalgia) as continuous probabilities derived from human ratings of the stimuli.

2. Axis-Token Rewriter (Stage 2)

A trained encoder-decoder language model (BART-base) acts as a rewriter that transforms the neutral caption $x$ into an emotion-styled caption $\hat{y}$ , conditioned on the decoded emotion vector $e$ .

Architecture: The model incorporates 34 learned "axis tokens" (one for each C&K emotion category). The brain-decoded emotion vector $e$ scales these axis tokens element-wise, which are then concatenated to the encoder's hidden state. This allows the decoder to interpret the emotion axes as additional virtual context tokens without disrupting the pretrained semantic representation of the input caption.
Training Objective: The model is trained with a Bernoulli-switched objective. With probability $\rho$ , it performs a reconstruction task (input $x$ with a zeroed emotion vector), and with probability $1-\rho$ , it performs an emotion-target task (input $x$ with emotion vector $e$ targeting a specific emotional caption). This explicit identity-preserving null branch teaches the model when to preserve semantic content and when to allow affective control.
Inference (Classifier-Free Guidance): To control the trade-off between semantic fidelity and affective expression, the system applies classifier-free guidance (CFG). The final encoder output is a linear extrapolation: $H_{cfg} = H_{null} + (1+w)(H_{cond} - H_{null})$ , where $w$ is the guidance weight. Higher $w$ values amplify affective expression at the cost of some content fidelity.

Key Contributions

First Continuous Brain-Conditioned Pipeline: EmoMind introduces the first system to condition text generation on a continuous, decoded emotion vector derived directly from fMRI, rather than relying on discrete categorical labels.
Three-Axis Validation Framework: The authors propose a rigorous evaluation framework spanning:
- Subject-Specificity: Measuring whether different subjects viewing the same scene produce distinct captions proportional to their neural differences.
- Structural Geometry (RSA): Assessing whether the relational structure between clips in the decoded emotion space is preserved in the generated captions.
- Causal Control (SWAP): Testing if the rewriter follows a shifted emotion target (from a different clip) rather than reverting to the input scene's inherent affect.
Synthetic-Brain Robustness Test: The framework includes a "TRIBE v2 substitution" test, replacing measured fMRI with synthetic activity predicted from video features to diagnose representational gaps in current video-to-brain encoders.

Experimental Results

The system was evaluated on two independent fMRI datasets (MindCaptioning and Horikawa 2020) against a strong baseline: GPT-4 prompted with the ground-truth top-5 categorical emotion labels.

Subject-Specificity: EmoMind significantly outperformed the GPT-4 baseline. The diversity of captions across subjects was 3.4× higher on emotion-vector similarity (CK34 cosine) and 11.6× higher on lexical diversity. Crucially, this diversity was organized by neural similarity; subjects with similar brain responses produced more similar captions, a structure absent in the label-prompted baseline.
Structural Geometry: The Spearman correlation between the brain-decoded Representational Dissimilarity Matrix (RDM) and the caption RDM was +0.635 for EmoMind, compared to +0.435 for GPT-4. This indicates that continuous brain decoding preserves the relational geometry of emotional experiences better than categorical labels.
Causal Control: In the SWAP test, EmoMind achieved an "own-leakage" rate of 2.8% (statistically indistinguishable from chance), meaning it successfully followed the shifted target without reverting to the original clip's affect. GPT-4 leaked the original clip's affect at 8.3%, demonstrating that continuous brain signals provide a cleaner control signal than categorical labels.
Synthetic-Brain Analysis: When fMRI was replaced by synthetic activity (TRIBE v2), causal control remained robust (leakage 5.2%), but structural geometry collapsed (RSA dropped from +0.635 to +0.166). This suggests that while current video-to-fMRI encoders can capture clip-specific affect, they fail to preserve the relational structure between clips.

Significance and Claims

The paper claims that continuous brain-decoded affect is a viable control signal for individualized affective caption generation, offering capabilities that categorical labels cannot achieve.

Individual Variability: The results establish that individual differences in emotional experience are encoded in distributed, high-dimensional neural patterns that are lost when aggregated into population-level labels. EmoMind successfully captures this "inter-subject variability," which is critical for translational applications in mental health where population averages often miss clinically meaningful signals.
Beyond Surface Metrics: While GPT-4 outperforms EmoMind on standard surface-overlap metrics (BLEU, ROUGE) because it has access to ground-truth text and labels, EmoMind dominates on metrics requiring person-specific affective structure.
Neuroscientific Insight: The dissociation between causal usability (preserved in synthetic brains) and structural usability (lost in synthetic brains) suggests that current video-to-fMRI encoders lack the interoceptive or subject-specific signals necessary to fully reconstruct the relational geometry of affect.

The authors conclude that EmoMind opens new directions for studying individual affective brain organization and provides a substrate for generating text that reflects not just what a person saw, but how they felt about it, in a manner specific to their unique neural signature.