🧬 biology

Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE

This paper introduces the Multi-Encoder-Decoder Variational Autoencoder (MED-VAE), a framework that achieves superior cross-subject neural alignment and generalizable decoding without requiring shared stimuli by anchoring neural representations to a common scaffold provided by a pretrained artificial neural network.

Original authors: Angeliki Papathanasiou, Jascha Achterberg, Thomas E. Nichols, Rui Ponte Costa

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Angeliki Papathanasiou, Jascha Achterberg, Thomas E. Nichols, Rui Ponte Costa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to translate a book written in a secret code. The problem is that you have 8 different people, each with their own unique version of the code. Even though they are all reading the exact same story, Person A writes "apple" as "red-fruit-circle," Person B writes it as "crunchy-sweet," and Person C writes it as "Orchard-1."

If you want to understand what they are all thinking about the story, you can't just compare their notes directly because the words don't match. Traditionally, scientists tried to solve this by making everyone read the same 872 pages of the book so they could build a dictionary based on those shared pages. But what if the people are reading different books, or the pages don't overlap? The old methods fail.

This paper introduces a new tool called MED-VAE that solves this problem without needing everyone to read the same pages. Here is how it works, using simple analogies:

1. The "Universal Translator" (The ANN Scaffold)

Instead of forcing the people to talk to each other directly, the researchers introduce a Universal Translator (an Artificial Neural Network, specifically a ResNet-50). This translator has already read millions of images and knows exactly what an "apple," a "car," or a "person" looks like in a standard, perfect way.

The Old Way: You ask Person A and Person B to describe the same apple, then you try to match their descriptions.
The New Way: You ask Person A to describe the apple, and you ask Person B to describe the apple. Then, you ask both of them to translate their descriptions into the Universal Translator's language.

Because the Universal Translator has a perfect, shared definition of what an apple is, Person A and Person B are forced to "speak the same language" when they talk to it, even if they never spoke to each other.

2. The "Two-Way Street" (The Architecture)

The researchers built a machine with two sides:

The Input Side: Each person has their own private translator that turns their brain scans (fMRI) into the Universal Translator's language.
The Output Side: There is a shared decoder that tries to turn that Universal language back into the Universal Translator's original "perfect" image features.

The magic happens because the machine is trained to do two things at once:

Make sure the Universal Translator's language can be turned back into a perfect image.
Make sure the Universal Translator's language can be turned back into that specific person's brain scan.

This creates a "pressure" on everyone's brain scans. To satisfy the machine, Person A's brain scan and Person B's brain scan must end up in the exact same spot in the "Universal Language" whenever they see similar things (like a cat), even if they saw different pictures overall.

3. The Results: A Better Map

When the researchers tested this, they found three major things:

A Neatly Organized Library: In the new shared space, the "books" (images) are sorted perfectly by category. If you look at a map of the data, all the "animals" are in one cluster, and all the "vehicles" are in another. The old methods were like a messy pile where animals and cars were mixed together.
No "Shared Pages" Needed: The old methods (like SRM and Procrustes) needed everyone to look at the same 872 images to learn how to align. MED-VAE didn't need that. It learned the alignment just by using the Universal Translator as a guide.
Better Prediction: Because the alignment is so good, if you take Person A's brain activity, translate it into the Universal Language, and then translate it back into Person B's brain language, you get a very accurate prediction of what Person B's brain would look like. It's like being able to read Person B's mind just by looking at Person A's, without ever having seen Person B's data before.

4. The "Noise" Filter

One interesting finding was about "noise." The old methods seemed to do a better job at reconstructing the exact brain scan during the experiment. However, the researchers realized this was because the old methods were accidentally memorizing the "static" or "noise" (like the person's heartbeat or breathing) that happened to be in the data at that exact moment.

When they tested the methods on new trials (checking if the signal was real or just noise), the old methods failed. MED-VAE, however, ignored the noise and kept the real signal. It's like the old methods were recording the background chatter along with the speech, while MED-VAE filtered out the chatter and kept only the clear voice.

Summary

The paper presents a way to align different people's brain activity without needing them to look at the same pictures. It does this by using a pre-trained AI as a "common ground" or scaffold. This creates a shared mental map where different brains can be compared, leading to better understanding of how we all see the world, and allowing us to predict one person's brain activity based on another's.

Technical Summary: Task-guided Cross-Subject Latent Alignment via Multi-Encoder-Decoder VAE

Problem Statement

Aligning neural activity across different individuals is essential for discovering shared computational principles and building generalizable decoders in systems neuroscience. However, traditional alignment methods (e.g., Shared Response Models, Procrustes analysis, Hyperalignment) rely on a critical constraint: they require shared stimuli viewed by all subjects to derive transformation matrices. This limitation severely restricts applicability to naturalistic paradigms where subjects view largely non-overlapping or non-shared data, preventing the aggregation of large-scale, cross-study neuroimaging datasets. Furthermore, methods trained solely on shared subsets often fail to generalize to held-out, non-shared stimuli, as their alignment transformations may not capture the full representational space required for broader datasets.

Methodology: MED-VAE

The authors introduce the Multi-Encoder-Decoder Variational Autoencoder (MED-VAE), a framework designed to achieve cross-subject alignment without requiring shared stimuli. The core insight is utilizing a pre-trained Artificial Neural Network (ANN) as a computational "scaffold" to anchor representations from different subjects into a common latent space.

Architecture

The MED-VAE employs a multi-pathway architecture:

Subject-Specific Encoders: Each subject $i$ has a dedicated encoder ( $Enc_{fMRI}^i$ ) that maps their fMRI responses ( $x_{fMRI}^i$ ) to a shared $d$ -dimensional latent space ( $z_i$ ).
Shared ANN Encoder: A shared encoder ( $Enc_{ANN}$ ) maps features from a pre-trained ANN (specifically ResNet-50) to the same latent space.
Subject-Specific Decoders: Each subject has a dedicated decoder ( $Dec_{fMRI}^i$ ) that reconstructs their original voxel patterns from the latent space.
Shared ANN Decoder: A universal decoder ( $Dec_{ANN}$ ) reconstructs the ANN features from any latent representation (whether derived from fMRI or the ANN itself).

Training Objective

The model is trained using a loss function that integrates four reconstruction pathways with KL regularization:

Within-modality fMRI: Reconstructing a subject's fMRI data from their own latent code.
Cross-modal ANN $\to$ fMRI: Reconstructing a subject's fMRI data from the latent code generated by the ANN encoder.
Within-modality ANN: Reconstructing ANN features from the ANN encoder's latent code.
Cross-modal fMRI $\to$ ANN: Reconstructing ANN features from a subject's fMRI-derived latent code.

The shared ANN decoder acts as the primary alignment mechanism. By forcing all subjects' latent representations to reconstruct the same hierarchical visual features and semantic relationships encoded by the ANN, the model imposes an implicit alignment pressure. This ensures that semantically similar stimuli (even if viewed by different subjects) map to proximal locations in the shared latent space. Crucially, the ANN scaffold is only required during training; at inference, subjects are projected into the shared space using only their fMRI encoders.

Key Contributions

Elimination of Shared Stimuli: MED-VAE achieves cross-subject alignment without requiring subjects to view identical stimuli, relying instead on the structural priors of a pre-trained ANN.
Semantic Organization: The method produces latent spaces with superior semantic organization, evidenced by higher silhouette scores and better cross-subject category decoding accuracy compared to traditional methods.
Robust Generalization: Unlike traditional methods that degrade on non-shared stimuli, MED-VAE demonstrates robust generalization to held-out images, capturing subject-invariant representations rather than overfitting to specific shared subsets.
Signal Preservation: Cross-trial evaluation reveals that while traditional methods may show higher within-trial reconstruction (often capturing trial-specific noise), MED-VAE preserves an equivalent level of stimulus-driven signal while suppressing noise.
Downstream Utility: The aligned space enables superior cross-subject neural prediction and image decoding, allowing the reconstruction of one subject's brain activity from another's latent representation.

Results

Experiments were conducted using the Natural Scenes Dataset (NSD), involving 8 subjects viewing ~10,000 natural images with minimal overlap.

Semantic Structure: MED-VAE latent spaces showed significantly higher silhouette scores (0.34) compared to SRM (0.22) and Procrustes (0.21), indicating coherent clustering of semantic categories. Leave-one-subject-out decoding also yielded higher accuracy for MED-VAE.
Alignment Metrics: MED-VAE achieved higher component-wise correlation (0.64 vs. 0.52/0.45) and Representational Similarity Analysis (RSA) scores (0.63 vs. 0.41/0.41) than baselines.
Generalization Gap: Traditional methods (SRM, Procrustes) showed a significant drop in performance when evaluated on a semi-shared held-out set (128 images) compared to their training set, whereas MED-VAE maintained consistent performance, indicating it did not rely on stimulus overlap for alignment.
Cross-Subject Retrieval: MED-VAE achieved a top-1 retrieval accuracy of 46.2% on the held-out set, more than double the performance of SRM (21.8%) and Procrustes (18.4%).
Noise vs. Signal: Within-trial reconstruction metrics favored traditional methods, but cross-trial evaluation (correlating reconstruction of Trial $i$ with ground truth of Trial $j$ ) showed no significant difference between methods. This confirms that traditional methods' advantage was due to reconstructing trial-specific noise, whereas MED-VAE preserved the stimulus-driven signal equally well.
Image Decoding: When cross-subject predictions were fed into a frozen MindEye2 decoder, MED-VAE outperformed baselines in high-level semantic measures (e.g., CLIP similarity, Inception scores), particularly in lower-dimensional (32D) spaces.

Significance and Claims

The paper claims that MED-VAE provides a framework to identify generalizable common subspaces for cross-subject predictions and downstream tasks without the bottleneck of shared stimuli. By replacing explicit stimulus matching with implicit constraints from a pretrained ANN, the method enables:

Population-level analyses on data from subjects scanned under different protocols or viewing different stimuli.
Cross-subject neural prediction and image decoding with minimal per-subject calibration.
Data augmentation for subjects with limited scanning time by borrowing aligned data from others.

The authors emphasize that the common space captures the computational organization of visual representation (categorical structure and semantic relationships) rather than arbitrary statistical correspondences. They note that the quality of alignment is fundamentally bounded by the model-brain correspondence of the chosen ANN scaffold (e.g., ResNet-50), suggesting that scaffolds with stronger dynamic or temporal modeling could further improve alignment in future work. The framework is presented as a step toward aggregating neural data across studies and individuals for more powerful neuroscience investigations.