Technical Summary: Masked Autoencoders Learn Perception-Relevant Representations from Resting State Neural Data

Problem Statement

Clinical neuroprosthetics face a critical data bottleneck in decoding subjective perception from neural activity. Training effective decoding models requires labeled data pairing neural recordings with perceptual reports, yet such data is scarce due to limited participant availability, short experimental sessions, and the difficulty of measuring subjective states. Conversely, hours of spontaneous neural activity accumulate during rest periods and between experimental blocks but remain largely underutilized. This imbalance is particularly severe at perceptual thresholds, where identical microstimulation can yield variable outcomes (perception vs. non-perception) due to fluctuations in cortical state. Supervised decoders, constrained by limited labeled examples, often fail to distinguish the subtle neural states that determine these threshold-level outcomes.

Methodology

Data Acquisition and Preprocessing

The study utilized data from a participant with acquired blindness implanted with a 96-channel Utah array in the primary visual cortex (V1). Three complementary datasets were employed:

1. Resting State: 252 sessions totaling 14.6 hours of spontaneous activity with no stimulation, used exclusively for pretraining.
2. Psychometric: Stimulation across 12 electrodes at six current levels (1–81 µA) to map intensity to perception.
3. Threshold: Stimulation at the perceptual threshold (50% detection rate) on 4 electrodes, where identical stimuli produced variable perceptual outcomes.

Neural data was processed into multi-unit activity (MUA). A key preprocessing distinction was applied: resting state data underwent continuous z-score normalization, while stimulation data used robust z-score normalization (median and median absolute deviation) to mitigate the impact of stimulation artifacts. Neural activity was extracted 170 ms post-stimulation onset for a 128 ms window to capture population responses while avoiding artifacts.

Model Architecture: Masked Autoencoder (MAE)

The authors employed a self-supervised learning framework using a Masked Autoencoder. The architecture incorporated standard modifications including pre-layer RMSNorm, SwiGLU, and QKNorm.

• Input: Signal windows of 96 electrodes × 128 ms were patchified (temporal patch size = 16, spatial patch size = 1).
• Masking: 75% of patches were randomly masked during training.
• Encoder: Processed only visible patches to generate latent representations.
• Decoder: Reconstructed masked patches using three inputs: (1) encoder outputs from visible patches, (2) learned mask tokens, and (3) a learned session embedding token.
• Positional Encoding: Separate learnable embeddings for temporal and spatial positions were summed for each signal embedding.

Training Objective

The model was trained for 1 million steps with a batch size of 32 using the AdamW optimizer. The loss function combined three terms to learn representations that generalize across sessions while preserving neural dynamics:

1. Reconstruction Loss: Mean Squared Error (MSE) calculated only on masked patches.
2. Correlation Loss: Measured between predicted and true signals.
3. Cross-Session Loss: An inverted SigLip formulation that pulls together samples from different sessions and pushes apart samples from the same session, forcing the model to learn underlying neural dynamics rather than session-specific features.

The final latent representation was obtained by averaging all encoder outputs, resulting in a 768-dimensional vector.

Evaluation Strategy

To evaluate the utility of the learned representations, the pretrained encoder was frozen. Features were extracted and fed into an L2-regularized logistic regression classifier to predict perception outcomes on the psychometric and threshold-level datasets. Performance was compared against two baselines:

1. Raw MUA data (flattened to 12,288 dimensions).
2. PCA-reduced MUA signals (768 dimensions).

Key Results

Emergence of Interpretable Structure

Despite receiving no supervision regarding stimulation or perception labels during pretraining, the MAE learned representations that captured complex brain organization:

• Spatial Organization: In t-SNE projections, trials from spatially adjacent electrodes clustered together, reflecting V1's retinotopic organization. Stimulation amplitude systematically shifted positions within these clusters.
• Perceptual State Separation: Trials resulting in perception ("True") and missed trials ("False") occupied partially distinct regions in the latent space. While substantial overlap remained (consistent with the stochastic nature of near-threshold perception), the separation indicated that spontaneous dynamics contain structure relevant to perceptual states.

Decoding Performance

The pretrained features significantly improved perception decoding accuracy compared to baselines:

• General Psychometric Task: The MAE approach achieved 84.1% balanced accuracy, outperforming raw MUA (80.2%) and PCA (82.4%). This improvement over PCA suggests the model captured relevant nonlinear structure missed by linear compression.
• Threshold-Level Task: On the more difficult task where perception varied stochastically for identical stimuli, the MAE achieved 64.0% accuracy, surpassing raw MUA (60.0%) and PCA (61.3%).

Analysis of the threshold-level latent space revealed that while some electrodes showed clean separation between perceived and missed trials, others exhibited heavy mixing. The authors attribute the ceiling in accuracy (64.0%) to the inherent stochasticity of near-threshold perception, where nearly identical neural states can lead to different outcomes.

Significance and Claims

The paper claims that spontaneous cortical activity is not merely noise but contains rich, task-relevant structure. By pretraining a masked autoencoder on 14.6 hours of unlabeled resting-state data, the authors demonstrated that self-supervised learning can effectively overcome the data bottleneck in clinical neuroprosthetics.

The core finding is that unsupervised pretraining on spontaneous activity allows a model to learn fundamental properties of the brain, such as spatial layout and perceptual state separation, which directly translate to improved decoding performance on labeled tasks. This approach offers a promising strategy to leverage abundant, unlabeled neural datasets to enhance neural decoding, particularly in scenarios where labeled data is scarce and perceptual thresholds are critical.

Limitations

The authors note that these findings are derived from a single subject, necessitating further replication to determine generalizability across participants. Additionally, the specific mechanisms by which spontaneous activity encodes perceptual structure remain speculative (e.g., exploring the same fluctuating brain states or replaying vision experiences). The current 128 ms analysis window may also be too brief to capture slower changes in cortical state, representing an area for future investigation.