Technical Summary: STSBENCH

Problem Statement

The primate visual system is functionally divided into two streams: the ventral stream, specialized for object recognition, and the dorsal stream, specialized for encoding spatial relations and motion. While convolutional neural networks (CNNs) pretrained on object recognition have proven highly effective at predicting neuronal responses in the ventral stream (e.g., V4, IT cortex), models for the dorsal stream remain underdeveloped. This gap is primarily attributed to a lack of large-scale datasets encompassing dorsal stream areas. Existing datasets, such as those containing recordings from area MT, are orders of magnitude smaller (e.g., ~45 neurons) compared to ventral stream datasets (thousands of neurons), thereby constraining the development of deep learning-based models for dorsal stream processing.

Methodology

The authors introduce STSBENCH, a large-scale dataset and benchmark designed to address the scarcity of dorsal stream data.

Dataset Collection:

• Subjects: Two male Rhesus macaques.
• Recording Technology: High-density extracellular recordings using Neuropixels 1.0 NHP probes positioned in the superior temporal sulcus (STS), specifically targeting the middle temporal (MT) and medial superior temporal (MST) areas.
• Stimuli: Monkeys viewed approximately 4,500 unique, 200ms natural video clips from the Ego4D dataset while performing a video fixation task.
• Scale: The dataset contains recordings from 2,244 neurons, representing a nearly 50-fold increase over existing dorsal stream datasets.
• Preprocessing: Neural data was spike-sorted using Kilosort4. Neurons were filtered for average firing rates (>2 Hz) and response reliability (>0.5). Waveforms were classified into putative cell types: regular-spiking (RS, excitatory), fast-spiking (FS, inhibitory), and axonal-spiking (AS).

Modeling Approach:
The paper evaluates two primary modeling tasks:

1. Encoding Models: These map visual stimuli to neuronal firing rates. The authors tested a suite of feature extractors followed by a factorized readout layer. Feature extractors included:
   • Hand-tuned 3D Gabor filters.
   • Pretrained 3D ResNets (trained on Kinetics for action recognition or self-motion estimation).
   • 3D CNNs trained end-to-end (varying depths from 1 to 7 layers).
   • Input to models was a 150×150 crop of the video.
2. Reconstruction Models: These map neural activity back to visual stimuli. The authors utilized a neural-conditional latent diffusion model.
   • Architecture: A vector-quantized variational autoencoder (VQ-VAE) maps images to a latent space, and a denoising U-Net generates images conditioned on neural activity.
   • Modification: The text-embedding input in standard Stable Diffusion was replaced with a vector of firing rates from the recorded neurons.
   • Metrics: Performance was evaluated using Peak Signal-to-Noise Ratio (PSNR) and Learned Perceptual Image Patch Similarity (LPIPS), compared against 'shuffled' and 'mean' null models.

Key Contributions

1. Release of STSBENCH: A dataset of single-neuron recordings from 2,244 neurons in the STS (MT/MST) in response to ~4,500 unique natural videos.
2. Benchmarking Encoding Models: An extensive evaluation of encoding models for dorsal stream responses, identifying that end-to-end trained 3D CNNs outperform existing baselines (hand-tuned Gabor filters and pretrained ResNets).
3. Neural Reconstruction: Demonstration of successful visual stimulus reconstruction from dorsal stream neural activity using a conditional latent diffusion model, validating the dataset's utility for decoding tasks.

Results

Encoding Performance:

• Model Depth: 3D CNNs trained end-to-end outperformed all baseline models. Performance increased with network depth, plateauing after five layers, suggesting that deep nonlinear computations are critical for predicting MT/MST responses.
• Pretrained vs. End-to-End: Features from earlier layers of pretrained 3D ResNets (Kinetics) were better predictors than later layers, indicating MT/MST neurons are more selective to lower-level visual representations than high-level action categories.
• Cell Type: Models performed significantly better for regular-spiking (putative excitatory) neurons compared to fast-spiking (putative inhibitory) neurons.
• Feature Analysis: Filters learned by a 1-layer end-to-end CNN included drifting Gabor-like patches (consistent with MT theory) as well as complex properties like 'on-off' responses, rotations, and chromatic features not typically emphasized in textbook MT models.

Reconstruction Performance:

• Dorsal Stream (MT/MST): Reconstructions successfully captured low-spatial frequency luminance contours and motion statistics (e.g., edges of objects) but lacked high-spatial frequency details and object identity (e.g., specific logos or textures). This aligns with the theoretical "where" function of the dorsal stream.
• Ventral Stream (V4) Comparison: Reconstructions from V4 data (TVSD) captured detailed form, color, and object identity ("what" function).
• Quantitative Metrics: The conditional diffusion model significantly outperformed 'shuffled' and 'mean' null models on both PSNR and LPIPS for dorsal stream data, confirming that the model utilized specific neural activity to condition image generation.

Significance and Claims

The authors position STSBENCH as a foundational resource for systems neuroscience, specifically for the dorsal stream. The paper claims:

• Bridging the Data Gap: STSBENCH provides the necessary scale to develop and benchmark deep learning models for dorsal stream processing, a field previously limited by small datasets.
• Model Refinement: The results demonstrate that simple 3D CNNs trained end-to-end can surpass current conceptual models (Gabor banks, self-motion estimators) in predicting dorsal stream activity, suggesting a need to refine theories of MT/MST processing.
• Functional Validation: The reconstruction results provide qualitative and quantitative evidence that dorsal stream neurons encode motion and luminance contrast, while lacking object identity details, thereby validating the functional segregation between dorsal and ventral streams at the single-neuron level.
• Future Utility: The dataset and baselines serve as a first step toward a comprehensive understanding of dorsal stream visual areas, enabling future studies to apply interpretability methods to deeper models and explore nonlinear computations in MT/MST.

The paper remains modest regarding its scope, acknowledging that the tested models were not exhaustive and that the dataset has limitations regarding potential "split" or "merge" errors in spike sorting and the inclusion of superficial neurons from adjacent areas. It does not claim to solve the dorsal stream problem entirely but rather provides the infrastructure for the community to do so.