Interpreting V1 Population Activity via Image-Neural Latent Representation Alignment

Technical Summary: Dual-Tower Image–Neural Alignment (DINA)

Problem Statement
Understanding the neural mechanisms underlying visual computation in the primary visual cortex (V1) remains a central challenge. While recent alignment-based approaches have improved the accuracy of decoding visual stimuli from brain activity, they often treat the alignment process as a "black box" optimized solely for predictive performance. Consequently, these methods provide limited insight into the specific neural computations and intermediate feature representations that give rise to successful decoding. Existing encoding and decoding models typically operate at the level of raw images, failing to explicitly model the intermediate feature maps that neural populations actually encode. There is a need for a framework that not only decodes stimuli accurately but also provides interpretable access to the latent feature maps to probe population-level computational principles.

Methodology: Dual-Tower Image–Neural Alignment (DINA)
To address this, the authors propose DINA, an interpretable contrastive framework that aligns visual stimuli and corresponding V1 population responses in a shared latent space at the level of intermediate feature maps. The architecture consists of two biologically motivated towers trained jointly:

1. Image Tower: Designed to mirror cortical visual processing, this tower employs a dual-stream architecture inspired by the parallel processing of local and global information.

   • Local Stream: Processes high-resolution inputs using depth-wise convolutions to extract spatially localized patterns.
   • Global Stream: Processes low-resolution inputs using self-attention mechanisms to capture long-range contextual dependencies.
   • Co-Attention: An inter-scale co-attention mechanism enables bidirectional information exchange between the local and global streams, integrating fine-grained structure with global context.
   • Output: The tower produces a compact, single-channel feature map (1×16×64) that preserves spatial structure.

2. Neural Tower: This tower reconstructs a latent representation from population-level neural responses (e.g., two-photon calcium imaging data) that matches the dimensionality of the image tower output.

   • Embedding: Individual neurons are mapped to learnable embedding vectors to capture neuron-specific response properties.
   • Self-Attention: A self-attention module models interactions within the neural population, allowing informative responses to be integrated and amplified.
   • Aggregation: Neuron-specific aggregation layers project the interaction-aware representations into scalar values, which are then passed through a Multi-Layer Perceptron (MLP) to generate the final latent feature map (16×64).

3. Training Objective: The two towers are trained using a contrastive objective (InfoNCE loss) over batches of image–neural pairs. This encourages the alignment of matched image–neural trials in the shared latent space while separating mismatched pairs.

Key Results
Evaluated on large-scale two-photon calcium imaging data from mouse V1 (involving ~10,000 neurons and 2,800 stimuli), DINA demonstrates the following:

• Decoding Performance: DINA achieves reliable neural-based decoding, with a mean top-1 retrieval accuracy of 36.8% and top-5 accuracy of 60.5% across four mice. It outperforms baseline models using standard visual encoders (VGG-16 and ViT) paired with MLP readouts.
• Nature of Aligned Representations: Analysis of the aligned feature maps reveals that successful decoding is primarily supported by coarse, low-level visual structure rather than semantic category information or fine-grained details. This is evidenced by:
  • High structural similarity (SSIM) and low perceptual distance (LPIPS) between image and neural feature maps.
  • Variance spectra showing rapid decay in higher-order components, similar to natural images but dominated by low-frequency structure.
  • Controlled stimulus manipulations: Retrieval accuracy drops sharply when low-dimensional structure is removed (whitened images) but remains high for low-dimensional (8D) images that retain coarse structure.
• Spatial Contributions: Interpretability analyses of the image tower show that aligned feature maps arise from multiple spatially distributed regions ("blobs") rather than uniform integration. These regions capture both shape and texture cues, with mixed representations being most common.
• Neural Readout Mechanisms: Ablation studies on the neural tower reveal that the aligned feature maps are predominantly reconstructed by sparse subsets of strongly responsive neurons (approximately 12% of the population is sufficient to recover high similarity). Furthermore, functional interactions (modeled via attention scores) are highly skewed, with a small fraction of connections carrying disproportionately large weights, consistent with sparse and efficient cortical coding.

Key Contributions

1. Analytical Framework: The authors formulate contrastive image–neural alignment not merely as a decoding tool, but as a general analytical framework for probing population-level neural computation.
2. Biologically Grounded Architecture: They propose a dual-tower model inspired by cortical visual processing (parallel local/global pathways and population interactions) that produces aligned feature maps in V1 while achieving reliable decoding.
3. Mechanistic Insights: Multi-level interpretability analyses reveal that V1 population coding is driven by low-level visual structure, spatially distributed stimulus contributions, and sparse neural readout mechanisms involving highly responsive neurons and non-uniform functional interactions.

Significance and Claims
The paper claims that DINA provides a principled framework for probing the computational mechanisms underlying visual processing in V1. By aligning representations at the intermediate feature map level, the framework moves beyond simple decoding accuracy to reveal how visual information is represented. The results confirm that V1 population activity encodes coarse, low-dimensional structures and that this encoding relies on sparse, distributed neural mechanisms. The authors position this work as a tool for mechanistic investigation in neuroscience, offering a pathway to extend such alignment frameworks to other brain regions to probe hierarchical organization, rather than as a direct tool for real-world decision-making or brain-computer interfaces.