What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment

Technical Summary: What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment

Problem Statement
Zero-shot visual decoding from electroencephalography (EEG) aims to infer visual semantics from non-invasive neural recordings without prior exposure to specific test categories. While recent advancements in large-scale datasets (e.g., THINGS-EEG) and pretrained vision-language models (e.g., CLIP) have facilitated this task, significant challenges remain. EEG signals are inherently noisy, non-stationary, and spatially coarse. Existing alignment methods often rely on "holistic" EEG embeddings that fail to capture the heterogeneous structure of neural responses. Specifically, these approaches frequently overlook the complementary temporal dynamics, spectral patterns, and spatial dependencies inherent in EEG data, leading to unstable representations that align weakly with visual semantics, particularly in cross-subject and cross-session scenarios.

Methodology
The authors propose a unified multiview EEG representation learning framework designed to construct a robust EEG encoder that aligns neural signals with pretrained visual semantic spaces. Rather than modifying the vision encoder, the method focuses on learning a stronger, structured EEG encoder that jointly models three complementary views:

1. Temporal Modeling (Input-Conditioned State-Space): To capture long-range and input-adaptive temporal dependencies, the framework employs a Selective State-Space Model (S3M). This module maintains a latent state that evolves over time based on input-dependent step sizes and projections. The output is further refined using 1D convolutions to capture local temporal patterns.
2. Spectral Modeling (Adaptive Learnable Wavelet Decomposition - ALWD): Moving beyond predefined frequency bands, the method introduces ALWD to dynamically select frequency representations conditioned on the input signal. It utilizes a bank of learnable low-pass and high-pass filters, synthesizing sample-specific weights to decompose EEG into low-frequency (stable oscillations) and high-frequency (transient activity) components. Separate S3Ms process these components, followed by cross-frequency attention mechanisms to integrate information across spectral scales.
3. Spatial Modeling (Attention-Modulated Graph Learning): Recognizing that electrodes capture interacting brain regions, the framework models spatial dependencies using a Graph Convolutional Network (GCN). It constructs a dynamic adjacency matrix combining a learnable global structure with input-dependent edge attention, allowing the model to capture both shared spatial structures and sample-specific functional interactions.

Alignment and Training
The temporal, spectral, and spatial features are fused into a compact EEG embedding ($z_{eeg}$) and aligned with visual embeddings ($z_{img}$) extracted from a frozen CLIP-ViT-H-14 encoder. The alignment is achieved through:

• Contrastive Learning: A symmetric contrastive loss aligns paired EEG-image embeddings in a shared semantic space.
• EEG-Specific Regularization: To address EEG variability, the authors introduce two regularizers:
  • Repetition Consistency Regularization (RCR): Enforces consistency across repeated trials of the same stimulus.
  • Prototype-Level Contrastive Regularization (CPCA): Enhances class-level semantic structure.

Key Contributions

• Unified Multiview Framework: The first framework to jointly model long-range temporal dynamics, adaptive spectral structure, and spatial electrode interactions for zero-shot EEG visual decoding.
• Complementary Encoder Design: Introduction of specific modules (S3M, ALWD, and attention-modulated GCN) that explicitly disentangle and model the distinct characteristics of EEG signals.
• Regularization Strategies: Incorporation of trial-level consistency and class-level semantic alignment to improve representation stability.
• Systematic Cross-Session Evaluation: The paper presents the first systematic evaluation of EEG-image decoding across different recording sessions, a setting previously unexplored in semantic decoding contexts.

Experimental Results
The method was evaluated on the THINGS-EEG benchmark (1,854 object concepts, 10 subjects) under three zero-shot settings:

• Within-Subject: Achieved 54.8% Top-1 and 85.6% Top-5 accuracy, outperforming the strongest baseline (UBP) by +3.9% and +5.9% respectively.
• Cross-Subject: Achieved 15.3% Top-1 and 45.4% Top-5 accuracy, showing a +2.9% Top-1 and +12.0% Top-5 improvement over UBP, indicating robust generalization across subjects.
• Cross-Session: Achieved 40.8% Top-1 and 78.0% Top-5 accuracy, demonstrating stability across recording sessions where most existing methods lack reported baselines.

Ablation studies confirmed that removing any of the three views (temporal, spectral, or spatial) or the regularization terms significantly degrades performance, validating the necessity of the multiview approach.

Significance and Claims
The paper claims that explicitly modeling the multiview neural structure of EEG signals significantly improves both semantic alignment and generalization in visual decoding tasks. By moving away from holistic embeddings toward a structured representation that respects the temporal, spectral, and spatial nature of brain activity, the proposed method provides a more reliable basis for EEG-visual alignment. The authors suggest that this approach offers a scalable path toward more robust neural decoding, particularly in realistic scenarios involving session-dependent variability and cross-subject generalization. The findings highlight that structured inductive biases are crucial for bridging the gap between noisy neural recordings and high-level visual semantics.