Technical Summary: NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity

Problem Statement

Human vision is fundamentally dynamic, relying on hierarchically organized regions to process continuously changing input for tasks such as obstacle avoidance, object tracking, and social inference. While recent in silico brain encoding models have successfully synthesized optimal static stimuli to probe selectivity in specific brain regions, prior work has largely been limited to the static image domain. Consequently, the representation of temporal structure in the visual cortex, particularly in regions dependent on motion, interaction, or event dynamics, remains underexplored. There is a critical need for methods capable of systematically generating and evaluating dynamic stimuli across a large joint spatiotemporal configuration space to understand how the brain processes naturalistic dynamic input.

Methodology: NEvo Framework

The authors introduce NEvo, a neural-guided evolutionary video synthesis framework designed to generate stimuli that maximally activate target regions of interest (ROIs) in the human visual cortex. The framework operates by coupling a dynamic brain encoding model with an evolutionary search procedure over a structured prompt space.

1. Neural-Guided Evolutionary Search

NEvo treats video synthesis as a black-box optimization problem over interpretable semantic, motion, and event-level descriptors.

• Prompt Space: The search space is structured as a Cartesian product of visual attribute categories ($P = A_1 \times \dots \times A_M$), covering appearance (subject, mood, lighting), motion (event structure, camera motion), and temporal dynamics. Prompts are represented as gene sequences.
• Evolutionary Algorithm: The method employs a genetic algorithm to optimize the synthesis objective.
  • Initialization: A population of candidate prompts is generated.
  • Evaluation: Prompts are rendered into videos by a generator ($G$) and scored by a dynamic encoding model ($S_v$) based on the predicted average activation of voxels in the target ROI ($r$).
  • Selection & Variation: The top fraction of candidates (elites) are selected as parents. Offspring are generated via crossover (swapping gene-sequence prefixes) and mutation (resampling attributes).
  • Iteration: This process iterates over generations to refine the population toward prompts yielding stronger predicted ROI activation.

2. Two-Stage Video Synthesis

To improve search efficiency in the high-dimensional spatiotemporal space, NEvo decomposes the optimization into two stages:

• Stage 1 (Static Content): A fast image generator searches the static subspace ($P_{img}$) to identify an optimal "visual anchor" (subject, scene, texture, lighting).
• Stage 2 (Dynamic Attributes): The best-scoring image is fixed as a conditioning input for an image-to-video generator. The search then optimizes the dynamic subspace ($P_{vid}$), covering motion profiles, temporal rhythm, and interaction types. This separation avoids expensive video evaluations for poor content configurations.

3. Dynamic Brain Encoding Model

The framework utilizes V-JEPA 2 as the feature backbone, chosen for its alignment with brain responses in prior large-scale evaluations.

• Architecture: Unlike models relying solely on final embeddings (e.g., CLIP), NEvo extracts multi-block features from V-JEPA 2, averaging them over space and time.
• Mapping: Voxel-wise ridge regressions are fitted to fMRI responses. Each voxel is assigned the model layer with the best validation performance, allowing cortical locations to align with different model depths.
• Training Data: The model is trained on the BOLDMoments dataset and a social interaction dataset, projected onto the fsaverage5 cortical surface.

Key Contributions

1. Novel Framework: Introduction of NEvo, the first model-based framework for studying visual selectivity under naturalistic dynamic stimulation using evolutionary prompt search.
2. High-Activating Stimuli: Synthesis of videos that exceed the activation levels of both handcrafted functional localizers and top-performing videos from large-scale datasets (e.g., Moments in Time) across ventral, dorsal, and lateral visual streams.
3. Dynamic Selectivity Characterization: Demonstration that temporal structure broadly shapes visual selectivity, with optimized videos outperforming static controls even in traditionally static-selective regions like the Fusiform Face Area (FFA).
4. Discovery of Functional Gradients: Application of NEvo along a searchlight trajectory in the lateral visual stream reveals a systematic progression from low-level spatial structure to body motion, and finally to complex social-dynamic features (joint action, communication).
5. Controlled Synthesis: Validation of the framework's ability to dissociate region-specific dynamic selectivity from static anchors by generating stimuli with abstract, non-naturalistic objects that still elicit region-specific dynamic responses (e.g., face-like interactions for pSTS vs. oscillatory motion for MT).

Results

• Performance: NEvo-generated stimuli achieved predicted activations in the top 99.8% of the Moments in Time dataset and 95.8% of handcrafted localizer responses on average.
• Dynamic Advantage: Optimized videos consistently outperformed their first-frame static controls, with the strongest gains in motion-sensitive regions (e.g., MT) and measurable gains in ventral regions (e.g., FFA).
• Ablation Studies:
  • The two-stage search strategy outperformed single-stage baselines.
  • The V-JEPA 2 multi-layer encoding model significantly outperformed CLIP-based baselines, highlighting the necessity of temporally grounded representations.
  • Evolutionary prompting proved more robust than gradient-based synthesis (BrainDiVE), which failed to generate meaningful dynamic stimuli in this setting.
  • Genetic search significantly outperformed random search and hill-climbing baselines.
• Lateral Stream Analysis: Searchlight synthesis revealed a gradient along the lateral pathway: early regions preferred high-contrast textures; mid-level regions (MT, EBA) favored body motion; posterior STS (pSTS) favored synchronized physical interactions; and anterior STS (aSTS) shifted toward face-to-face communicative content.

Significance and Claims

The paper claims that NEvo enables in silico exploration of dynamic visual selectivity, moving beyond the limitations of static image synthesis. By recovering known selectivities and revealing systematic differences in sensitivity to temporal dynamics, the framework provides new predictions for in vivo experiments. The authors posit that NEvo offers a scalable, controlled approach to designing dynamic stimuli, potentially aiding in the discovery of functional gradients in less-characterized brain regions.

The authors acknowledge limitations, noting that synthesized stimuli reflect both neural selectivity and potential biases in the underlying encoding and generation models. They emphasize that future validation with human neuroimaging is necessary and that the current vision-only model limits the characterization of regions like aSTS where communication relies on audiovisual cues. The work is presented as a step toward closed-loop neuroimaging and hypothesis-driven stimulus design.