MindPilot: Closed-loop Visual Stimulation Optimization for Brain Modulation with EEG-guided Diffusion

1. Problem Statement

The paper addresses a critical gap in Brain-Computer Interface (BCI) research: while decoding neural signals into intent is well-studied, the reverse challenge—using controlled visual stimuli to steer brain activity—remains underexplored, particularly in non-invasive settings.

• Core Challenge: Designing images that consistently elicit specific neural responses is difficult because:
  • Human subjective states lack clear quantitative measures.
  • EEG signals are noisy, variable, and non-differentiable (gradients cannot be backpropagated through the brain).
  • Existing methods are often invasive (relying on cortical recordings) or limited to low-level flicker stimuli, failing to engage higher-order cognitive states with naturalistic images.
• Goal: To develop a framework that uses non-invasive EEG feedback to iteratively optimize and generate naturalistic images that drive the brain toward a specific target state (semantic or spectral).

2. Methodology: MindPilot Framework

MindPilot is a closed-loop framework that treats the brain as a "black-box" function. It combines black-box optimization with diffusion-based generative models to synthesize stimuli without requiring explicit rewards or differentiable gradients from the brain itself.

A. Problem Formulation

The system seeks an optimal image $u^*$ that maximizes the similarity between the EEG features evoked by the image and a target feature vector $y_{target}$:
$$u^* = \arg \max_u \text{sim}(f(g(u)), y_{target})$$
Where:

• $g(u)$: The black-box forward process (the brain or a proxy model) mapping image $u$ to EEG signal $x$.
• $f(x)$: An EEG encoder extracting features (semantic or spectral) from the signal.
• $\text{sim}(\cdot)$: Cosine similarity.

B. Key Components

1. Black-Box Proxy Model ($g$):
   • To avoid the cost and latency of real-time human EEG during the search phase, MindPilot uses a pre-trained surrogate model (e.g., AlexNet, ResNet, ViT, DINO) trained to predict EEG responses from visual features.
   • These models are trained on the THINGS-EEG2 dataset to regress visual features to 17-channel EEG signals.
2. Feature Extraction ($f$):
   • Semantic Features: Aligned with CLIP embeddings (using ATM-S encoder).
   • Spectral Features: Derived from Power Spectral Density (PSD) of EEG signals.
3. Closed-Loop Optimization Strategies:
   • Interactive Search (Retrieval): Uses a roulette-wheel selection mechanism. It updates the selection probability of images in a database based on similarity scores. It employs a Direct Reward Update (Exponential Moving Average for top-$k$ images) and a Spreading Update (propagating rewards to similar images based on CLIP similarity) to refine the search distribution.
   • Heuristic Generation (Synthesis): Integrates Diffusion Models (SDXL-Lightning) with IP-Adapter. Instead of calculating gradients through the black box, it uses a Gaussian Process (GP) surrogate to estimate a "pseudo-target embedding" in the latent space. This allows for gradient-free optimization where new images are generated via crossover and mutation of latent embeddings, guided by the GP's prediction of reward gradients.

3. Key Contributions

1. First Non-Invasive Closed-Loop Framework: MindPilot is the first system to use non-invasive EEG to guide the generation of naturalistic, high-level semantic images, moving beyond low-level flicker paradigms.
2. Pseudo-Model Guidance Mechanism: Introduces a novel strategy to bypass the non-differentiability of the brain by using a surrogate model and GP-based pseudo-gradients, enabling efficient optimization without explicit reward models.
3. Generalizable Optimization: The framework is agnostic to the specific backbone model (CNNs, Transformers) and target features (semantic, spectral, or subjective ratings), offering a "plug-and-play" recipe for brain-stimulus co-optimization.
4. Comprehensive Validation: Validated across three levels:
   • Simulation: Using synthetic EEG from proxy models.
   • Offline Search: Retrieving targets from a large image database.
   • Real-Time Human Experiments: Validating control over mental matching and emotion regulation.

4. Experimental Results

The authors conducted extensive experiments using the THINGS-EEG2 dataset and 10 human participants.

• Proxy Model Performance: Various architectures (AlexNet, ResNet, ViT, DINO) achieved competitive prediction accuracy (Pearson's $r \approx 0.16-0.18$ against noise ceiling), validating their use as effective black-box proxies.
• Interactive Search:
  • MindPilot consistently improved similarity scores over random sampling baselines.
  • EEG embeddings showed significant correlation with CLIP representations, confirming that semantic similarity is a valid proxy for neural alignment.
• Stimulus Generation:
  • Semantic Targeting: In a 10-round closed-loop experiment, MindPilot reduced L1 error significantly compared to random baselines and offline diffusion. It achieved semantic alignment (CLIP similarity) comparable to the theoretical upper bound (ATM-S decoder), despite not having access to the ground-truth EEG of the target image.
  • PSD Targeting: Successfully modulated neural spectral patterns, showing progressive alignment between image features and EEG PSD across iterations.
• Human Validation:
  • Mental Matching: Participants' subjective ratings of image similarity correlated strongly with the model's EEG-derived scores.
  • Emotion Regulation: The system successfully steered participants' emotional states toward positive valence (ratings increased from ~0.45 to ~0.60), demonstrating the ability to modulate affective states without explicit EEG feature targets.

5. Significance and Impact

• New Paradigm for BCIs: Shifts the focus from decoding (reading the brain) to modulation (writing to the brain) using naturalistic stimuli.
• Neuro-AI Interface: Establishes a functional link between perception and brain states, opening avenues for cognitive enhancement, neurorehabilitation, and bidirectional human-AI interaction.
• Overcoming Non-Differentiability: The "black-box proxy + pseudo-gradient" approach provides a robust solution for optimizing inputs to biological systems where gradients are unavailable.
• Clinical Potential: The ability to regulate emotional states and steer cognitive representations non-invasively suggests potential applications in treating mood disorders, attention deficits, and neurorehabilitation.

Limitations & Future Work:
The authors note challenges such as the "metamer" problem (different stimuli yielding similar EEG), inter-subject variability, and latency issues. Future work aims to incorporate subject-adaptive learning, extend to other modalities (e.g., fMRI), and optimize for real-time deployment.