💻 computer science

Dual-Stream EEG Decoding for 3D Visual Perception

This paper presents a bio-inspired dual-stream EEG decoding model that successfully reconstructs 3D visual perceptions by separately decoding object identity and spatial orientation through ventral and dorsal pathways, utilizing circular regression and EEG-conditioned diffusion for accurate 3D reconstruction.

Original authors: Ninon Lizé Masclef, Taisija Demcenko, Antonella Catanzaro, Nataliya Kosmyna

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ninon Lizé Masclef, Taisija Demcenko, Antonella Catanzaro, Nataliya Kosmyna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your brain is a highly sophisticated factory with two specialized assembly lines working together to understand what you see. One line figures out what an object is (a banana, a tiger, a face), and the other line figures out where it is and how it's turning in space.

For a long time, computer scientists tried to build artificial brains that only had one of these lines. They were great at recognizing objects but terrible at understanding how those objects move or rotate in 3D space. This paper introduces a new "factory" design that mimics the human brain's two-line system to decode what people are seeing just by reading their brainwaves (EEG).

Here is a breakdown of their work using simple analogies:

1. The Problem: The "Flat" Brain

Think of standard computer vision models like a person looking at a 3D sculpture through a flat piece of glass. They can tell you it's a statue of a tiger, but if the tiger starts spinning, the computer gets confused. It struggles to understand the 3D shape and the rotation.

The researchers wanted to fix this by building a system that doesn't just "see" the object, but also tracks its spin, just like your brain does.

2. The Solution: A Two-Track System

The team built a "Dual-Stream" model, which is like hiring two different experts to listen to the brain's electrical signals:

The "What" Expert (Ventral Stream): This part of the model focuses purely on identity. Is it a banana? Is it a panda? It ignores the rotation and just says, "That's a banana."
The "Where/How" Expert (Dorsal Stream): This part focuses on geometry and movement. It doesn't care if it's a banana or a tiger; it cares that the object is currently tilted at a 45-degree angle and spinning.

By separating these tasks, the model can handle the complexity of a spinning 3D object much better than a single "do-it-all" model.

3. The Experiment: The VR Spin

To test this, the researchers put 11 people in a Virtual Reality headset.

The Stimulus: The participants watched 78 different 3D objects (like fruit, animals, and sports balls) spin around continuously for 8 seconds.
The Recording: While they watched, the researchers recorded their brainwaves using a cap with 64 sensors (like a high-tech swim cap).
The Goal: Could the computer look at those brainwaves and guess both what the person was looking at and how much it was rotated?

4. The Results: Reading the Mind's Map

The results were impressive:

Identity: The model could correctly guess the object category (e.g., "It's a tiger") about 68% of the time, even while it was spinning.
Rotation: It could guess the angle of rotation with an error of only 10 to 11 degrees. That's like looking at a clock and guessing the time within about 20 minutes of the actual hour.
3D Reconstruction: The coolest part? They used these two pieces of information (the object name and the angle) to feed a "magic generator" (a diffusion model). This generator created a 3D video of the object spinning, based only on the person's brainwaves. It wasn't perfect, but it clearly looked like the object the person was seeing.

5. The Surprise: It's Not Just One Line

The researchers expected that the "What" part of the brain would only use the "Ventral" sensors (usually at the back of the head) and the "Where" part would only use the "Dorsal" sensors (top and sides).

The discovery: It's not that simple.
Think of the brain's sensors like a choir. The researchers found that for the computer to work well, it needed a dynamic mix of singers from different sections (back, top, and even motor-related areas) at different times.

Sometimes the "motor" sensors (which usually help you move your hands) were singing loudly to help figure out the rotation.
The "What" and "Where" lines weren't working in isolation; they were chatting and swapping information over time.

Summary

This paper shows that if you want a computer to understand 3D objects the way humans do, you can't just give it one brain. You need to give it two specialized teams that work together. By splitting the job into "What is it?" and "How is it spinning?", they were able to read brainwaves and reconstruct 3D videos of rotating objects.

Most importantly, they proved that the brain doesn't use a single, static switch to do this; it uses a complex, time-varying dance of signals across different parts of the brain to make sense of the 3D world.

Technical Summary: Dual-Stream EEG Decoding for 3D Visual Perception

Problem Statement

Current neural networks for visual perception, particularly Convolutional Neural Networks (CNNs), struggle with 3D spatial and geometric understanding. While they align with the biological ventral visual pathway ("what" system) for object recognition, they exhibit biases toward texture over shape and fail to maintain representational consistency across viewpoints. This limitation extends to brain decoding: existing models often struggle to differentiate viewpoints within object classes and extract orientation features from EEG signals. Furthermore, standard brain decoding approaches typically treat 3D perception as a single-pathway problem, failing to leverage the distinct but interacting geometric structures of the biological ventral ("what") and dorsal ("where/how") pathways.

Methodology

The authors propose a bio-inspired dual-stream decoding architecture that mirrors biological vision by implementing separate modules for object identity and spatial orientation. The methodology is structured around three complementary tasks:

1. Experimental Design and Dataset

Participants: 11 subjects viewed 78 distinct 3D objects (across six semantic categories) in an immersive VR environment (Meta Quest 2) to leverage binocular disparity.
Stimuli: Objects underwent continuous 360° rotation over 8 seconds ( $d\theta/dt = \pi/4$ rad/s).
Data Collection: Neural activity was recorded via a 64-channel ANT Neuro EEG system (512 Hz). Each subject completed 468 trials (78 objects $\times$ 6 repetitions).

2. Model Architectures

The study benchmarked three architectures (EEGNet, EEGConformer, CTNet) and developed a specific dual-stream pipeline:

Ventral Stream (Object Identity): A classification task using EEGNet to predict one of six semantic categories. The best-performing model (EEGNet for Subject 5) achieved up to 68% accuracy.
Dorsal Stream (Spatial Orientation): A circular regression task adapted from EEGNet. Instead of a standard classification head, it outputs a 2D unit vector $[\sin(\theta), \cos(\theta)]$ to handle periodic boundary conditions naturally. The loss function computes the squared chord length between predicted and target vectors.
3D Reconstruction Pipeline: The authors developed an EEG-conditioned multiview diffusion model based on MVDream.
- Conditioning: A frozen EEGNet object classifier generates a 512-dimensional embedding (projected to 1024 dimensions) serving as the "object identity" token.
- Angle Injection: A separate angle encoder predicts rotation angles, which are injected via an angle-aware timestep embedding.
- Generation: A diffusion U-Net uses cross-attention where multi-view latents act as queries and the projected EEG embeddings act as keys/values. The model is fine-tuned using Low-Rank Adaptation (LoRA).

3. Interpretability Analysis

To investigate the functional specialization of the pathways, the authors applied Gradient-weighted Class Activation Mapping (Grad-CAM) to the convolutional layers. They defined three anatomically motivated channel subsets:

Ventral: Temporal and occipital channels (T7, T8, O1, O2, etc.).
Dorsal: Parietal and central channels (Pz, Cz, P1, P2, etc.).
Motor/Premotor: Central and frontal motor regions (C3, C4, FC3, FC4, etc.).

Key Results

Classification and Regression Performance

Object Classification: EEGNet achieved the highest mean accuracy across subjects, with Subject 5 reaching 68% accuracy (significantly above the 16.7% chance level).
Angle Regression: All models demonstrated robust performance with a Mean Absolute Error (MAE) of 10–11° using 1-second time windows.
3D Reconstruction: The EEG-conditioned multiview diffusion model achieved an SSIM of 0.856 ± 0.038 and LPIPS of 0.275 ± 0.061 on the held-out test set, indicating good generalization and geometric coherence across viewpoints.

Interpretability Findings

Contrary to the hypothesis of a static "ventral-only" dominance for object decoding, Grad-CAM analysis revealed:

Dynamic Integration: Successful decoding relies on a time-resolved integration across ventral, dorsal, and motor-related channels rather than a static binary readout.
Motor Involvement: Object decoders exhibited significant motor channel activation, and angle decoders benefited from early motor bias.
Temporal Structure: The dissociation between object and angle models suggests a partial disentangling of invariant and equivariant components. Object decoders were penalized for overweighting ventral channels early and dorsal channels late, suggesting that robust 3D representation arises from a balanced, temporally structured integration across all three subspaces.

Significance and Claims

The paper claims three primary contributions to the fields of neural decoding and bio-inspired 3D vision:

Novel Architecture: It presents the first dual-stream architecture for 3D brain decoding that explicitly separates viewpoint-tolerant object recognition ("what") from viewpoint-dependent spatial transformation ("where/how"), achieving robust performance in both classification and angular regression.
Neurophysiological Insight: It provides interpretable evidence that 3D perception in EEG signals involves dynamic interactions between dorsal, ventral, and motor pathways. The findings challenge the notion of static ventral dominance, suggesting instead that geometric representation in the brain (and its decoding) relies on temporally structured integration across these systems.
Feasibility of Neural-to-3D Generation: The work establishes the feasibility of generating 3D objects directly from non-invasive EEG signals using multiview diffusion models conditioned on neural activity, preserving both semantic identity and geometric consistency across viewpoints.

The authors conclude that while their approach successfully decodes 3D shape perception, further investigation is required to quantify the benefits of biological versus purely geometric inductive biases by comparing their results with steerable CNNs and equivariant architectures.