💻 computer science

Lost in Visual Translation: A VLM-Assisted Perceptual-Semantic Coherence Framework for EEG-to-Image Reconstruction

This paper proposes a novel BCI-aware framework that utilizes Vision Language Models to evaluate EEG-to-image reconstructions by distinguishing perceptual fidelity from semantic recoverability, introducing the BCI-Coherence Score (BCS) to overcome the limitations of traditional pixel-based and representation metrics.

Original authors: Sukriti Tiwari, BHVSP Subrahmanyam, Nidhi Goyal, Sai Amrit Patnaik

Published 2026-07-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sukriti Tiwari, BHVSP Subrahmanyam, Nidhi Goyal, Sai Amrit Patnaik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a picture you saw in a dream to a friend, but your brain is sending the message through a very fuzzy, static-filled radio. The picture you get back is blurry, maybe a little distorted, and missing some details. Now, imagine you have a robot judge trying to grade how well your friend's drawing matches the original dream.

For a long time, these robot judges have been using the wrong rulebook. They were built to grade high-definition photos, so if your drawing was a little blurry, they would give it a terrible score, even if your friend clearly drew a "dog" instead of a "cat." Or worse, they might give a high score to a drawing that looks super crisp and pretty but is actually a picture of a "pizza" when you meant to draw a "dog."

This is exactly the problem researchers Sukriti Tiwari and her team at Mahindra University found when looking at EEG-to-image reconstruction. This is the technology that tries to turn brain waves (EEG) back into pictures. Because brain waves are weak and noisy, the resulting images are often messy. The team analyzed 6,855 pairs of original images and their brain-wave reconstructions and discovered that the standard robot judges were failing in two funny but frustrating ways:

The "Harsh" Judge: This judge gets angry at any blur. It gives a low score to a drawing that is a bit fuzzy but still clearly shows the right object. It's like failing a student for having a messy handwriting when they got the right answer.
The "Blind" Judge: This judge is fooled by pretty pictures. It gives a high score to a crisp, beautiful image that is actually the wrong object entirely. It's like giving an A+ to a student who wrote a perfect essay about the wrong topic.

The paper argues that we need to stop asking, "Does this look exactly like the photo?" and start asking, "Can we still figure out what the object is, even if it's messy?"

To fix this, the team didn't just build one new robot; they built a panel of four expert art critics (using powerful Vision-Language Models called InternVL3, SAIL-VL, OLA-7B, and Ovis2-8B). Instead of just giving a single number, these critics were asked 12 specific questions about each pair of images:

Perceptual questions: "Is the shape roughly right?" "Are the colors similar?" "Is it too noisy to see?"
Semantic questions: "Is it the right type of animal?" "Is it the right number of objects?" "Does it make sense in the scene?"

The team found that these four critics didn't always agree perfectly. Sometimes one thought a shape was "somewhat" right while another said "no." But by taking the middle ground (the median) of their answers, they created a new, super-smart scoring system called the BCI-Coherence Score (BCS).

Think of BCS as a "translator" that learned from the four experts. Instead of needing to run all four heavy-duty robots for every new image, you can just use this lightweight BCS translator. It looks at the blurry brain-wave image and the original picture and predicts what the expert panel would have said.

The results were promising. When the team tested this new translator, it was very good at guessing the experts' opinions on whether the meaning was preserved (with a correlation of 0.850) and the visual look was preserved (with a correlation of 0.700).

To make sure this wasn't just a computer trick, they also asked 18 human volunteers to do the grading. The humans found something interesting: judging just the "look" (perception) was really hard and inconsistent (humans disagreed a lot). But when they judged the "meaning" (semantic) or the "whole vibe" (coherence) together, they agreed much more strongly. The BCS score matched this human behavior, suggesting that for brain-wave images, it's better to care about whether the idea survived the noise than whether the pixels match perfectly.

In short, the paper suggests that for decoding brain waves into images, we need a new kind of ruler—one that is tolerant of fuzziness but strict about the meaning. The team has made their new ruler, the BCI-Coherence Score, available for others to use, helping future researchers stop getting fooled by pretty but wrong pictures or harshly punishing messy but correct ones.

Technical Summary: Lost in Visual Translation

Problem Statement
The evaluation of EEG-to-image reconstruction faces a fundamental mismatch between standard computer vision metrics and the specific characteristics of neural decoding. EEG signals are non-invasive, noisy, and spatially coarse, resulting in reconstructions that are often blurry, distorted, and low-detail. Current evaluation practices rely on metrics designed for generic image restoration or generation (e.g., SSIM, PSNR, LPIPS, CLIP similarity). The authors argue these metrics fail in the EEG regime for two complementary reasons:

Harshness: Metrics penalize reconstructions that are visually degraded but semantically recoverable (i.e., the intended object is recognizable despite poor fidelity).
Semantic Blindness: Metrics assign high scores to visually plausible images that depict incorrect semantic content (e.g., the wrong object category).

Existing studies show that automatic scores often diverge from human judgment, yet recent EEG-to-image research continues to rely on these protocols despite the unique ambiguity of EEG compared to modalities like fMRI.

Methodology
The paper proposes a BCI-aware Perceptual-Semantic Coherence Framework to address these limitations. The methodology proceeds in three stages:

Empirical Analysis of Current Metrics:
The authors analyzed 6,855 ground-truth/reconstruction pairs from four datasets (ATM, ENIGMA, BrainVis, DreamDiffusion). They introduced two probes:
- Caption-Level Semantic Probe: Using BLIP to generate captions and SBERT to measure semantic similarity, they quantified "Caption Harshness" (metrics penalizing semantically close outputs) and "Caption Blind-Spot Rates" (metrics rewarding semantically distant outputs).
- Controlled Perceptual Degradation: They applied six types of degradations (blur, noise, downsampling, JPEG, color attenuation, shift) to ground-truth images to test metric sensitivity.
- Findings: Pixel-level metrics (MSE, SSIM) showed near-zero correlation with semantic consistency. Representation metrics (OpenCLIP, DreamSim) correlated better with semantics but still conflated perceptual and semantic errors, failing to distinguish between recoverable meaning and visual fidelity.
VLM-Assisted Annotation Protocol:
To separate perceptual from semantic recoverability, the authors utilized four open-weight Vision-Language Models (VLMs): InternVL3, SAIL-VL, OLA-7B, and Ovis2-8B.
- Structured Questions: Each image pair (ground truth vs. reconstruction) was evaluated via 12 structured questions: 6 perceptual (layout, shape, texture, color, artifacts, holistic recoverability) and 6 semantic (category, specific identity, function, quantity, scene, holistic semantic recoverability).
- Scoring Scale: Answers were mapped to a three-level scale: no (0), somewhat (0.5), and yes (1).
- Consensus Aggregation: The authors computed Tolerant Perceptual Alignment Scores (T-PAS) and Tolerant Semantic Alignment Scores (T-SAS) by aggregating VLM responses. They used the median of the four VLMs to preserve partial disagreement rather than forcing a hard majority label.
- Reliability: Analysis showed moderate agreement (Fleiss' $\kappa$ = 0.363) overall, with higher agreement on concrete attributes (shape, function) than on holistic recoverability judgments.
BCI-Coherence Score (BCS) Distillation:
To avoid the computational cost of running multiple VLMs for every evaluation, the authors distilled the multi-VLM consensus into a lightweight image-pair regressor, the BCS.
- Architecture: BCS uses frozen visual encoders (SigLIP, CLIP, DINOv3) to extract features from the reference and reconstruction. It constructs pairwise comparison features (concatenation, difference, element-wise product, cosine similarity) and feeds them into a residual MLP.
- Training: The model is trained to predict the median consensus scores ( $z_{i,q}$ ) for each question. A weighted loss function ( $L$ ) prioritizes high-consensus VLM labels and down-weights ambiguous instances.
- Output: BCS predicts aggregate T-PAS and T-SAS scores directly from an image pair.

Key Results

Metric Failure: Conventional metrics showed weak semantic alignment (e.g., MSE $\rho_{cap} \approx -0.01$ ) and inconsistent sensitivity to specific degradation types.
BCS Performance: The distilled BCS model (4-teacher continuous ensemble) achieved strong correlation with the VLM consensus on held-out data:
- T-PAS: MAE = 0.079, Pearson $r$ = 0.700.
- T-SAS: MAE = 0.082, Pearson $r$ = 0.850.
Human Validation: In a targeted human study involving 18 annotators, coherence judgments (evaluating perceptual and semantic alignment jointly) showed high reliability ( $\kappa = 0.882$ , $\alpha = 0.882$ ). In contrast, isolated perceptual judgments were highly unreliable ( $\kappa = 0.110$ ), while semantic judgments showed moderate reliability ( $\kappa = 0.483$ ). This supports the framework's design of evaluating perceptual and semantic axes jointly.

Significance and Claims
The paper claims that EEG-to-image evaluation must shift from generic visual similarity to perceptual-semantic recoverability. The proposed framework offers:

A Diagnostic Protocol: It explicitly identifies and quantifies "harshness" and "blind spots" in current metrics.
A Robust Evaluation Signal: The BCS provides a reusable, compact evaluator that separates tolerable signal-induced degradation from semantic loss, aligning more closely with human judgment than existing metrics.
Validation of Joint Coherence: The human validation results suggest that while humans struggle to consistently judge low-level visual fidelity in noisy EEG reconstructions, they can reliably judge the joint coherence of the reconstruction, validating the T-PAS/T-SAS design.

The authors conclude that BCS provides a necessary evaluation signal for future benchmarking, though they note that extending this approach to fMRI and cross-subject settings remains an open direction.

Technical Summary: Lost in Visual Translation

More like this