💻 computer science

Rethinking Brain Decoding with CLIP: The Role of Adversarial Robustness

This paper demonstrates that utilizing adversarially robust variants of CLIP, rather than standard models, significantly enhances fMRI-based brain decoding performance by promoting stable, perceptually structured representations that better align with neural activity.

Original authors: Byeongseo Bok, Futa Waseda, Jun Liu, Isao Echizen

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Byeongseo Bok, Futa Waseda, Jun Liu, Isao Echizen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Reading Minds with AI

Imagine scientists are trying to build a "mind-reading" machine. They use an MRI scanner to look at a person's brain while they look at a picture. The goal is to figure out exactly what picture the person is seeing just by looking at the brain activity.

To do this, they need a translator. The brain speaks in "neural signals" (fMRI data), but we need to translate that into something we understand, like a description of an image. For a long time, scientists have used a very popular AI translator called CLIP. CLIP is great at understanding the connection between pictures and words.

The Problem: Even though CLIP is popular, it's not a perfect translator for the human brain. The paper suggests that CLIP is like a student who is really good at passing a specific test by memorizing "tricks" or "shortcuts" (like noticing that a picture of a dog usually has grass in the background), rather than truly understanding the core concept of the dog. The human brain, however, doesn't rely on those shortcuts; it sees the dog itself. Because the AI and the brain are looking at the picture in slightly different ways, the translation isn't perfect.

The Solution: The "Stress-Test" Translator

The authors asked a simple question: What if we use a version of the AI that has been "stress-tested" to ignore those tricks?

In the world of AI, there is a technique called Adversarial Training. Imagine you are teaching a student to recognize a stop sign.

Standard Student (CLIP): You show them a stop sign. They learn it. But if you put a tiny, invisible sticker on it that looks like a speed limit sign to a computer, they might get confused and think it's a speed limit sign. They rely on fragile details.
Stress-Tested Student (Robust CLIP): You show them the stop sign, but you also show them thousands of versions with weird stickers, blurs, and distortions. You force them to learn the essential shape of the sign, ignoring the messy background or tiny tricks. They become "robust."

The paper argues that this "Stress-Tested Student" (Adversarially Robust CLIP) actually speaks the same language as the human brain better than the standard student does.

What They Did (The Experiment)

The researchers set up a fair race. They kept everything exactly the same—the brain scanner data, the computer code, and the training method. The only thing they changed was the "translator" (the target representation).

Team A: Used the standard CLIP model.
Team B: Used the "Stress-Tested" versions (called FARE and TeCoA).

They tested this on two different sets of brain data (datasets) to see who could guess the image from the brain scan more accurately.

The Results: The Stress-Tested Team Wins

The results were clear and consistent:

Better Guessing: The "Stress-Tested" models were much better at guessing which image a person was looking at based on their brain activity. In one test, the accuracy jumped by 13%, which is a huge deal in this field.
Better Alignment: When they measured how closely the AI's internal map matched the brain's map, the robust models were much closer. It's like the AI and the brain were finally on the same page.
Zero-Shot Success: Even when they showed the brain images the AI had never seen before, the robust models still guessed correctly more often than the standard ones.

The "Why": Looking at the Map

The researchers didn't just stop at "it works better." They wanted to know why. They used a tool called an Attribution Map, which is like a heat map showing which parts of an image the AI is paying attention to.

Standard CLIP: The heat map was a bit scattered. It seemed to be looking at the whole picture, including background details that might be "tricks."
Robust CLIP: The heat map was more focused and structured. It ignored the background "noise" and focused on the main object, just like a human does.

The most interesting finding was that the two models were looking at the images in completely different ways. They didn't just tweak the same features; they reorganized how they saw the world. The robust model stopped relying on the "shortcuts" that the standard model used, and that shift is what made it align better with the human brain.

The Takeaway

The paper concludes that if you want to decode human brain activity, choosing the right AI model matters more than you might think.

You don't need to build a brand-new machine or change the brain scanner. You just need to swap the standard AI translator for one that has been "stress-tested" to be robust against tricks. By doing this, the AI stops looking for easy shortcuts and starts seeing the world the way our brains do, leading to much better mind-reading results.

In short: To read a human mind, use an AI that has learned to ignore the distractions and focus on the truth.

Problem Statement

Brain decoding aims to infer stimulus-related representations from brain signals, typically by mapping fMRI responses to the latent embeddings of computational models. Recently, the Contrastive Language-Image Pre-training (CLIP) model has become a standard target representation due to its rich vision-language embedding space. However, aligning fMRI signals with standard CLIP representations remains challenging.

The authors argue that standard CLIP is optimized for web-scale image-text contrastive learning rather than neural alignment. Consequently, its embeddings may capture "non-robust" or shortcut-driven features that are statistically predictive for image-text matching but are not faithfully reflected in human brain activity. Previous works have attempted to improve decoding by adding biological constraints (e.g., visual attention mechanisms) or enforcing geometric similarity between embeddings and fMRI responses. This paper posits that the issue lies not just in the pipeline, but in the intrinsic properties of the target representation itself. Specifically, the authors hypothesize that adversarially robust representations, which suppress non-robust features and rely more on stable, perceptually structured cues, may align better with the invariance observed in human neural representations.

Methodology

The study employs a controlled experimental design where the fMRI decoder and the entire training pipeline are kept fixed, while only the target representation model is varied.

Datasets:
- NSD (Natural Scene Dataset): A large-scale vision-fMRI dataset containing 73,000 images and fMRI data from 8 subjects. The authors evaluate on subjects 1, 2, 5, and 7.
- GOD (Generic Object Decoding Dataset): Designed to test zero-shot generalization with non-overlapping train-test semantics, containing 1,250 images and data from 5 subjects.
Target Models:
- Baseline: Standard CLIP (ViT-L/14 backbone).
- Robust Variants: Two adversarially trained CLIP variants:
  - TeCoA: A robust variant where the visual encoder is adversarially fine-tuned under text-guided supervision.
  - FARE: A robust variant using unsupervised adversarial fine-tuning without text supervision.
- The authors evaluate different levels of adversarial strength (e.g., FARE-2/4, TeCoA-2/4) based on the perturbation budget ( $\epsilon$ ).
Decoder Architecture:
- A linear decoder maps fMRI voxel data to predicted embeddings.
- Training Objectives: The study utilizes two objectives:
  1. InfoNCE Loss (Contrastive): Optimizing the alignment between predicted and target embeddings using a contrastive loss.
  2. Ridge Regression: A standard baseline in brain decoding.
Evaluation Metrics:
- Decoding Performance: Top- $k$ accuracy (Recall@k) for fMRI-image retrieval on both NSD and GOD datasets.
- Neural Alignment: Pearson correlation, Cosine similarity, and Representational Similarity Analysis (RSA) between fMRI responses and target representations.
- Attribution Analysis: Using the framework from Palazzo et al. [40], the authors compute pixel-level attribution maps to analyze which image features the decoders rely on. Metrics include normalized entropy, IoU of top- $k$ attributed regions, and Spearman rank correlation between attribution maps.

Key Results

The empirical results demonstrate that replacing standard CLIP with adversarially robust variants consistently improves brain decoding performance and neural alignment.

Improved Alignment and Retrieval:
- On the NSD dataset, robust variants (FARE and TeCoA) achieved higher alignment scores (Pearson correlation, Cosine similarity, RSA) compared to standard CLIP. TeCoA-4 consistently achieved the highest alignment scores.
- In fMRI-image retrieval, robust decoders outperformed the standard CLIP baseline. Notably, FARE-4 improved Top-1 accuracy by 13% relative to the CLIP baseline under contrastive decoding.
- These improvements held true across both contrastive learning and ridge regression objectives, suggesting the target representation choice is more critical than the training objective.
Zero-Shot Generalization (GOD Dataset):
- The improvements were reproducible on the GOD dataset, which enforces strict zero-shot conditions. Robust decoders maintained higher retrieval accuracy and zero-shot fMRI-text classification performance compared to standard CLIP.
- TeCoA decoders generally outperformed FARE decoders in zero-shot text classification tasks.
Attribution Analysis:
- Spatial Structure: While all models exhibited high global spatial diffusion (high entropy), robust variants showed stronger scale-dependent structural variation. Their entropy decreased as spatial resolution became coarser, indicating more structured attribution patterns compared to the stable entropy of standard CLIP.
- Feature Disagreement: There was consistently low agreement between standard CLIP and robust variants. Top-k Intersection-over-Union (IoU) for highly attributed pixels was extremely low (below 0.1), and Spearman rank correlations were weak or negative. This suggests that adversarial training does not merely refine feature usage but induces a systematic reorganization of feature importance, relying on structurally distinct image features.

Contributions

The paper makes the following specific contributions:

Demonstration of Robustness Benefits: It shows that adversarially trained CLIP variants (FARE and TeCoA) improve the alignment between fMRI responses and model representations, yielding significantly higher fMRI-image retrieval accuracy on the NSD dataset.
Zero-Shot Validation: It validates that these improvements generalize to zero-shot settings on the GOD dataset, confirming that the gains are not due to dataset-specific overfitting.
Attribution Insights: Through attribution analysis, it reveals that robust models exhibit low agreement with standard CLIP in feature usage, producing spatially concentrated and distinct patterns, thereby suggesting that adversarial robustness reorganizes visual feature importance.

Significance and Claims

The authors conclude that the choice of target representation is a critical factor in neural decoding pipelines. They argue that adversarial robustness serves as a useful inductive bias for improving the alignment between computational models and brain activity.

The paper claims that because human neural representations are relatively invariant to small perturbations and less reliant on non-robust cues, models trained to suppress such cues (via adversarial training) naturally produce representations that are more compatible with fMRI signals. This finding suggests that future brain decoding efforts should consider the intrinsic properties of the target model, such as robustness, rather than focusing solely on architectural modifications or pipeline engineering. The work highlights that robustness affects not only predictive accuracy but also the fundamental way visual features are utilized in decoding brain activity.