BRAIN: Bias-Mitigation Continual Learning Approach to Vision-Brain Understanding

1. Problem Statement

The paper addresses a critical, previously overlooked issue in Vision-Brain Understanding (VBU): the inconsistency and bias in brain signals caused by human memory decay over time.

• Context: VBU models typically use fMRI data to reconstruct or retrieve visual stimuli. Large-scale datasets (e.g., Natural Scenes Dataset - NSD) are collected over months or years.
• The Core Issue: As participants are repeatedly exposed to visual stimuli across multiple sessions, their visual working memory decays. This leads to:
  • Reduced confidence and accuracy in recognizing previously seen images.
  • Weaker and more uncertain fMRI signals in later sessions compared to earlier ones.
  • Distribution Shift: The statistical distribution of brain signals ($p_t(x)$) changes over time ($t$), creating a bias where later data points contain poorer visual context.
• Impact on Existing Models: Traditional VBU methods treat all data points as equal and train on the entire dataset at once (static training) or combine sessions without accounting for this temporal drift. This leads to:
  • Representation Bias: The model learns from inconsistent data, causing feature distributions to scatter.
  • Catastrophic Forgetting: When models are updated incrementally (Continual Learning), they tend to forget knowledge from earlier, high-quality sessions when exposed to noisy, biased later sessions.
  • Performance Degradation: Empirical analysis shows that model accuracy drops significantly as training progresses through later sessions.

2. Methodology: The BRAIN Approach

The authors propose BRAIN (Bias-Mitigation Continual Learning), a framework designed to incrementally learn from brain signals while mitigating bias and preventing forgetting. The method consists of three main components:

A. Problem Formulation

The task is defined as Continual Vision-Brain Understanding (CVBU). The model $F$ learns a sequence of datasets $\mathcal{D} = \{\mathcal{D}_1, \dots, \mathcal{D}_T\}$, where each $\mathcal{D}_t$ corresponds to a specific data collection session. The goal is to align fMRI features ($z_t$) with visual centroids ($c_t$) while managing the shifting distribution $p_t(x)$.

B. De-bias Contrastive Learning (DCL)

To address the distribution shift caused by memory decay, the authors introduce a weighted contrastive loss.

• Bias Modeling: They observe that the bias factor $w_t$ correlates with the participant's response accuracy ($r(t)$) in that session. As accuracy drops, uncertainty rises, and the signal becomes noisier.
• Weighting Mechanism: They define a bias weight $w_t = e^{1-r(t)}$.
  • If $r(t)$ is low (high uncertainty), $w_t$ increases, assigning a higher penalty to the loss for that sample to force the model to pay attention to the bias.
  • If $r(t)$ is high, $w_t$ is lower.
• Loss Function: The standard contrastive loss is modified to include this weight:
  $$\mathcal{L}_C = -\log \frac{\exp(z_t \cdot c_t)}{\sum_{z'} \exp(z' \cdot c_t) \cdot e^{1-r(t)}}$$
  This effectively re-weights the gradient updates to counteract the bias introduced by later, less reliable sessions.

C. Angular-based Forgetting Mitigation (AFM)

To prevent catastrophic forgetting without relying on rigid weight regularization (which can hinder adaptability) or large replay buffers (which raise privacy concerns), the authors propose a knowledge distillation strategy based on angular distance.

• Limitation of Euclidean Distance ($L_2$): Standard distillation uses $L_2$ norm to minimize the distance between old and new feature vectors. However, $L_2$ is sensitive to magnitude changes (scale), which can occur naturally as the model adapts to new data distributions.
• Angular Metric: The authors propose measuring the divergence between the old feature representation ($z_{t-1}$) and the new one ($z_t$) using the angle between them, rather than their Euclidean distance.
  $$\mathcal{D}_f(z_{t-1}, z_t) = \left\| 1 - \frac{z_{t-1}}{\|z_{t-1}\|} \cdot \frac{z_t}{\|z_t\|} \right\|_2^2$$
• Benefit: This metric is robust to scale variations and noise. It ensures the model retains the direction (semantic meaning) of the features learned in previous sessions while allowing the magnitude to adapt to new data.

D. Total Objective

The final training objective combines the De-bias Contrastive Loss and the Angular-based Forgetting Mitigation loss:
$$\mathcal{L} = \mathcal{L}_C + \lambda_{CL} \mathcal{L}_{CL}$$

3. Key Contributions

1. Discovery of Signal Bias: The paper is one of the first to statistically and experimentally demonstrate that human memory decay causes a systematic bias in fMRI signals across sessions, negatively impacting VBU models.
2. Novel Continual Learning Framework: Introduced BRAIN, a continual learning approach specifically tailored for the non-stationary nature of brain data, allowing incremental updates without waiting for full dataset collection.
3. De-bias Contrastive Learning (DCL): A new loss function that dynamically weights samples based on participant response accuracy to mitigate the impact of noisy, biased data.
4. Angular-based Forgetting Mitigation (AFM): A novel distillation loss that uses angular distance instead of Euclidean distance to preserve knowledge, proving more robust to feature scale shifts in continual learning.
5. State-of-the-Art Performance: Demonstrated that the proposed method outperforms existing non-continual and continual learning baselines.

4. Experimental Results

The approach was evaluated on the Natural Scenes Dataset (NSD) involving 8 participants and over 73,000 fMRI trials.

• Benchmarks: Evaluated on two tasks: Brain-to-Image Retrieval and Image-to-Brain Retrieval across various continual learning settings (e.g., training on 20 sessions, updating every 2, 5, or 10 sessions).
• Performance Gains:
  • vs. Non-CL: The BRAIN approach significantly outperformed standard training (W/o CL), which suffered severe performance drops (e.g., a 43% drop in accuracy) due to forgetting.
  • vs. SOTA CL Methods: It surpassed established continual learning baselines like LwF (Learning without Forgetting) and PLOP.
  • Specific Metrics: In the (20, 2) setting for Brain-to-Image retrieval, BRAIN achieved 57.97% accuracy, compared to 18.06% for non-CL and 54.7% for LwF.
• Ablation Studies:
  • Removing DCL resulted in a 1-4% performance drop, confirming the necessity of bias mitigation.
  • Replacing AFM with $L_2$ regularization caused a 1-3% drop, validating the superiority of the angular metric.
  • Rehearsal vs. No Rehearsal: The rehearsal-free approach (using AFM) performed better than rehearsal-based methods, suggesting that storing only 10% of data is insufficient for this specific task.
• Comparison with SOTA: When compared to the state-of-the-art MindEye model (using the same architecture but different loss), BRAIN achieved higher retrieval scores (95.3% vs. 90.1% for Brain-to-Image).

5. Significance

• Biological Plausibility: The method aligns better with the biological reality of human learning, where memory and perception evolve over time, rather than assuming static data distributions.
• Efficiency and Privacy: By avoiding large replay buffers (rehearsal), the method addresses data privacy concerns inherent in fMRI data while remaining computationally efficient.
• Generalizability: The insights regarding distribution shifts due to human factors (memory decay) and the use of angular metrics for distillation offer valuable lessons for other human-in-the-loop machine learning tasks.
• Advancement in Neuro-AI: This work bridges the gap between neuroscience (understanding memory decay) and AI (designing robust continual learning algorithms), pushing the boundaries of what is possible in decoding brain activity.