What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval

Technical Summary: Source Attribution for Non-Invasive Brain-to-Language Retrieval

Problem Statement
Non-invasive neural language decoding has advanced rapidly with pretrained speech and language models, yet reported accuracy often conflates genuine stimulus-evoked neural evidence with non-neural structural artifacts. The paper identifies three primary sources of inflated performance:

1. Decoder Priors and Evaluation Shortcuts: Models may exploit variable input durations, padding boundaries, or mask patterns rather than neural content.
2. Embedding-Based Metric Limitations: Metrics like BERTScore reward semantic similarity or fluency, allowing signal-blind outputs (e.g., generic fillers) to appear successful even when they fail to recover specific stimulus content.
3. Data Leakage: Improper train-test splits (e.g., by subject rather than stimulus identity) allow repeated stimulus content to leak into evaluation sets.

The core methodological challenge is attribution: distinguishing whether a reported gain stems from decodable linguistic information in the neural signal or from these structural and evaluation artifacts.

Methodology
The authors reframe the problem using a stimulus-locked MEG-to-audio retrieval framework as an auditing tool. Instead of open-ended text generation, the task involves retrieving a time-aligned audio window from an explicit candidate pool for each neural window. This setup makes chance levels, candidate sets, and rank errors directly observable.

The methodology decomposes performance into three distinct sources:

1. Structural Shortcuts: Controlled by enforcing fixed-duration windows and stimulus-identity splits (ensuring the same audio content does not appear in both training and test sets).
2. Window-Level Stimulus-Locked Evidence: Measured via independent fixed-window retrieval, isolating the discriminability of single neural windows against the audio pool.
3. Cross-Window Contextual Aggregation: Addressed via a new inference-time intervention called Group Context Bias (GCB).

Key Contributions

• Quantifying Structural Nuisances: The authors demonstrate that under variable-length decoding, replacing real MEG with Gaussian noise yields 66.3% Rank@1 (R@1) accuracy due to sequence-length shortcuts. Enforcing fixed-duration windows and stimulus-identity splits collapses this noise baseline to near-chance levels, effectively isolating the non-neural structural leakage.
• Isolating Local Evidence: Under these strict controls, independent fixed-window retrieval recovers measurable MEG–audio discriminability (e.g., 44.4% R@1 on the Gwilliams dataset), proving that window-level neural signals are decodable. However, error analysis reveals that 95.7% of Top-1 errors involve selecting a candidate from the wrong sentence, indicating a bottleneck in sentence-level competition rather than within-sentence ambiguity.
• Auditing Contextual Aggregation (GCB): The paper introduces GCB, an additive logit bias applied at inference time. GCB pools evidence across windows belonging to the same query sentence to identify the most supported "sentence buckets" and applies a soft bias to candidates within those buckets. Crucially, GCB leaves the frozen neural encoder, audio embeddings, and candidate pool unchanged, allowing the contextual contribution to be measured independently of local evidence.
• Diagnostic Controls: The study employs rigorous controls to validate attribution:
  • Oracle Diagnostic: Restricting candidates to the ground-truth sentence raises R@1 from 44% to 88%, confirming the existence of "contextual headroom."
  • Perturbation Tests: Randomizing sentence groupings eliminates the GCB gain, proving the effect relies on coherent sentence structure.
  • Evidence Attenuation: Mixing real MEG with stimulus-mismatched noise shows that GCB gains vanish as local stimulus-locked evidence approaches chance, confirming GCB amplifies existing evidence rather than generating it.
  • Negative Controls: On the Brennan EEG dataset (where local retrieval is near chance), GCB yields no positive gain, further supporting that contextual aggregation requires reliable local evidence.

Experimental Results

• Structural Leakage: Variable-length decoding with Gaussian noise achieves 66.3% R@1; fixed-length controls reduce this to 1.53% (near random).
• Local vs. Contextual Performance:
  • Gwilliams (MEG): Local baseline R@1 = 44.4%. With GCB, R@1 increases to 52.1% (+7.7 pp).
  • MOUS (MEG): Local baseline R@1 = 21.7%. With GCB, R@1 increases to 28.6% (+6.9 pp).
  • Brennan (EEG): Local baseline R@1 < 1%. GCB provides no improvement, confirming the method's dependence on signal quality.
• Error Correction: GCB primarily corrects "near-miss" errors where the correct sentence bucket was not selected by the local model but is supported by cross-window consistency. It recovers approximately 18% of the theoretical headroom exposed by the oracle diagnostic.
• Robustness: The gains are stable across different temporal backbone architectures (Dense-TCNN vs. Exp-dilated CNN) and transfer across datasets without retuning.

Significance and Claims
The paper argues that non-invasive neural language decoding should be treated as an attribution problem rather than a simple performance benchmark. The authors claim that:

1. Performance is Multi-Source: Reported scores often reflect a mixture of structural shortcuts, local neural evidence, and contextual aggregation.
2. Retrieval as an Audit: Stimulus-locked retrieval provides a controlled endpoint to separate these sources, unlike generative pipelines where these factors are entangled.
3. Contextual Aggregation is Measurable: GCB demonstrates that cross-window contextual information can be isolated and quantified as a distinct source of performance gain, separate from encoder learning.
4. Evidence Dependence: Contextual aggregation is not a "magic bullet"; it amplifies reliable, stimulus-locked local evidence. When local evidence is weak (as in EEG or attenuated MEG), contextual aggregation fails to improve performance.

The authors conclude that convincing neural-language decoding must not only show what output was produced but also to what extent that output is supported by stimulus-locked neural evidence versus decoder priors or evaluation artifacts. They advocate for retrieval-style probes and task-specific controls over segmentation, context, and candidate structure to establish this attribution in future work.