An adversarial approach to guide the selection of preprocessing pipelines for ERP studies

This paper proposes an adversarial approach that uses realistically simulated signals injected into real EEG data as ground truth to objectively evaluate and select preprocessing pipelines, thereby optimizing noise removal while preserving neural signal integrity to enhance the reproducibility and interpretability of ERP studies.

The Gist

Imagine you are a chef trying to make the perfect soup. You have a pot of delicious broth (your brain's electrical signals), but it's been contaminated with dirt, hair, and maybe a few stray pebbles (noise from eye blinks, muscle movements, or bad equipment).

Your goal is to get a clean, pure bowl of soup to serve to your guests (publish your research). To do this, you have a massive toolbox of cleaning methods: some use a fine sieve, others use a magnet, some use a chemical filter, and others use a high-powered blender. The problem? No one knows which tool works best for your specific pot of soup.

If you pick the wrong tool, you might remove the dirt but also throw away the carrots (the actual brain signal you care about). If you pick the wrong tool, you might leave the dirt in, making the soup taste terrible. Worse, if you keep changing your cleaning method every time you cook, no one can trust your recipes, and other chefs can't replicate your results.

This paper is about building a fair "Taste Test" to help chefs (scientists) choose the right cleaning tool.

The Problem: The "Circular" Trap

Usually, scientists try to figure out the best cleaning method by looking at their own data. But this is like a student grading their own homework. They might unconsciously pick the cleaning method that makes their results look the most exciting or significant. This is dangerous because it leads to "false positives"—finding effects that aren't actually there.

The Solution: The "Magic Ingredient" Injection

The authors came up with a clever, "adversarial" (competitive) way to test cleaning tools without cheating. Here is how they did it, using a simple analogy:

1. The Real Soup: They took a real recording of a person's brain activity (the soup with dirt).
2. The Magic Ingredient (Ground Truth): They created a perfect, known signal (like a specific, pure spice blend) using a computer model. They knew exactly what this "spice" looked like.
3. The Injection: They secretly injected this "Magic Ingredient" into the dirty soup. Crucially, they didn't tell the cleaning tools what the ingredient was.
4. The Cleaning Challenge: They ran the soup through six different popular cleaning pipelines (the different chefs' methods).
5. The Taste Test: After cleaning, they checked the soup.
   • Did the dirt disappear?
   • Most importantly: Did the "Magic Ingredient" survive? Was it still there, or did the chef accidentally throw it away thinking it was dirt?

They measured this using a score called RMSE (Root Mean Squared Error). Think of this as a "Distance Score."

• Low Score: The cleaned soup looks almost exactly like the original soup with the magic ingredient. (Great job!)
• High Score: The soup is either still dirty, or the magic ingredient is gone/distorted. (Bad job!)

The "Adversarial" Twist

Instead of just saying "Chef A is the best," they pitted the chefs against each other in a massive tournament. They ran the test thousands of times with different random samples of soup.

The result wasn't a single winner. Instead, they produced a Probability Map.

• Example: "If you have 100 trials (batches of soup), Chef Henare is 70% likely to do a better job than Chef Delorme."
• Example: "But if you only have 5 trials, Chef Makoto is actually the winner."

Key Findings: It Depends on the Situation!

The study found that there is no single "Best Chef" for every situation. The winner depends entirely on how much data you have:

• The Aggressive Cleaner (Makoto): This method is very strict. It throws out almost everything that looks suspicious.

  • Good for: When you have very little data (few trials). It cleans so aggressively that even a small amount of noise is removed, leaving a clear signal.
  • Bad for: When you have lots of data. Because it's so aggressive, it sometimes throws away the "Magic Ingredient" (the real brain signal) along with the dirt. If you have lots of data, you don't need to be so aggressive; you can let the averaging process clean the noise naturally.

• The Gentle Cleaners (Prep, Henare): These methods are more careful. They keep more of the original data.

  • Good for: When you have lots of data. They preserve the "Magic Ingredient" perfectly, and since you have so many trials, the random noise averages itself out anyway.
  • Bad for: When you have very little data. They might leave too much dirt behind because they aren't aggressive enough.

Why This Matters

This paper gives scientists a flexible guide rather than a rigid rulebook.

• No More Guessing: You don't have to guess which cleaning method to use. You can run this "Taste Test" on your own data (using a pilot study) to see which method preserves your specific signal best.
• No Cheating: Because the "Magic Ingredient" is fake and injected, the scientists can't cheat to make their results look better. They are testing the cleaning process, not the hypothesis.
• Context is King: It teaches us that the "best" method changes based on your experiment. If you have 500 trials, use one method. If you have 20, use another.

The Takeaway

Think of this paper as a smart menu for EEG researchers. Instead of forcing everyone to eat the same dish, it tells you: "If you are cooking for a small crowd, use the Spicy Cleaner. If you are cooking for a huge banquet, use the Gentle Cleaner."

By using this method, researchers can ensure their "soup" is clean, their "ingredients" are safe, and their scientific recipes are trustworthy and reproducible.

Technical Summary

1. Problem Statement

Electroencephalography (EEG) data is inherently contaminated by non-neuronal noise (e.g., eye blinks, muscle activity, cardiac signals, and electrical interference). While preprocessing is essential to remove these artifacts, there is no consensus on the optimal method.

• The Challenge: The vast number of available techniques, their combinatorial use in pipelines, and adjustable parameters create a "garden of forking paths." Researchers often adopt ad hoc strategies based on expertise rather than objective evidence.
• Limitations of Current Evaluation:
  • Simulated Data: Using purely simulated noise can introduce circularity if the simulation models resemble the cleaning algorithms being tested.
  • Real Data: Using real EEG recordings lacks a "ground truth" neuronal signal, making it impossible to know if a cleaning method removed noise or distorted the actual neural signal.
  • Bias: Selecting a pipeline based on whether it produces statistically significant results in a specific study introduces confirmation bias.
• Goal: To develop an objective, flexible, and blind framework for evaluating and comparing preprocessing pipelines that maximizes the signal-to-noise ratio (SNR) without distorting the neural signal.

2. Methodology

The authors propose an adversarial framework that injects a known "ground truth" signal into real EEG data to evaluate pipeline performance.

A. Ground Truth Generation (Study 1)

• Source Signal: A linear sinusoidal chirp (frequency increasing from 2Hz to 30Hz over 1 second) was generated to mimic the diverse time-frequency dynamics of Event-Related Potentials (ERPs).
• Forward Modeling: A forward model was created using the ICBM152 anatomy template and the Boundary Element Method (BEM) via the Brainstorm toolbox.
• Projection: The source signal was projected to sensor space (64 electrodes) using the SEREEGA toolbox, creating a matrix of "clean" ground truth data ($x$).
• Injection: This ground truth matrix was injected into a real EEG recording (collected during a Simon task) at the onset of 300 simulated events. The injection formula was $T = D + x$, where $D$ is the real noisy data and $T$ is the resulting trial.
  • Note: The ground truth amplitude (2.2 µV) was kept low relative to raw EEG (2000 µV) to simulate realistic conditions.

B. Pipeline Selection

Six publicly available or published pipelines were tested (all implemented in EEGLAB):

1. EEGLAB: Automated ERP pipeline.
2. Delorme_2023: Optimized for ERPs.
3. Makoto: Based on the "Makoto's preprocessing pipeline" webpage.
4. Prep: Automated early-stage pipeline (augmented with ICA).
5. Henare_2018: A pipeline used in the authors' lab.
6. Henare_2018_Once: A variation of the above without ICA on a 1Hz filtered copy.

C. Adversarial Analysis

To compare pipelines ($P_A$ vs. $P_B$) objectively:

1. Common Trials: Only trials retained by all pipelines were used to ensure fair comparison.
2. Epoching: Baseline periods were removed; only the injected signal window was analyzed.
3. Resampling: For each iteration, $N$ trials (e.g., 100) were randomly sampled and averaged to create a grand average.
4. Metric: The Root Mean Squared Error (RMSE) was calculated between the pipeline's grand average and the known ground truth.
   • Lower RMSE indicates better noise removal with less signal distortion.
5. Probability Calculation: The process was repeated 100,000 times. The metric reported is the probability $P(RMSE_A \leq RMSE_B)$, representing the likelihood that Pipeline A performs as well as or better than Pipeline B.

D. Study 2 (Validation)

The authors identified a potential bias in Study 1: aggressive Independent Component Analysis (ICA) pipelines (like Makoto) might have isolated and removed the ground truth signal, mistaking it for noise.

• Correction: A new ground truth was generated using a real Independent Component (IC) from a mental rotation task that was highly likely to be classified as "brain activity" by IClabel.
• Constraint: Pipelines were forced to retain any IC highly correlated with the new ground truth, ensuring the signal wasn't removed due to strict noise-labeling criteria.

3. Key Results

Study 1 Results (Chirp Signal)

• No Universal Winner: No single pipeline consistently outperformed all others across all comparisons.
• Trial Number Effect: Performance was highly dependent on the number of trials averaged ($N$).
  • Low Trial Counts ($N \leq 25$): Makoto's pipeline performed best (highest probability of low RMSE). Its aggressive noise removal was beneficial when averaging could not cancel out noise.
  • High Trial Counts ($N \approx 100$): Henare_2018 and EEGLAB performed best. Aggressive removal (like Makoto) became counterproductive, likely distorting the signal, while averaging handled the remaining noise.
• Specific Trends: Delorme_2023 and Makoto generally showed lower probabilities of outperforming others when $N=100$.

Study 2 Results (Real IC Signal)

• Validation: The results remained consistent with Study 1, confirming that the performance trends were not solely artifacts of the ICA removal criteria.
• Correction Impact: Only EEGLAB and Delorme_2023 had flagged a ground-truth-correlated IC as noise in Study 1. The forced retention in Study 2 slightly improved their performance relative to others but did not change the overall ranking trends.
• Consistency: Henare_2018 and Henare_2018_Once consistently showed high probabilities (>0.7) of performing well across comparisons.

Descriptive Statistics

• Computation Time: Prep was the slowest (over 1 hour in Study 1, 30 mins in Study 2), while Makoto took ~17 mins. EEGLAB and Henare variants were fastest (5-6 mins).
• Data Loss: Makoto removed the most Independent Components (43 in Study 1), while Prep removed none.

4. Key Contributions

1. Objective Evaluation Framework: Introduced a "ground truth injection" method that allows researchers to evaluate pipelines on their own data without biasing the main experimental results.
2. Adversarial Probability Metric: Replaced binary "best/worst" conclusions with a probabilistic matrix ($P(RMSE_A \leq RMSE_B)$), offering a nuanced view of pipeline performance.
3. Trial-Number Dependency: Demonstrated that the "best" pipeline is context-dependent, specifically regarding the number of trials available for averaging.
   • Single-trial/low-trial analysis: Aggressive denoising (e.g., Makoto) is preferred.
   • High-trial ERP analysis: Moderate denoising (e.g., Henare, EEGLAB) is preferred to preserve signal integrity.
4. Open Science: Provided public code (OSF, GitHub) for the injection toolbox and analysis scripts, enabling reproducibility.

5. Significance and Implications

• Reproducibility: The method addresses the "researcher degrees of freedom" problem by providing a standardized, blind way to select preprocessing steps before analyzing experimental hypotheses.
• Informed Decision Making: It shifts the paradigm from seeking a "one-size-fits-all" pipeline to a tailored approach. Researchers can now select pipelines based on their specific experimental constraints (e.g., number of trials, signal type, available time).
• Blindness to Hypothesis: By using simulated trials and injected signals, the method ensures that pipeline selection is independent of the study's primary findings, reducing the risk of p-hacking or analytical bias.
• Flexibility: The framework is adaptable. Researchers can modify the ground truth (e.g., specific ERP components like P300), the forward model (using individual MRI data), or the similarity metric (e.g., time-frequency analysis) to suit their specific research questions.

In conclusion, the paper argues that there is no single "best" preprocessing pipeline. Instead, researchers should use this adversarial framework to empirically determine which pipeline best preserves their specific signal of interest under their specific experimental conditions.