🧠 neurology

Device sensitivity and false alarms can reshape regression-to-the-mean in simulated epilepsy trials

This study demonstrates that imperfect seizure detection devices, characterized by reduced sensitivity or elevated false alarm rates, significantly amplify regression-to-the-mean and inflate the apparent placebo response in simulated epilepsy clinical trials.

Original authors: Goldenholz, D. M., Goldenholz, S. R., Bhansali, R. M., Kaptchuk, T. J., Westover, M. B.

Published 2026-08-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Goldenholz, D. M., Goldenholz, S. R., Bhansali, R. M., Kaptchuk, T. J., Westover, M. B.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to measure how much a plant grows, but you only check it once a week. If you happen to check it right after a massive rainstorm when it's unusually tall, and then check it again a month later when it's back to its normal size, you might think the plant actually shrank. In reality, it just went back to its average height. Scientists call this "regression to the mean." It's a tricky statistical quirk that happens whenever you pick a group of people or things based on a moment when they are acting unusually extreme.

Now, imagine you are testing a new medicine for epilepsy, a condition where people have seizures. To join the study, patients must have a certain number of seizures during a "baseline" period. If a patient has a really bad week by chance, they get into the study. Later, when their seizures naturally calm down to their usual level, it looks like the medicine worked, even if it did nothing at all. This fake improvement is often called the "placebo response."

But here is the twist: what if you aren't counting seizures with a diary, but with a high-tech smartwatch or sensor? These devices are amazing, but they aren't perfect. Sometimes they miss a real seizure (like a sleepy guard missing a thief), and sometimes they think a sneeze is a seizure (a false alarm). This paper asks a big question: Do these little mistakes by the devices change how much we see this "fake improvement" in clinical trials? The authors didn't test real people; they built a giant, super-smart computer simulation to see what happens when the counting tools make mistakes.

The Paper's Story: When Counting Tools Get Clumsy

The researchers used a computer program called CHOCOLATES, which acts like a digital time machine. It created 100,000 fake patients with realistic seizure patterns, including the natural ups and downs that happen in real life. They then ran a simulated clinical trial on these digital people. In this fake world, there was no actual medicine given—just a "placebo" group—to see how much improvement happened purely because of natural changes and the way the study was set up.

The team tested two main ways the devices could mess up the count:

Missing the action (Low Sensitivity): Imagine a camera that only catches 1 out of every 10 seizures.
Seeing ghosts (False Alarms): Imagine a camera that thinks every time a leaf falls, a seizure happened.

What They Found

The results were surprising and showed that the quality of the device changes the math of the trial.

When the device was perfect (catching 100% of seizures and making zero mistakes), about 38.2% of the eligible patients showed this "regression to the mean" effect, and the group looked like they improved by about 14.7% on average. This is the baseline for a perfect world.

But when the device got worse, the numbers jumped up dramatically:

If the device missed a lot of seizures (dropping sensitivity to just 10%): The number of patients showing this fake improvement jumped to 64.8%, and the group looked like they improved by a huge 48.1%.
If the device made false alarms (up to 1 fake alarm per day): Even though the researchers tried to subtract the fake alarms from the count, the "regression to the mean" still jumped to 53.2%, and the fake improvement rose to 31.3%.

The Takeaway

The paper suggests that the tools we use to count seizures are not just passive observers; they are active players that can change the outcome of the game. If a device is bad at catching seizures, it accidentally selects patients who had a "lucky" (or unlucky) high-seizure week just by chance, making the trial look like a huge success later on when things return to normal.

The authors argue that we can't just treat device accuracy as a simple checklist item. Instead, we need to treat the device's sensitivity and its error rate as part of the trial's design itself. Before running a real study, scientists should simulate how their specific device will behave, because a clumsy counter can make a placebo look like a miracle cure, or hide a real cure entirely. In short, if your ruler is bent, your measurements of the world will be bent too.

Technical Summary: Device Sensitivity and False Alarms Reshape Regression-to-the-Mean in Simulated Epilepsy Trials

Problem Statement
Automated seizure detection devices are increasingly utilized in epilepsy clinical trials, yet their performance characteristics—specifically sensitivity and false alarm rate (FAR)—are typically evaluated for patient safety or diary accuracy rather than their impact on trial design. Epilepsy randomized controlled trials (RCTs) are inherently vulnerable to apparent placebo responses driven by regression-to-the-mean (RTM), a phenomenon where patients selected during periods of high seizure frequency naturally return to their long-term average. The authors posit that imperfect seizure detection (missed seizures due to low sensitivity or added noise due to high FAR) may alter the measured baseline seizure frequency, thereby changing eligibility criteria and distorting the apparent placebo response and RTM effects.

Methodology
The study employed a simulation approach using CHOCOLATES, an open-source, realistic seizure diary simulator that recapitulates observed properties such as heterogeneous long-term rates, seizure clustering, and cycles. The simulation modeled a standard placebo arm in a randomized trial with a 2-month baseline period and a 3-month test period.

Eligibility Criteria: Modeled after focal seizure trials (e.g., cenobamate studies), requiring a mean baseline frequency of $\ge$ 4 seizures/month, at least 3 seizures per baseline month, and no seizure-free interval exceeding 25 days.
Device Modeling:
- Sensitivity: Implemented probabilistically at the daily seizure-count level (each true seizure has a probability of detection).
- False Alarm Rate (FAR): Implemented as Poisson-distributed false alarms added daily.
- Correction: For conditions with FAR > 0, an "idealized" correction factor was applied by subtracting the expected false alarm count (FAR $\times$ duration) from the total seizure counts to improve signal quality.
Experimental Design: Two sweeps were conducted with 100,000 potential participants each:
1. Sensitivity Sweep: FAR fixed at 0; sensitivity varied from 10% to 100%.
2. FAR Sweep: Sensitivity fixed at 100%; FAR varied from 0 to 1 alarm/day.
Metrics:
- RTM Fraction: The proportion of eligible participants whose baseline frequency exceeded their effective long-term frequency, with the test period frequency moving closer to that long-term average.
- Placebo Median Percentage Change (MPC): Calculated as $100 \times (1 - \text{test-period frequency} / \text{baseline frequency})$ .

Key Results
The simulation demonstrated that device imperfections significantly alter the magnitude of RTM and the apparent placebo response, even when statistical corrections are applied.

Impact of Reduced Sensitivity (FAR = 0):
- As sensitivity decreased from 100% to 10%, the fraction of eligible participants exhibiting RTM increased from 38.2% to 64.8%.
- The placebo MPC increased from 14.7% to 48.1%.
- Lower sensitivity enriched the eligible cohort with individuals who had unusually high measured baseline periods relative to their true long-term frequency, leading to larger apparent improvements.
Impact of Increased FAR (Sensitivity = 100%):
- Despite the application of expected false alarm corrections, increasing FAR from 0 to 1 alarm/day increased the RTM fraction from 38.2% to 53.2%.
- The placebo MPC increased from 14.7% to 31.3%.
- The authors note that false alarms can still alter daily/monthly eligibility behaviors and trial selection, even when the total count is corrected.

Significance and Claims
The paper argues that device performance characteristics should be treated as fundamental trial-design parameters rather than mere validation metrics. The primary significance of the findings is that two trials with identical biological treatment effects and eligibility rules could yield different apparent placebo responses solely due to differences in the detection devices or algorithms used.

The authors propose an extension to the historical framework of RTM:

RTM Type 1: Temporary unusual circumstances (e.g., illness).
RTM Type 2: Natural variability returning to a typical state.
RTM Type 3 (Proposed): The impact of measurement noise (from seizure detectors) on the system. The study demonstrates that RTM Type 3 alone can inflate placebo response when other factors are held constant.

Conclusion
The authors conclude that objective seizure detection does not automatically guarantee more objective trial statistics; rather, it makes trial statistics dependent on detection algorithm performance in a non-linear fashion. They recommend that investigators specify how sensitivity, FAR, and correction methods will influence eligibility, baseline counts, and outcomes before finalizing RCT designs. Furthermore, device validation should report performance specifically within the seizure types and clinical settings matching the planned trial population.

Technical Summary: Device Sensitivity and False Alarms Reshape Regression-to-the-Mean in Simulated Epilepsy Trials

More like this