📊 statistics

How accurate are Bayes factor-based null hypothesis tests? A simulation study

This simulation study validates the accuracy of Bayes factor-based null hypothesis tests in common psychological designs using the brms and bridgesampling packages, demonstrating that estimates are reliable when no algorithmic warnings are issued but should be disregarded when such warnings appear.

Original authors: Daniel J. Schad, Martin Modrák

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel J. Schad, Martin Modrák

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: does a new clue actually prove your suspect is guilty, or is it just a coincidence? In the world of science, especially in fields like psychology and linguistics, researchers often face this same question. They have data, and they have two competing stories: one where a specific effect exists (the "guilty" story) and one where nothing is happening (the "innocent" story). To decide which story is better, scientists use a mathematical tool called a Bayes factor. Think of a Bayes factor as a super-precise scale that weighs the evidence. If the scale tips heavily toward the "guilty" story, the researcher feels confident; if it stays balanced, they know the evidence is weak.

However, calculating the exact weight on this scale is often like trying to count every single grain of sand on a beach while standing on a moving boat. It's mathematically impossible to do perfectly, so scientists use clever computer shortcuts to get an approximate answer. The big worry is: how close is that approximation to the truth? If the computer's guess is off, the detective might convict an innocent person or let a guilty one go free. This is the exact problem tackled in a new study by Daniel J. Schad and Martin Modrák. They wanted to know if the popular computer tools scientists use to weigh these Bayes factors are actually accurate, or if they are secretly broken in ways that could lead to wrong conclusions.

To test this, the authors didn't just look at real data; they built a "simulation lab." Imagine they created thousands of fake experiments where they knew the answer beforehand. They generated fake data using a specific set of rules, then asked the computer tools to calculate the Bayes factor and see if the tool's conclusion matched the truth they had planted. They focused on three very common types of experimental designs used in psychology and linguistics: a simple 2x2 design, a design mixing subjects and items (like different people reading different words), and a more complex version of that mix.

The results were a tale of two very different outcomes, depending on a tiny detail: a warning message. When the computer software ran its calculations and did not show a warning message, the Bayes factors were spot-on. The scale was perfectly calibrated; the tool's estimate of the evidence matched the truth almost exactly. It was as if the detective's scale was working perfectly, giving reliable verdicts every time.

However, the story changed dramatically when the software did issue a warning. In the simulations where the computer said, "Hey, I'm having trouble estimating this number," the results became unreliable. The Bayes factors were no longer accurate; they were biased and wildly variable. Sometimes the tool would exaggerate the evidence, making a weak effect look strong (a "liberal" bias), and other times it would downplay a real effect, making it look like nothing was happening (a "conservative" bias). The authors found that in these warning cases, the computer was essentially guessing with a lot of noise, and the results could not be trusted.

Interestingly, the authors tried to fix this by telling the computer to work harder—running more calculation steps (increasing from 10,000 to 40,000 iterations). While this helped a tiny bit, it didn't stop the warnings or fully fix the bias. The study suggests that the problem isn't just about doing more math; it's about the shape of the data being too tricky for the current shortcuts to handle smoothly.

So, what does this mean for the future of science? The authors conclude that for the common experimental designs they tested, Bayes factors are a great tool—but only if the computer stays silent. If the software flashes a warning, the researchers should stop and not trust the result. It's a bit like driving a car: if the "Check Engine" light is off, you can drive with confidence. But if that light is blinking, you shouldn't just ignore it and keep speeding; you need to pull over and fix the problem before you trust the speedometer. The study serves as a crucial reminder that in the high-stakes game of scientific discovery, paying attention to the computer's warning signs is just as important as the data itself.

Technical Summary: How accurate are Bayes factor-based null hypothesis tests? A simulation study

Problem Statement
Bayes factors (BFs) have become a standard tool for quantifying evidence and comparing models in cognitive sciences, particularly for null hypothesis testing in factorial designs using linear mixed-effects models (LMMs). However, for complex models common in psychology and psycholinguistics, exact Bayes factors cannot be computed analytically. Instead, researchers rely on numerical approximations, typically involving Markov-chain Monte Carlo (MCMC) sampling followed by bridge sampling to estimate marginal likelihoods (e.g., using the bridgesampling and brms R packages).

The central problem addressed is that the accuracy of these approximations is often unknown. It is unclear whether the computed Bayes factors correspond to the true Bayes factors or if they suffer from bias (liberal or conservative) due to computational imprecisions in the MCMC or marginal likelihood estimation steps. Biased approximations can lead to erroneous conclusions regarding the strength of evidence for hypotheses.

Methodology
The authors employ Marginal Simulation-Based Calibration (SBC) specifically adapted for Bayes factors, a method recently proposed by Schad et al. (2023). This technique tests whether the posterior model probabilities derived from simulated data match the prior model probabilities used to generate that data. If the estimation is accurate, the average posterior probability should equal the prior probability; deviations indicate bias.

The study utilizes the following workflow:

Simulation Design: Three common experimental designs in cognitive psychology and psycholinguistics were simulated:
- Design 1: A 2x2 repeated measures design with random effects for subjects only (inspired by 2-step decision-making tasks).
- Design 2: A Latin square design with one 2-level fixed factor and crossed random effects for subjects and items.
- Design 3: A 2x2 Latin square design with crossed random effects for subjects and items.
Procedure: For each design, data were simulated from a "true" model ( $H_1$ ) and a null model ( $H_0$ ) where specific fixed effects were set to zero. Prior model probabilities were set (e.g., $p(H_0)=0.5$ or $0.8$).
Estimation: Models were fitted using brms with MCMC sampling (10,000 or 40,000 iterations) and marginal likelihoods were estimated using the bridgesampling package (using the "warp3" method).
Diagnostics: The authors analyzed the results based on whether the bridge sampler issued a warning message ("logml could not be estimated within maxiter..."). They compared:
- The deviation of average posterior probabilities from priors.
- Bayesian t-tests on these deviations.
- Reliability diagrams (calibration plots) to assess if posterior probabilities accurately predict the true model.
Sensitivity Analyses: The study varied MCMC iterations (10k vs. 40k), the number of SBC simulations (200 vs. 1000), and prior model probabilities ( $p(H_0)=0.8$ vs. $0.2$) to test robustness.

Key Results
The study yields a bifurcated conclusion depending on the presence of computational warnings:

No Warning Messages: When the bridge sampler did not issue a warning, Bayes factor estimates were found to be accurate.
- The average posterior model probabilities matched the prior probabilities.
- Reliability diagrams showed estimates aligned with the diagonal, indicating proper calibration.
- Bayesian t-tests supported the null hypothesis that there was no deviation between prior and posterior (e.g., $BF_{01} > 10$ ).
- This held true across all three designs and various fixed effects (main effects and interactions).
Warning Messages: When the bridge sampler issued a warning, Bayes factor estimates were biased and highly variable.
- Direction of Bias: The bias was not consistently one-directional but depended on the prior probability.
  - When $p(H_0)$ was high (0.8), warnings led to posterior probabilities for $H_1$ that were too high (liberal bias), potentially indicating evidence for an effect when none exists.
  - When $p(H_0)$ was low (0.2), warnings led to posterior probabilities for $H_1$ that were too low (conservative bias), potentially masking true effects.
- Mechanism: The bias was driven by high variance in the estimated log marginal likelihoods. The warnings indicated that the posterior geometry was irregular (e.g., small residual standard deviations or random effect variances near zero), causing the bridge sampling algorithm to fail in accurately estimating the marginal likelihood.
- Mitigation Attempts: Increasing MCMC iterations from 10,000 to 40,000 only slightly reduced the frequency of warnings (from ~37% to ~32.5%) without resolving the bias in the warned cases. Increasing the number of SBC simulations to 1,000 confirmed that the bias in warned cases was robust and clear.

Significance and Claims
The paper claims to provide a practical validation of Bayes factor estimation in commonly used factorial designs within the cognitive sciences. Its primary contributions are:

Validation of Standard Tools: It confirms that the brms/bridgesampling workflow produces accurate Bayes factors for standard designs provided the algorithm does not flag a warning.
Diagnostic Utility of Warnings: The authors argue that warning messages from the bridge sampler are not merely technical nuisances but critical indicators of unreliable results. They assert that Bayes factor results accompanied by such warnings should not be trusted, as they are likely biased and uninformative.
Methodological Recommendation: The study advocates for the use of marginal SBC and calibration plots as a necessary step to verify the accuracy of Bayes factor estimates before trusting individual analysis results.
Practical Guidance: For researchers encountering warnings, the paper suggests that reparametrization (e.g., sum-to-zero constraints, dropping random effects) might resolve the issue, but if not, one should avoid reporting the Bayes factor or consider alternative methods (e.g., ROPE, posterior credibility intervals, or other BF estimation methods like Savage-Dickey, pending their own validation).

The authors maintain a modest tone, acknowledging that their results are specific to the designs, priors, and software versions tested. They emphasize that while the current tools work well under specific conditions, the community must remain vigilant about computational warnings and validate their specific analysis setups using simulation-based calibration.

Technical Summary: How accurate are Bayes factor-based null hypothesis tests? A simulation study

More like this