📊 statistics

Stop Using the Wilcoxon Test: Myth, Misconception and Misuse in IR Research

This paper argues that the widespread use of the Wilcoxon signed-rank test in Information Retrieval benchmarking is based on a misleading narrative and is methodologically unsound, as it frequently fails to control Type I error rates and should be abandoned in favor of more reliable statistical methods.

Original authors: Julián Urbano

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Julián Urbano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge in a cooking competition. You have two chefs, Chef X and Chef Y, and you ask them to cook the same 50 dishes. You want to know: Is Chef X actually better, or did they just get lucky with the specific ingredients they were given?

To answer this, researchers in Information Retrieval (the field of search engines) have been using a specific mathematical tool called the Wilcoxon test for decades. They were told by textbooks that this tool is the "safe, backup plan" to use when the data looks messy or weird. They thought it was the "seatbelt" of statistics.

This paper argues that the seatbelt is actually a trap. The author, Julián Urbano, says we should stop using the Wilcoxon test immediately because it doesn't just fail; it actively lies to us, especially in the world of search engines.

Here is the breakdown of the paper's argument using simple analogies:

1. The Great Myth: "Non-Parametric Means No Rules"

The Myth: Textbooks taught researchers that the Wilcoxon test is "non-parametric," which sounds like it means "it has no rules" or "it works with any kind of data." People thought, "If my data isn't perfectly smooth and bell-shaped, I'll just use Wilcoxon because it's the 'free-for-all' option."

The Reality: The paper reveals that Wilcoxon is not a free-for-all. It has a very strict, hidden rule: Symmetry.

The Analogy: Imagine a seesaw. The Wilcoxon test only works if the seesaw is perfectly balanced. If the weight on one side is slightly heavier than the other (asymmetry), the test breaks.
The Problem: In the real world of search engines (IR data), the data is almost never perfectly balanced. It's lopsided. By using Wilcoxon on lopsided data, researchers are essentially trying to balance a seesaw that is already tipped over. The result? The test screams "There's a difference!" even when there isn't one.

2. The Misconception: "The T-Test Needs Perfect Data"

The Misconception: People were told that the standard Student's t-test (the "normal" test) only works if the data is a perfect, smooth bell curve. Since search engine scores are often jagged or discrete, people thought, "Oh, the t-test will fail, so I must use Wilcoxon."

The Reality: The t-test is actually much tougher than people think.

The Analogy: Think of the t-test as a heavy-duty truck. It was designed to drive on smooth highways (perfect bell curves), but thanks to a principle called the "Central Limit Theorem," it can actually handle bumpy, rocky, and jagged roads just fine. As long as you have enough data points (like driving for a long enough distance), the truck smooths out the bumps.
The Finding: The paper shows that even when the data is weird, the t-test stays on the road and gives the right answer. It doesn't crash.

3. The Catastrophic Misuse: Why Wilcoxon Fails

The paper ran thousands of computer simulations using real search engine data to see what happens when you use these tests on "messy" data.

The T-Test: When the data was weird (skewed, heavy tails, or discrete), the t-test kept its error rate steady at 5%. It was reliable.
The Wilcoxon Test: When the data was slightly lopsided (asymmetric), the Wilcoxon test went haywire.
- The Analogy: Imagine a smoke detector that is so sensitive it goes off when you just toast a piece of bread. In the paper's simulations, as the sample size got bigger, the Wilcoxon test started screaming "FIRE!" (rejecting the null hypothesis) almost 100% of the time, even when there was no fire at all.
- Why? Because it stopped testing for "who is better" and started testing for "is the data lopsided?" Since search engine data is almost always lopsided, the test falsely claims a winner exists when there is none.

4. The Conclusion: Stop Using the "Safe" Option

The author concludes that the belief that Wilcoxon is the "safe alternative" is a dangerous lie.

The T-Test is the robust truck that handles real-world bumps.
The Wilcoxon Test is a fragile glass house that shatters the moment the wind blows (asymmetry).

The Verdict: The paper urges the Information Retrieval community to stop using the Wilcoxon test entirely. Continuing to use it creates a false sense of security and leads researchers to believe they have found improvements in their search engines when they have actually just found statistical noise.

In short: The "backup plan" is actually the one that causes the most accidents. It's time to retire it and stick with the tool that actually works: the t-test.

1. Problem Statement

In Information Retrieval (IR) evaluation, researchers frequently compare system performance using statistical significance tests. The Student's $t$ -test and the Wilcoxon signed-rank test are the two dominant methods. A widespread belief in the IR community, fueled by textbooks and guidelines, is that the Wilcoxon test is a "safer," non-parametric alternative to the $t$ -test when metric scores (e.g., NDCG, AP) are not normally distributed.

The author argues that this narrative is misleading and harmful. The core problem is that the IR community routinely misapplies the Wilcoxon test based on a misunderstanding of its assumptions. While the $t$ -test is often rejected due to perceived non-normality of scores, the Wilcoxon test is adopted under the false premise that it is "assumption-free." In reality, the Wilcoxon test relies on a symmetry assumption regarding the distribution of paired differences. The paper posits that IR data often violates this symmetry, causing the Wilcoxon test to lose control of its Type I error rate (false positives), whereas the $t$ -test remains robust.

2. Methodology

The paper employs a multi-faceted approach combining literature review, theoretical analysis, and empirical simulation:

Systematic Literature Review: The author analyzed 25 Statistics textbooks (covering general statistics, non-parametric statistics, and field-specific applications) to trace the origins of the "Wilcoxon is safer" myth. The review focused on how these texts define non-parametric methods, describe the $t$ -test assumptions, and present the Wilcoxon test.
Theoretical Analysis:
- $t$ -test: The paper clarifies that the $t$ -test does not require the raw scores ( $X, Y$ ) to be normal, but rather the sample mean of the differences ( $\bar{D}$ ) to be approximately normal. By the Central Limit Theorem (CLT), this condition is met for moderate sample sizes even if the underlying distribution of differences ( $D$ ) is non-normal.
- Wilcoxon Test: The paper highlights that the Wilcoxon signed-rank test is a test of the median, which equals the mean only if the distribution of differences is symmetric. If symmetry is violated, the test effectively becomes a test of symmetry rather than a test of mean differences, leading to invalid $p$ -values.
Empirical Simulation (Monte Carlo):
- Data Source: Real-world IR data was collected from TREC tracks (Web, Microblog, Deep Learning, Clinical Decision Support, Precision Medicine), yielding 53,703 paired difference samples.
- Distribution Modeling: The author simulated 100,000 samples under the null hypothesis ( $H_0: \mu_D = 0$ $H_{0} : μ_{D} = 0$ ) using four specific non-normal distributions to model IR data characteristics:
  1. Asymmetry: Using Asymmetric Generalized Normal (AGN) and Tukey G-and-H (TGH) distributions.
  2. Tail Heaviness: Using Symmetric Generalized Normal (SGN) and TGH.
  3. Discreteness: Using Irregular Beta-Binomial (IBB) to simulate metrics like P@k and RR@k.
  4. Multimodality: (Briefly discussed but excluded from detailed simulation due to quantification difficulties).
- Parameters: Simulations varied sample sizes ( $n = 5, 50, 500, 5000$ ) and levels of deviation from normality (from low to "pathologically high").
- Metric: The primary metric was the Type I error rate at $\alpha = 0.05$ .

3. Key Contributions

A. Deconstruction of the "Myth, Misconception, Misuse"

The Myth: Non-parametric methods are "assumption-free."
- Correction: They are not. The Wilcoxon test assumes symmetry of the difference distribution. This is a stronger and often more easily violated assumption than the approximate normality required for the $t$ -test mean.
The Misconception: The $t$ $t$ -test requires normally distributed scores.
- Correction: The $t$ -test requires the sample mean to be normal. Due to the CLT, this holds true for IR sample sizes ( $n \ge 50$ ) even with skewed or heavy-tailed data.
The Misuse: Using Wilcoxon as a "safe" fallback for non-normal data.
- Correction: Because IR data is frequently asymmetric, using Wilcoxon introduces a severe risk of inflated Type I errors, making it less safe than the $t$ -test.

B. Empirical Evidence of Failure

The simulations demonstrated that:

Discreteness and Tail Heaviness: Both the $t$ -test and Wilcoxon test perform well (maintaining $\approx 5\%$ error rates) under discrete and heavy-tailed distributions, provided $n$ is not trivially small.
Asymmetry (The Critical Failure):
- The $t$ -test remains robust. Even under pathological asymmetry, the error rate stays near 5% for $n \ge 500$ and only slightly inflates (up to ~10%) for $n=50$ .
- The Wilcoxon test fails catastrophically. As sample size increases, the error rate skyrockets. Under strong asymmetry with $n=500$ , the test rejects the null hypothesis nearly 100% of the time, even when no true difference exists. It effectively tests for asymmetry rather than performance difference.

C. Analysis of Textbook Inconsistencies

The review revealed that only ~50% of the surveyed textbooks explicitly mention the symmetry assumption for the Wilcoxon test. Many present it vaguely as a test for "ordinal data" or "location," obscuring the critical symmetry requirement. This lack of clarity in foundational literature has propagated the misuse into IR research.

4. Results

Type I Error Rates:
- Symmetric/Heavy-tailed/Discrete: Both tests maintain nominal error rates ( $\approx 0.05$ ).
- Asymmetric:
  - $t$ -test: Error rates remain close to 0.05 for $n \ge 500$ ; slight inflation for $n=50$ under extreme skew.
  - Wilcoxon: Error rates increase linearly with asymmetry and sample size. At $n=5000$ with high asymmetry, the error rate reaches 1.0 (100% false positive rate).
IR Data Reality: Analysis of TREC data confirms that IR metric differences are frequently asymmetric (skewness $|\gamma| \approx 0.5$ to $3.0$), placing them in the "danger zone" for the Wilcoxon test.

5. Significance and Recommendations

Retirement of Wilcoxon: The paper concludes that the Wilcoxon signed-rank test should be abandoned in IR evaluation. Its use is unjustified because it is structurally prone to failure under the very conditions (asymmetry) that characterize IR data.
Re-evaluation of the $t$ -test: The $t$ -test is recommended as the standard for paired comparisons in IR, as it is robust to the non-normality (skewness, kurtosis, discreteness) typical of IR metrics, provided sample sizes are reasonable ( $n \ge 50$ ).
Methodological Shift: The author advocates moving beyond simple pairwise hypothesis testing toward explicit modeling (e.g., linear mixed models, Generalizability Theory). These approaches can better handle multiple sources of variance (topics, collections, assessors) and make assumptions transparent, rather than relying on "black box" significance tests.
Broader Impact: The paper warns against importing methodological rules from other fields without scrutiny. It highlights how textbook simplifications can lead to decades of methodological error in a specific domain.

In summary, the paper provides a rigorous statistical and empirical argument that the "safe" alternative (Wilcoxon) is actually the riskier choice in IR, and calls for an immediate shift back to the $t$ -test or more sophisticated modeling frameworks.