📈 economics

Classification testing: A new framework for drawing qualitative conclusions from quantitative estimates

This paper introduces "classification testing," a new statistical framework that allows researchers to assign quantitative estimates to substantively relevant qualitative classes with controlled error rates or declare results inconclusive, arguing that this approach improves upon standard hypothesis testing by better adjudicating rival possibilities and exposing hypotheses to refutation.

Original authors: Andrew C. Eggers, Zikai Li

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrew C. Eggers, Zikai Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of social science, researchers spend their time measuring things that cannot be seen directly: how a new policy changes a community, whether a specific message alters a person's vote, or if a treatment improves a patient's health. They collect data and crunch numbers to find the average effect of these actions. However, the final story they tell is rarely a long string of decimals. Instead, they translate these precise numbers into simple, qualitative judgments. They decide if an effect is real or non-existent, positive or negative, or so small it does not matter. To support these judgments, scientists rely on a standard method called hypothesis testing. This method acts like a gatekeeper, designed to protect the public from false alarms by ensuring that when a researcher claims a result is real, the odds of that claim being a mistake are kept very low.

For decades, this system has worked well for confirming a specific prediction, but it has a significant blind spot. It is built to prove that something is happening, but it struggles to prove that nothing is happening, or to decide between two competing possibilities with equal fairness. If a researcher sets out to prove a treatment works, the system allows them to declare victory if the evidence is strong enough. But if the evidence is weak, the system simply says the researcher failed to prove their point; it does not allow them to confidently declare that the treatment actually does nothing. This creates an uneven playing field where some conclusions are easy to reach while others remain out of reach, potentially skewing the entire scientific conversation toward only the most optimistic findings.

A new framework proposed by political scientists Andrew Eggers and Zikai Li offers a different way to navigate these uncertainties. They call it "classification testing." Instead of forcing a researcher to pick a single favorite theory to defend, this approach asks them to divide all possible outcomes into distinct categories based on what actually matters for the real world. For example, a researcher might divide the possibilities into three groups: the effect is positive and large enough to care about, the effect is negative and large enough to care about, or the effect is so small it is negligible. The goal is not just to see if a number is different from zero, but to sort the result into the right bucket.

The power of this new method lies in its ability to treat all these categories with the same level of statistical rigor. In the old system, a researcher could only confidently say "the effect is positive" if they rejected the idea that it was zero or negative. They could never confidently say "the effect is zero" or "the effect is negative" with the same level of certainty unless they had set up a very specific, often awkward, test beforehand. Classification testing removes this bias. It allows a researcher to look at the data and say, with controlled error rates, that the result falls into the "negligible" category, or the "positive and substantial" category, or that the data is simply too muddy to decide. It turns the scientific question from "Can I prove my hypothesis?" into "What is the most honest description of this result?"

To see how this works in practice, the authors applied their method to a well-known field experiment involving media consumption. The original study examined what happened when regular viewers of Fox News were exposed to CNN. The researchers had measured thirty-two different outcomes, ranging from changes in political preferences to shifts in trust in news sources. When they applied the standard testing method used in most social science journals, they found that sixteen of these outcomes were "significant" and sixteen were not. This left half the results in a state of uncertainty, where the researchers could not confidently say the effect was real or non-existent.

When the authors re-analyzed the same data using classification testing, the picture became much clearer. By sorting the results into categories of "positive and substantial," "negative and substantial," or "negligible," they were able to make confident, error-controlled statements about twenty-nine of the thirty-two outcomes. They could confidently say that for some measures, the effect was large and positive. For others, they could confidently say the effect was so small it was negligible. Even for the results that the old method labeled as "not significant," the new method often allowed them to conclude that the effect was truly negligible, rather than just unknown. The only time they had to say "inconclusive" was when the data was genuinely too noisy to support any firm claim.

This shift changes the incentives for how research is conducted. Under the old rules, researchers were often pressured to find a positive result to get published, leading them to design studies that were likely to confirm their expectations. The new framework encourages a more open approach. Because the method allows for confident conclusions that an effect is small or non-existent, researchers can study genuinely open questions without fear that a "null" result is a failure. It treats the possibility of a small effect as a valid, testable outcome rather than a dead end.

The authors argue that this approach does not make science less strict; in fact, it maintains the same high standards for avoiding false claims. It simply expands the range of questions that can be answered with confidence. By allowing scientists to declare that an effect is negligible with the same certainty they declare it is large, the method reduces the pressure to force data into a specific narrative. It suggests that the most valuable scientific conclusion is not always a dramatic discovery, but sometimes a clear, honest assessment that a particular intervention does not make a meaningful difference. This clarity, the authors suggest, is the key to building a more reliable and less biased understanding of how the social world works.

Technical Summary: Classification Testing: A New Framework for Drawing Qualitative Conclusions from Quantitative Estimates

Problem Statement
The paper identifies a fundamental limitation in current quantitative social science practice: the asymmetry of standard hypothesis testing. Conventional methods (e.g., Wald-style null hypothesis significance testing) are designed to test a single research hypothesis against a null. This structure allows researchers to make error-controlled affirmative conclusions only when they reject the null in favor of their research hypothesis. If the null cannot be rejected, no error-controlled conclusion can be drawn; the research hypothesis cannot be "proven wrong" with the same statistical rigor that it can be "proven right."

This asymmetry creates several issues:

Adjudication Failure: When the goal is to answer an open question (e.g., "Is the effect positive or negative?") rather than test a specific prediction, standard practice forces researchers to arbitrarily designate one possibility as the null, leaving the rival possibility untested.
Incentive Distortion: Because only the research hypothesis can be affirmed, researchers are incentivized to select hypotheses they believe are highly likely to be true to avoid "null results," potentially leading to a literature focused on settled questions rather than genuinely open inquiries.
Inadequate Qualitative Summaries: While social scientists frequently convert quantitative estimates into qualitative conclusions (e.g., "the effect is positive," "negligible," or "substantial"), current testing frameworks do not provide error-controlled guarantees for all such qualitative distinctions simultaneously.

Methodology: Classification Testing
The authors propose "classification testing" as a framework to replace or augment standard hypothesis testing. The core innovation is shifting from testing a single hypothesis to partitioning the parameter space ( $\Theta$ ) into mutually exclusive, logically exhaustive "classes" representing distinct qualitative conclusions.

The Framework:
- Partitioning: The researcher defines classes $C_1, C_2, \dots, C_K$ (e.g., negative, zero/negligible, positive).
- Error Control: The goal is to construct acceptance regions ( $A_j$ ) for each class such that the probability of misclassifying the true parameter $\theta$ into a wrong class is bounded by a significance level $\alpha$ for any value of $\theta$ .
- Optimality: The authors adopt a "maximin" criterion, selecting the classification rule that maximizes the "worst-case power" (the minimum probability of correctly classifying $\theta$ across all classes and all possible values of $\theta$ ).
- Outcome: The test either assigns $\theta$ to a specific class with error control or declares the result "inconclusive" if the data does not strongly support any single class.
Three Specific Test Types:
1. Sign Classification: Partitions $\theta$ into negative ( $C_-$ ) and positive ( $C_+$ ) classes (separated by 0). This is equivalent to running two one-sided tests without multiplicity correction, allowing error-controlled conclusions for both signs or an inconclusive result.
2. Magnitude Classification: Partitions $\theta$ into a "negligible" class ( $C_0 = (-\delta, \delta)$ ) and a "substantial" class ( $C_\pm$ ). Unlike standard equivalence tests (TOST), which can only affirm negligibility, this test allows for error-controlled affirmation of both negligibility and substantiality. The authors derive optimal acceptance regions that adjust for the noise level ( $\sigma/\delta$ ), addressing the conservativeness of TOST.
3. Sign-and-Magnitude Classification: Partitions $\theta$ into three classes: negative/substantial, negligible, and positive/substantial. This combines the objectives of the previous two tests, allowing for refined qualitative distinctions.
Implementation:
- For simple cases (like sign classification), the procedure approximates checking if a $(1-2\alpha)$ confidence interval lies entirely within a class.
- For more complex cases (magnitude and sign-and-magnitude), the authors provide an R package and specific algorithms to calculate optimal acceptance regions that strictly maintain the $\alpha$ error rate, particularly when the estimator is noisy (where simple confidence interval rules fail).
- Combining Tests: The paper outlines rules for combining results from multiple classification tests (e.g., a sign test and a magnitude test) to form a refined conclusion, provided the error guarantees of the component tests apply to the specific bounds of the combined claim.

Key Contributions

Theoretical Unification: The framework unifies and generalizes existing methods such as directional two-sided tests (Kaiser, 1960; Jones and Tukey, 2000), equivalence testing (TOST), and three-sided hypothesis testing (Goeman et al., 2010). It demonstrates that these are specific instances of a broader classification principle.
Improved Power and Symmetry: The authors demonstrate that classification testing offers symmetric error control. It allows researchers to affirm any class (including the "null" or "negligible" class) with the same error guarantees used to reject it in standard practice.
Superiority to TOST: The magnitude classification test is shown to be more powerful than the standard Two One-Sided Tests (TOST) procedure for equivalence testing. It avoids the "conservative" behavior of TOST when standard errors are large relative to the equivalence margin, allowing for affirmative conclusions of substantiality that TOST cannot provide.
Practical Application: The authors provide an R package to facilitate implementation and apply the framework to a re-analysis of the Broockman and Kalla (2025) field experiment.

Results (Application to Broockman and Kalla, 2025)
The authors re-analyzed 32 pre-registered indices from a field experiment regarding the effects of exposing Fox News viewers to CNN.

Standard Two-Tailed Test: Yielded 16 significant and 16 non-significant results.
Classification Testing:
- The Sign Test classified 18 estimands by direction (17 as positive and one as negative), providing error-controlled directional claims for two additional cases that were non-significant in the two-tailed test.
- The Sign-and-Magnitude Test classified 2 indices as "positive and substantial" and 12 as "negligible," leaving 18 as inconclusive.
- Combined Claims: By integrating the sign and magnitude results, the authors produced error-controlled classifications for 29 of the 32 indices (approx. 91%), compared to only 16 (50%) for the standard two-tailed test.
Qualitative Richness: The classification approach provided richer qualitative summaries, distinguishing between effects that were merely "significant" and those that were "substantial," and identifying effects as "negligible" with error control.

Significance and Claims
The paper argues that classification testing is superior to current practice for two primary reasons:

Adjudication: It is the appropriate tool for answering open questions where the goal is to distinguish between rival possibilities rather than to test a single pre-specified hypothesis.
Refutation: Even when testing a single research hypothesis, classification testing is more informative because it exposes the hypothesis to the risk of refutation (affirming a rival class) rather than just the risk of non-affirmation.

The authors suggest that this framework may indirectly improve research incentives. By allowing error-controlled conclusions for a wider range of outcomes (including "negligible" or "negative" findings), it reduces the pressure on researchers to find "positive" results consistent with their hypotheses, potentially encouraging the study of genuinely open questions. The paper concludes that classification testing formalizes the common practice of drawing qualitative conclusions from confidence intervals while providing rigorous, optimal error control that fixed confidence intervals cannot guarantee in all scenarios.

More like this