📊 statistics

Reliability of decisions based on tests: Fourier analysis of Boolean decision functions

This paper employs Fourier analysis of Boolean functions, alongside concepts from test theory and graphical models, to demonstrate that weighted sum scores provide the most reliable and stable decision functions for test outcomes by ensuring robustness against measurement errors and proportional influence of items.

Original authors: Lourens Waldorp, Maarten Marsman, Denny Borsboom

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lourens Waldorp, Maarten Marsman, Denny Borsboom

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge in a high-stakes game show where contestants answer a series of yes-or-no questions. Your job is to decide if they pass or fail. In the world of psychology and education, this is exactly what happens every day with tests: a student answers a bunch of questions, and a score determines if they graduate, get a license, or pass a medical exam. For decades, scientists have relied on complex, invisible "ghosts" (called latent variables) to explain why people answer questions the way they do. They imagine a hidden trait, like "intelligence," floating around and causing the answers. But what if we don't need the ghost? What if we just look at the questions themselves and how they talk to each other? This paper dives into that question, asking: How do we make sure our "Pass/Fail" decision is fair and doesn't flip wildly just because someone made a tiny mistake on one question? To understand this, we need to know about "Boolean functions" (which are just fancy math words for rules that take a bunch of yes/no inputs and spit out a single yes/no output) and "Fourier analysis" (a tool usually used to break down sound waves into notes, but here used to break down decision rules into their most basic parts). The authors want to know if the standard way of scoring tests—just adding up the points—is actually the best, most stable way to make a decision, or if there's a better way to weigh the questions.

The authors of this paper, Lourens Waldorp, Maarten Marsman, and Denny Borsboom, decided to treat a test not as a mystery to be solved by invisible ghosts, but as a network of friends chatting with each other. They used a mathematical technique called Fourier analysis of Boolean functions to investigate how reliable our test decisions really are. Think of a test as a giant, complex machine where every question is a gear. If you nudge one gear (a student changes an answer from right to wrong), does the whole machine stop working and give a different result? Or is the machine sturdy enough that a tiny nudge doesn't matter?

The researchers discovered that the most common way to score tests—simply adding up the correct answers (a "sum score") or using a weighted version of it—is actually a very special and robust choice. They found that this method, which they call a Linear Threshold Function (LTF), is like a sturdy bridge. If a few cars (answers) swerve or make a mistake, the bridge doesn't collapse; the decision to "pass" or "fail" stays the same. In fact, they showed mathematically that if you want a decision rule that is stable (reliable) and treats all questions fairly, the sum score is one of the best tools you can use. It's the only type of rule that satisfies a famous set of fairness criteria known as May's Theorem, which basically says: "If you want a decision that is fair, consistent, and doesn't let one single question hijack the whole result, you should use a sum score."

To figure this out, the team used a clever trick. They imagined a "noise" experiment where they randomly flipped some answers (like a student guessing or making a silly mistake) and watched how the final decision changed. Using their Fourier analysis, they could see exactly how much "influence" each question had on the final verdict. They found that if a question is isolated—meaning it doesn't connect well with the other questions in the test—it has almost no influence, which is a bad sign for a good test. But if the questions are well-connected, the sum score acts like a perfect filter, smoothing out the noise so that the final decision remains stable.

The paper also suggests a new way to think about what a "true score" actually is. Instead of imagining a hidden "intelligence" causing the answers, they propose that a student's true ability is defined by the specific pattern of answers they give to the questions they are directly connected to. It's like saying your "true" opinion on a topic isn't a hidden feeling inside you, but rather the sum of how your friends (the other questions) influence you. This approach allows them to calculate the reliability of a test without needing to believe in those invisible ghosts.

In their simulations, the researchers tested this idea with a made-up test of 35 questions. They found that even when they couldn't perfectly map out which questions were connected to which (the "graph" of the test), the Fourier analysis still gave them accurate results about how stable the test was. They showed that as long as the test is "balanced"—meaning no single question is too powerful or too weak—the sum score remains the most reliable decision-maker. If a test has a "dictator" question (one that decides the outcome all by itself), the decision becomes unstable and unfair. But with a balanced sum score, the decision is robust, meaning a few mistakes won't ruin a student's chance of passing.

Ultimately, this paper argues that we don't need to rely on complicated, unproven theories about hidden traits to know if a test is good. By looking at the math of how the questions interact, we can prove that the simple act of adding up scores is a scientifically sound, stable, and fair way to make life-changing decisions. It turns out that sometimes, the simplest way to count the votes is also the most reliable way to decide the winner.

Technical Summary: Reliability of Decisions Based on Tests via Fourier Analysis of Boolean Functions

Problem Statement
Tests are frequently utilized to make binary decisions (e.g., pass/fail) based on item responses. Traditionally, the psychometric properties of such tests are evaluated through latent variable models (e.g., Item Response Theory), which assume unobserved constructs causally influence observed items. However, the authors argue that these models often rely on unrealistic assumptions, such as the existence of latent variables or infinite tail events. Furthermore, it is often unclear how small perturbations in item responses (measurement error) affect the final decision. The paper seeks to determine the requirements for an appropriate decision function without relying on latent variable models. Specifically, it investigates whether the standard weighted sum score is the optimal method for aggregating items and what properties make a decision function reliable (stable) against noise.

Methodology
The authors employ Fourier analysis of Boolean functions to analyze decision rules. A decision function $f$ is modeled as a mapping from $n$ binary items (encoded as $\{-1, 1\}$ ) to a binary outcome. The methodology proceeds through several key steps:

Graphical Modeling Framework: To accommodate dependencies between items (violating the assumption of independent and identically distributed variables), the authors embed the analysis within graphical models, specifically the Ising model. They utilize a pseudo-likelihood factorization (the product of conditional probabilities) rather than the joint probability. This allows for the definition of local conditional independence, where an item depends only on its neighbors in the graph.
p-Biased Fourier Analysis: Standard Fourier analysis assumes uniform distributions. The authors extend this to a p-biased setting where items have different probabilities of success ( $p_i$ ). They define a transformed basis function $\phi(x_i) = (x_i - \mu_i)/\sigma_i$ using conditional expectations ( $\mu_i$ ) and variances ( $\sigma_i$ ) derived from the Ising model's conditional distributions. This ensures orthogonality of the basis functions under the pseudo-likelihood measure.
Stability and Reliability Definitions:
- Influence: The influence of an item $i$ on the decision is defined as the probability that flipping the value of item $i$ changes the decision outcome.
- Reliability/Stability: The authors introduce a noise model where each item is flipped with probability $\frac{1}{2}(1-\rho)$ . Decision reliability is defined as the correlation between the decision on the original data and the noisy data. Stability is the difference in probability between the decisions remaining the same versus changing.
Estimation Algorithm: To apply these techniques to real data, the authors propose an algorithm:
- Estimate Ising model parameters (thresholds and edge weights) using Lasso penalized pseudo-likelihood (nodewise logistic regression).
- Use these estimates to compute conditional probabilities and the transformed variables $\phi(x_i)$ .
- Calculate Fourier coefficients, specifically focusing on singleton sets (first-order coefficients) to approximate decision reliability and item influence.
- Theoretical guarantees are provided using desparsified Lasso to establish convergence rates ( $1/\sqrt{m}$ ) for the estimates.

Key Contributions and Results
The paper establishes several theoretical and practical results regarding decision functions:

Characterization of Linear Threshold Functions (LTFs): The authors demonstrate that Linear Threshold Functions (which include weighted sum scores) possess desirable properties:
- May's Theorem: LTFs (specifically the majority function) are the unique functions satisfying monotonicity, oddness, and anonymity.
- Stability: In balanced tests (where items have similar influence/degree in the graph), LTFs are the most stable decision functions against noise. The stability of the majority function is shown to be $S_\rho(\text{maj}_n) = \frac{2}{\pi}\arcsin(\rho)$ as $n \to \infty$ , which is higher than any other function with small item influences.
- Rousseau's Criterion: LTFs maximize the agreement between the decision and the sum of items that are above their average probability, effectively maximizing the total influence of items on the decision.
True Score Definition: The authors propose a definition of the "true score" based on the Ising model's linear threshold function ( $\ell_{\beta_i}$ ) representing the "energy" of a node given its neighbors. This definition is observable (unlike latent variables) and aligns with conditions of monotonicity, local independence, and conditional association.
Item Influence and Graph Structure: The Fourier coefficients of singleton sets are shown to correlate with the degree of nodes in the item dependency graph. Items with low influence (low Fourier coefficients) correspond to isolated nodes in the graph, suggesting they should be excluded from the test.
Numerical Illustrations: Simulations using $n=35$ $n = 35$ items and $m=100$ $m = 100$ observations demonstrate that:
- Fourier coefficients correlate highly (0.70) with node degrees.
- Decision stability increases with the correlation $\rho$ of the noise and decreases with higher decision thresholds.
- Even when the underlying graph structure is not perfectly recovered (low signal-to-noise ratios), the estimates of conditional probabilities and Fourier coefficients remain accurate, suggesting the method is robust to graph recovery errors.

Significance and Claims
The paper claims to shift the focus of test evaluation from latent variable models to observable decision functions. Its primary significance lies in providing a mathematical framework to:

Evaluate Decision Stability: Quantify how robust a pass/fail decision is to measurement error without assuming a latent trait.
Justify the Sum Score: Offer theoretical arguments (based on stability, Rousseau's criterion, and test theory connections) that weighted sum scores (LTFs) are the most appropriate and stable aggregation methods for binary decisions.
Diagnose Item Quality: Use Fourier coefficients to identify items that are "isolated" (have low influence) or disproportionately dominant, thereby aiding in test construction and item selection.

The authors conclude that a weighted sum score is an appropriate basis for binary decisions because it fits test theory (via observable conditional association), ensures decision stability, and satisfies the criterion that the decision should match the input direction. They suggest that this framework allows for individual-level definitions of true scores based on time-series data, though this is presented as a potential extension rather than a current application.

Technical Summary: Reliability of Decisions Based on Tests via Fourier Analysis of Boolean Functions

More like this