📈 economics

Manipulation testing based on Benford's Law for discrete scores

This paper introduces a novel manipulation testing framework for Regression Discontinuity Designs that leverages Benford's Law to detect structural imbalances in running variables by analyzing directional density components, offering a parameter-free, more sensitive alternative to traditional McCrary-type tests.

Original authors: Roy Cerqueti, Marco Ventura

Published 2026-07-16✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Roy Cerqueti, Marco Ventura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out if a game is being played fairly. In the world of economics and social science, researchers often use a clever trick called a "Regression Discontinuity Design" (RDD) to see if a specific policy actually works. Think of it like a school scholarship: if you score 90 or higher on a test, you get a free trip; if you score 89, you get nothing. The idea is that students who score 89 and 90 are almost identical, so if the 90s do better later in life, it must be because of the trip, not because they were smarter to begin with. But here's the catch: what if someone cheats? What if a student who scored 89 somehow "manipulated" their score to get a 90? If that happens, the whole experiment is ruined because the two groups aren't identical anymore.

To catch cheaters, scientists usually look at the "density" of scores around the cutoff line. In a fair game, the number of people scoring just below the line should roughly match the number of people scoring just above it, creating a smooth, continuous hill of data. If there's a sudden jump or a gap right at the line, it looks like someone pushed the data. However, traditional tools for spotting this jump have a blind spot: they often assume the data is perfectly smooth and continuous, like a flowing river. But in the real world, scores are often "discrete," meaning they are like steps on a staircase—you can get a 90 or a 91, but you can't get 90.5. When the data is a staircase, the old tools sometimes miss the cheating because they are looking for a smooth river that doesn't exist.

This is where a new paper by Roy Cerqueti and Marco Ventura steps in with a fresh idea. They propose using a mathematical rule called Benford's Law to catch manipulation in these "staircase" scenarios. You might know Benford's Law from crime dramas; it's the observation that in many naturally occurring lists of numbers (like populations of cities or electricity bills), the first digit isn't random. Instead, the number 1 appears as the first digit about 30% of the time, while 9 appears less than 5% of the time. It's a natural pattern that emerges from how numbers grow in the real world. The authors realized that if someone fakes data to push scores over a cutoff, they break this natural pattern.

The paper suggests a new way to test for cheating that doesn't rely on the researcher guessing how to set up their tools. Instead, it uses Benford's Law to automatically find the perfect "window" of data to look at, right around the cutoff line. They then split the data into two groups: those just below the line and those just above. They check if the "steps" on the staircase follow the natural Benford pattern on both sides. If the pattern looks different on one side compared to the other, it suggests an asymmetry in the data distribution.

In their simulations, the authors found that their new method is like a high-powered microscope compared to the old tools. While the traditional tests often missed the asymmetry when the data was discrete (like a staircase), this new method successfully spotted it. They tested it on two real-world examples: one about political elections in Turkey and another about college scholarships in Colombia. In both cases, the traditional tests confirmed that the data showed continuity (meaning no obvious manipulation was found), but their new method detected subtle signs of asymmetry in the distribution that the old tools missed. The authors suggest that this doesn't necessarily mean the studies were fake or that manipulation definitely occurred, but it does mean researchers should be extra careful. They argue that this new test should be used alongside the old ones as a "precautionary safeguard," offering a more detailed look at whether the data is truly behaving naturally or if there are structural imbalances that need further investigation.

Technical Summary: Manipulation testing based on Benford's Law for discrete scores

Problem Statement
Regression Discontinuity Designs (RDD) rely on the critical identifying assumption that units cannot perfectly manipulate the running variable (score) to cross a treatment cutoff. Traditional validation methods, such as the McCrary test and its extensions (Otsu et al., 2013; Cattaneo et al., 2020), assess this by testing for continuity in the density of the running variable around the cutoff. However, these methods face significant limitations when applied to discrete scores, which are common in empirical applications. In discrete settings, the concept of continuity is inapplicable, and researchers must rely on weaker concepts of asymmetry or simple counting procedures (Frandsen, 2017; Cattaneo et al., 2024). Furthermore, standard density-based tests often require researchers to select discretionary parameters (e.g., polynomial orders, kernel functions, bandwidths) that can skew outcomes. Additionally, the rigid symmetric setup of classic McCrary tests may overlook systemic imbalances if the manipulation is directional rather than symmetric.

Methodology
The authors propose a novel manipulation test grounded in Benford's Law (BL), a natural statistical regularity describing the non-uniform distribution of leading digits in many real-world datasets. The methodology is designed specifically for discrete running variables and proceeds in two main phases:

Optimal Bandwidth (BW) Selection via Iterative Procedure:
Unlike traditional methods where the researcher chooses the bandwidth, this approach derives an optimal, data-driven BW using an iterative algorithm.
- The procedure partitions the support of the running variable ( $R_i$ ) into left ( $g^{(-)}$ ) and right ( $g^{(+)}$ ) sets relative to the cutoff ( $C$ ).
- It iteratively defines endogenous thresholds ( $X^{(-)}_n, X^{(+)}_n$ ) such that the probability mass within these intervals conforms to the Benford distribution ( $B_j = \log_{10}(1 + 1/j)$ ).
- The iteration continues until the empirical distribution of the data within the derived intervals minimizes the Mean Absolute Deviation (MAD) from the theoretical Benford distribution, subject to a pre-defined acceptance level $\alpha$ .
- This yields an optimal bandwidth defined by the thresholds $(X^{(-)}_{n^*,1}, X^{(+)}_{n^*,1})$ that best satisfy BL properties without researcher discretion.
Asymmetry Testing:
Within the optimal BW, the authors propose two complementary methods to detect manipulation by assessing asymmetry:
- Tolerance Threshold Approach: This method compares the endogenous thresholds on the left ( $X^{(-)}_j$ ) and right ( $X^{(+)}_j$ ) of the cutoff. Under perfect symmetry, the distance from the cutoff to corresponding thresholds should be equal ( $C - X^{(-)}_j = X^{(+)}_j - C$ ). The method normalizes these distances into probabilities and calculates a distance metric $D$ between the two sets of probabilities.
- MAD Approach: This method constructs symmetric thresholds ( $Y^{(-)}_k, Y^{(+)}_k$ ) relative to the cutoff (e.g., $Y^{(+)}_k$ is the point on the right equidistant to $X^{(-)}_k$ on the left). It then calculates the empirical probability mass in these symmetric intervals and compares the resulting distributions to the Benford distribution using separate MAD statistics for the left ( $MAD^{(-)}$ ) and right ( $MAD^{(+)}$ ) sides.
- Diagnostic Granularity: A key feature is the ability to detect "partial symmetry." If $MAD^{(+)}$ indicates asymmetry while $MAD^{(-)}$ indicates symmetry (or vice versa), the test can pinpoint the specific direction of manipulation, distinguishing between cases where units manipulate scores to cross the cutoff versus cases where they do not.

Key Contributions

Discrete Score Applicability: The framework addresses the gap in manipulation testing for discrete running variables where continuity assumptions fail.
Elimination of Discretionary Parameters: By using Benford's Law to determine both the optimal bandwidth and the expected density thresholds, the method removes researcher-chosen parameters (kernel, polynomial order, bandwidth selection) that often influence the results of McCrary-type tests.
Directional Sensitivity: Unlike symmetric density tests, this approach separates data into directional components, allowing researchers to identify the exact origin of deviations and detect hidden manipulation that standard frameworks might miss.
Theoretical Adaptation: The authors adapt BL from a law of digits to a discrete probability distribution of thresholds, overcoming limitations regarding dataset cardinality and bounded ranges often associated with standard BL applications.

Results

Simulations: In a simulation exercise using a perfectly symmetric Normal distribution with a shifted cutoff (creating an asymmetric bandwidth), the proposed method successfully detected the induced asymmetry.
- The method correctly identified that the left side (shorter interval) conformed to symmetry ( $MAD^{(-)}$ was low), while the right side (longer interval) showed significant deviation ( $MAD^{(+)}$ was high).
- In contrast, a standard McCrary-type test applied to the same asymmetric interval failed to reject the null hypothesis of symmetry (p-value = 0.850), demonstrating the superior diagnostic power of the BL-based approach in detecting nuanced asymmetries.
Empirical Applications:
- Meyersson (2014) Dataset: Analyzing the margin of victory in Turkish municipal elections, the test identified significant right-sided asymmetry ( $MAD^{(+)} > MAD^{(-)}$ ), suggesting manipulation or concentration of units on the right side of the cutoff.
- Londoño-Vélez et al. (2020) Dataset: Analyzing the SISBEN wealth index in Colombia, the test detected greater left-sided asymmetry ( $MAD^{(-)} > MAD^{(+)}$ ), consistent with the incentive structure where units below the cutoff are eligible.
- In both cases, the method provided a more granular breakdown of density behavior than traditional tests, identifying specific directional deviations.

Significance and Claims
The paper positions this methodology as a vital precautionary safeguard and a necessary complement to traditional McCrary-type tests. The authors claim that while the McCrary test remains the industry standard, its reliance on symmetric bandwidths and continuous density assumptions limits its diagnostic power, potentially leading to false negatives when underlying asymmetries exist.

By leveraging the universal regularity of Benford's Law, the proposed framework offers a "more prudent" and "granular lens" for evaluating density behavior. It allows researchers to:

Avoid arbitrary parameter choices.
Detect manipulation even when the overall density appears continuous or when the manipulation is directional.
Distinguish between "left-symmetry" and "right-symmetry," providing economic context to the statistical findings (e.g., determining if manipulation is driven by incentives to cross the cutoff).

The authors conclude that this approach enhances the robustness of RDD validity checks, particularly in discrete settings, though they acknowledge the need for further research to establish clearer operational thresholds for MAD values dependent on sample size.

More like this