📊 statistics

Balancing Covariates in Survey Experiments

This paper proposes a stratified rejective sampling and rerandomization design to address covariate imbalance in survey experiments, establishing a design-based asymptotic theory that demonstrates improved estimation efficiency and a more concentrated limiting distribution for the average treatment effect compared to existing methods.

Original authors: Pengfei Tian, Jiyang Ren, Yingying Ma

Published 2026-05-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengfei Tian, Jiyang Ren, Yingying Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to test a new recipe (the "treatment") to see if it makes a cake taste better than the old one (the "control"). You want to know if the new recipe actually causes the improvement, or if the cake just happened to be better because you used better eggs or a hotter oven.

In the world of social science and economics, this is called a survey experiment. Researchers want to test policies or ideas on a group of people to see what works. But there's a catch: if the group of people you pick isn't perfectly balanced, your results might be misleading.

This paper by Tian, Ren, and Ma proposes a new, super-precise way to run these experiments to ensure the results are fair and accurate. Here is how they do it, broken down into simple steps:

1. The Problem: The "Unbalanced Plate"

Imagine you are picking 100 people to taste your new cake.

Simple Random Sampling: You close your eyes and pick 100 people. On average, this works. But in a specific group of 100, you might accidentally pick 80 people who love chocolate and only 20 who hate it. If your new recipe has chocolate, you can't tell if the cake is good because of the recipe or because you picked too many chocolate lovers.
The Issue: Even with random selection, small groups often end up unbalanced. The "treatment" group might have more educated people, or the "control" group might have more young people. This imbalance creates noise, making it hard to see the true effect of the recipe.

2. The Old Solution: "Sorting into Bins" (Stratification)

To fix this, researchers usually use Stratification.

The Analogy: Before picking people, you sort the whole population into bins based on important traits (like age, income, or education). Then, you pick an equal number of people from each bin.
The Result: This ensures your sample looks like the real population. It's like making sure your cake-tasting panel has the exact same ratio of chocolate-lovers to vanilla-lovers as the whole city. This is much better than just closing your eyes, but it doesn't catch every imbalance, especially for traits you didn't sort by.

3. The New Solution: The "Triple-Check" System (SRSRR)

The authors propose a three-step "Triple-Check" system called Stratified Rejective Sampling and ReRandomization (SRSRR). Think of this as a strict quality control process with three layers of security:

Layer 1: The Bins (Stratification)
Just like the old method, you first sort everyone into bins (strata) based on major traits. This is your foundation.
Layer 2: The "Reject" Filter (Rejective Sampling)
After you pick your initial group from the bins, you check them against the entire population.
- The Metaphor: Imagine you have a checklist of 50 traits (age, income, voting history, etc.). You pick your 100 people, run a computer check, and ask: "Does this group look exactly like the city?"
- The Action: If the group is slightly off-balance (e.g., too many people from one neighborhood), you reject the whole group and start over. You keep picking new groups until you find one that passes the checklist. This ensures the sample is perfectly representative.
Layer 3: The "Re-Roll" (ReRandomization)
Once you have a perfect group of 100 people, you need to split them into the "New Recipe" group and the "Old Recipe" group.
- The Metaphor: You shuffle the deck of 100 people and deal them into two piles. But before you accept the deal, you check: "Are the two piles balanced?"
- The Action: If the "New Recipe" pile accidentally has more chocolate lovers than the "Old Recipe" pile, you reject that split. You shuffle and deal again until the two piles are perfectly balanced on all those 50 traits.

Why is this better?
The paper shows that doing both the "Reject" step (for the sample) and the "Re-Roll" step (for the assignment) creates a much tighter, more precise result than doing just one or neither. It's like using a high-precision scale instead of a guess.

4. The "Math Magic" (The Results)

The authors didn't just guess this would work; they did the heavy math to prove it.

The Shape of the Truth: In standard experiments, the results follow a "Bell Curve" (Normal Distribution). The authors found that with their new Triple-Check method, the results follow a "Bell Curve with a Squeeze."
What that means: The results are still centered on the truth, but they are tighter. Imagine shooting arrows at a target. Standard methods might have your arrows spread out in a wide circle. The SRSRR method squeezes all your arrows into a tiny, tight cluster right in the bullseye. This means you need fewer people to get the same level of confidence, or you get a much more precise answer with the same number of people.

5. The Final Polish: "Adjusting the Recipe"

Even with the Triple-Check, there might be tiny, tiny imbalances left over. The authors suggest a final step in the analysis phase called Covariate Adjustment.

The Analogy: If you notice your "New Recipe" group had slightly more chocolate lovers by a tiny margin, you can use a mathematical formula to "subtract" that advantage from the final score. It's like adjusting the final taste score to account for the fact that the group was slightly biased.
The Result: This makes the estimate even more efficient, squeezing that cluster of arrows even tighter.

Summary

The paper argues that to get the most accurate answer about whether a new policy or program works, you shouldn't just rely on luck. You should:

Sort people into groups (Stratification).
Reject bad samples that don't match the population (Rejective Sampling).
Re-roll the assignment until the treatment and control groups are perfectly matched (ReRandomization).
Adjust the final numbers for any tiny remaining differences.

By combining these steps, researchers can get a clearer, more reliable picture of cause-and-effect, saving time and resources while avoiding false conclusions.

Technical Summary: Balancing Covariates in Survey Experiments

Problem Statement
Survey experiments are widely regarded as the gold standard for estimating treatment effects in target populations, combining random sampling with random treatment assignment. While simple random sampling and complete randomization balance covariates on average, finite samples often exhibit covariate imbalance. This imbalance occurs both between the sampled units and the target population (sampling stage) and between the treatment and control groups (assignment stage). Such imbalances can lead to conditional bias and increased variability in treatment effect estimates. While stratification is a standard technique to improve representativeness and efficiency, and rejective sampling or rerandomization are used to balance covariates, existing literature has not fully explored the design-based asymptotic theory for combining these methods in a general survey experiment setting, particularly when allowing for diverging numbers of strata and vanishing sampling fractions.

Methodology
The authors propose a Stratified Rejective Sampling and ReRandomized (SRSRR) experimental design. This two-stage approach integrates three distinct mechanisms:

Stratification: The population is partitioned into strata based on predictive covariates.
Stratified Rejective Sampling: In the sampling stage, units are drawn via stratified random sampling. However, samples are rejected if the Mahalanobis distance between the weighted sample mean of pre-sampling covariates ( $W$ ) and the population mean exceeds a pre-specified threshold ( $a_S$ ). This ensures the sample is representative of the population regarding $W$ .
Stratified ReRandomization: In the treatment assignment stage, sampled units are assigned to treatment or control via stratified randomization. Assignments are rejected if the Mahalanobis distance between the treatment and control group means of assignment-stage covariates ( $X$ ) exceeds a threshold ( $a_T$ ).

The paper develops a design-based asymptotic theory for the stratified difference-in-means estimator ( $\hat{\tau}$ ) under this design. The framework allows for a general asymptotic regime where the number of strata ( $K_N$ ) and stratum sizes ( $N_{[k]}$ ) can diverge, and the sampling fraction ( $f = n/N$ ) can tend to zero. The authors employ a Hájek coupling technique to reduce the estimator to a sum of independent random variables, handling the joint randomness of sampling and treatment assignment.

Furthermore, the authors propose a covariate adjustment method in the analysis stage. This linearly adjusted estimator ( $\hat{\tau}_{adj}$ ) utilizes additional covariates collected at the analysis stage ( $E$ for the population and $C$ for the sample) to further reduce residual imbalance.

Key Contributions

Design-Based Asymptotic Theory for Stratified Randomization: The paper establishes that under stratified random sampling and assignment, the stratified difference-in-means estimator is consistent and asymptotically normal. It derives the asymptotic variance and provides conditions for optimal stratum-specific sampling and treatment proportions to minimize this variance.
Asymptotic Theory for SRSRR: The authors derive the limiting distribution of $\hat{\tau}$ under the SRSRR design. Unlike standard designs, this limiting distribution is not normal; it is shown to be a convolution of a normal distribution and two truncated normal distributions. This distribution is more concentrated around the true average treatment effect than that of standard stratified designs.
Optimal Design Parameters: The paper provides theoretical results for determining optimal stratum-specific sampling proportions ( $f_{[k]}$ ) and treatment proportions ( $e_{[k]1}$ ) to minimize asymptotic variance, both with and without covariate adjustment.
Covariate Adjustment: The study demonstrates that a linearly adjusted estimator using analysis-stage covariates is asymptotically more efficient than the unadjusted estimator under both standard stratified and SRSRR designs. It provides conservative variance estimators and valid confidence intervals for these adjusted estimators.

Results

Theoretical Findings:
- The asymptotic variance of the SRSRR estimator is strictly smaller than that of the stratified randomized survey experiment (SRSE), with the reduction depending on the squared multiple correlations between the treatment effect estimator and the covariate imbalances ( $R^2_W$ and $R^2_X$ ).
- The limiting distribution of $\sqrt{n}(\hat{\tau} - \tau)$ under SRSRR is a convolution of a normal variable and two truncated normal variables, which are more concentrated at zero than the corresponding normal variables.
- The proposed covariate-adjusted estimator ( $\hat{\tau}_{adj}$ ) achieves the smallest asymptotic variance among the class of linearly adjusted estimators.
Numerical Studies:
- Simulations across various scenarios (small strata, large strata, and mixed) confirm that SRSRR significantly reduces the standard deviation and root mean squared error (RMSE) compared to completely randomized and standard stratified designs.
- In synthetic data experiments, combining stratification, rejective sampling, and rerandomization reduced the standard deviation by approximately 30.6% relative to a completely randomized baseline.
- Regression adjustment provided further efficiency gains, reducing standard deviations by 21% to 49% relative to unadjusted estimators.
- Real-data analysis using the Cooperative Congressional Election Study (CCES) corroborated these findings, showing that SRSRR and covariate adjustment lead to shorter confidence intervals and improved efficiency compared to non-stratified or unadjusted approaches.

Significance
The paper claims that the proposed SRSRR design and associated analysis methods offer a robust framework for improving the precision of treatment effect estimates in survey experiments. By explicitly addressing covariate imbalance at both the sampling and assignment stages, the method enhances the representativeness of the sample and the balance between treatment groups. The development of a design-based asymptotic theory that accommodates diverging strata and vanishing sampling fractions extends the applicability of these methods to a wide range of practical survey settings. The authors emphasize that their approach is purely design-based, meaning the validity of the estimates and inference procedures does not rely on the correct specification of an underlying outcome model. The proposed staged workflow—stratification, followed by rejective sampling and rerandomization, and finally regression adjustment—is presented as a practical strategy to maximize estimation efficiency.