📊 statistics

Power-Optimal Covariate Adjustment for Switchback Experiments

This paper proposes a power-optimal CUPAC methodology for switchback experiments with unequal cluster sizes that improves statistical power by specifically balancing the prediction of noise between and within randomization units, rather than merely targeting overall predictive accuracy.

Original authors: Sergei Pankratev

Published 2026-09-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Sergei Pankratev

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of online platforms, where millions of transactions happen every hour, companies rely on experiments to decide if a new feature actually works. Imagine a food delivery app testing a new way to route drivers. To know if the change helps, the company cannot simply try it on half its users and ignore the other half; the users are too interconnected, and the timing of their orders matters. Instead, researchers use a method called a switchback experiment. In this setup, the entire system switches between the old way and the new way in short bursts, like a lighthouse beam sweeping across the water. Every hour, or every few minutes, the whole network flips to the new method, then flips back. This creates a series of time blocks where the entire system is treated as a single unit.

The goal of these experiments is to measure the difference in outcomes, such as how long it takes to deliver food, with as much precision as possible. To get a clear signal from the noise, data scientists often use a technique called covariate adjustment. Think of this as using a weather forecast to predict today's temperature. If you know the temperature yesterday and the season, you can predict today's temperature much better than if you just guessed. In an experiment, scientists use data from before the test started to predict what would have happened without the new feature. By comparing the actual results to these predictions, they can strip away the random fluctuations and see the true effect of the change. This process is standard practice, but it assumes that every piece of data is equally important.

A new study by Sergei Pankratev at DoorDash challenges this assumption for switchback experiments. The research reveals that in these specific types of tests, the standard way of using past data to clean up results is actually missing the most important part of the story. In a switchback experiment, the data comes in chunks: a specific group of drivers and customers during a specific hour. These chunks, or cells, vary wildly in size. Some hours are busy with thousands of orders; others are quiet with only a few. The standard method treats every single order as an equal data point, trying to predict the outcome for each individual order. However, because the experiment switches the entire system at once, the real source of uncertainty is not the individual orders, but the differences between these time chunks. The standard method spends its effort trying to predict the noise of individual orders, which the experiment design already averages out, while ignoring the larger swings between the time blocks that actually determine whether the experiment succeeds or fails.

Pankratev developed a new approach that fixes this misalignment. Instead of training the prediction model to be accurate for every single order, the new method trains the model to be accurate for the average outcome of each time block. It forces the computer to pay attention to the big picture: how the system behaves during a busy hour versus a quiet one. The study shows that this shift in focus is not just a minor tweak; it is a fundamental reweighting of what the model learns. The researchers found that in situations where the individual orders are highly unpredictable, the standard method fails to reduce the uncertainty of the experiment. It leaves the results fuzzy and makes it hard to tell if a change is real. The new method, by contrast, sharpens the focus on the time blocks, significantly reducing the uncertainty and making the experiment much more powerful.

To prove this, the researchers ran thousands of simulated experiments on a computer. They created a digital world with 200 different locations and 24 hours of activity, generating 4,800 distinct time blocks. In these simulations, they tested two different ways of training the prediction model. One group used the traditional method, which treats every order equally. The other group used the new method, which prioritizes the accuracy of the time-block averages. They also varied the complexity of the computer models, making them simple with few decision points and complex with many. The results were clear and consistent. When the individual orders were chaotic and unpredictable, the traditional method struggled. Even when the researchers made the computer models much larger and more powerful, the traditional method did not get much better. It was like giving a better telescope to someone looking at the wrong part of the sky; the extra power was wasted because the target was misaligned.

The new method, however, thrived in these chaotic conditions. By focusing on the time blocks, the model learned to predict the swings in the system that mattered most. As the models grew larger, the new method became even more effective, steadily reducing the uncertainty of the experiment. The study showed that this improvement translated directly into a higher chance of detecting a real effect. In the most difficult scenarios, where the noise was highest, the new method increased the statistical power of the experiment by 26 to 35 percentage points compared to the old way. This means that with the same amount of data, a company could be much more confident in its results, or it could achieve the same confidence with far fewer hours of testing.

The research also addressed a second part of the process: how the final numbers are calculated after the experiment is over. The study found that using the new training method is not enough on its own. If the model is trained to focus on time blocks, but the final calculation still treats every order as equal, the benefits are lost. The researchers showed that the calculation step must also be adjusted to match the training. When both the training and the calculation are aligned to focus on the time blocks, the experiment reaches its full potential. If they are mismatched, the gains disappear. This two-step alignment ensures that the effort put into training the model is fully realized in the final answer.

The findings suggest that for companies running these types of experiments, the way they prepare their data is just as important as the experiment design itself. The study does not claim that the old method is useless; in situations where the system is very stable and the time blocks are similar, the old method works fine. But in the messy, real world of online platforms where some hours are packed and others are empty, the old method leaves a lot of value on the table. The new approach offers a way to extract more clarity from the same data, turning a noisy signal into a clear answer. It is a reminder that in science, the tools we use to measure the world must be tuned to the specific nature of the world we are measuring. By reweighting the focus from the individual to the group, the researchers have provided a more precise lens for seeing how changes truly affect the systems we rely on every day.

Technical Summary: Power-Optimal Covariate Adjustment for Switchback Experiments

Problem Statement

In online experimentation, switchback designs assign treatment to spatial clusters crossed with time windows (cells) rather than individual units. While standard variance reduction techniques like CUPED (Control Using Pre-Experiment Data) and CUPAC (Control Using Prediction As Covariate) are effective in standard A/B tests where randomization and analysis occur at the same unit level, they are suboptimal in switchback experiments.

The core issue arises from unequal cluster sizes and the hierarchical structure of the data. In switchbacks, the variance of the treatment-effect estimator is dominated by between-cell variation (variation across randomization units) rather than within-cell idiosyncratic noise. Standard CUPAC models are trained to minimize unit-level Mean Squared Error (MSE). Because unit-level MSE implicitly weights within-cell and between-cell errors equally, the model allocates its finite capacity to reducing within-cell noise—variation that the experimental design naturally averages away—while neglecting the between-cell signal that actually governs statistical power. Furthermore, the standard analysis-time adjustment coefficient is typically estimated via ordinary least squares (OLS) at the unit level, which repeats this misalignment by under-weighting the between-cell relationship.

Methodology

The paper proposes a unified framework for Power-Optimal CUPAC that aligns both the covariate training objective and the analysis-time adjustment with the specific variance decomposition of the switchback estimator.

1. Theoretical Framework and Variance Decomposition

The author formalizes the outcome $y_{b,i}$ for unit $i$ in cell $b$ as a multilevel model containing macro-level effects (spatial cluster, time window, and interactions) and idiosyncratic residual noise. They derive an exact decomposition of the treatment-effect estimator's variance:
$\text{Var}(\hat{\tau}) \approx \frac{4}{JH} \left[ \frac{S_{res}}{\bar{n}} + S_{macro} \left( \frac{1}{\bar{n}} + 1 + cv^2 \right) \right]$
Where:

$\bar{n}$ is the mean cell density.
$cv^2$ is the squared coefficient of variation of cell sizes.
$S_{res}$ and $S_{macro}$ represent the shares of variance within and between cells, respectively.

The term $1 + \bar{n}(1 + cv^2)$ acts as a penalty factor that inflates the weight of between-cell variance relative to within-cell variance. In typical large-scale settings, this factor can be hundreds of times larger than 1, meaning between-cell errors are vastly more costly to power than within-cell errors.

2. Power-Optimal Training Loss

To minimize the estimator variance, the control variate $g$ must be trained to minimize a specific weighted sum of prediction errors rather than standard unit-level MSE. The author defines the Power-Optimal Loss ( $L_{power}$ ):
$L_{power}(g) = \frac{1}{\bar{n}} \text{MSE}_{within}(g) + \left( \frac{1}{\bar{n}} + 1 + cv^2 \right) \text{MSE}_{macro}(g)$
This loss function explicitly up-weights the between-cell (macro) prediction error by the factor $1 + \bar{n}(1 + cv^2)$ . This forces the model to prioritize learning the signal that varies across randomization units, which is the component that determines statistical power.

3. Matched Analysis-Time Coefficient

The paper demonstrates that training with $L_{power}$ is insufficient if the analysis-time adjustment coefficient $\theta$ is estimated using standard unit-level OLS. The optimal coefficient $\theta^*$ must also be estimated using a loss function that matches the training weights. The author shows that the standard unit-level OLS coefficient under-weights the between-cell covariance by the same factor $1 + \bar{n}(1 + cv^2)$ , effectively undoing the gains from the optimized training. The proposed solution is to estimate $\theta$ using the same composite loss used for training.

Key Contributions

Identification of Misalignment: The paper identifies that standard CUPAC workflows are misaligned with switchback designs because they treat within-cell and between-cell errors as equally important, despite the design making between-cell errors orders of magnitude more critical for power.
Derivation of Power-Optimal Loss: The author derives a closed-form training loss ( $L_{power}$ ) that is an affine functional of the estimator variance. Minimizing this loss directly minimizes an upper bound on the realized estimator variance.
Two-Stage Alignment: The work establishes that both the training stage (loss function) and the analysis stage (coefficient estimation) must be re-weighted by the same design-specific factor to achieve maximum power.
Theoretical Bounds: The paper proves that minimizing $L_{power}$ provides a tight upper bound on the variance of the estimator even when the analysis coefficient is re-optimized, provided the covariate is calibrated.

Simulation Results

The author validates the framework using a Monte Carlo simulation with gradient-boosted trees (LightGBM) across varying model capacities (number of leaves) and variance regimes ( $S_{macro}$ ).

Variance Regime Dependence: When within-cell noise dominates ( $S_{macro} = 0.15$ ), standard unit-level MSE performs poorly (relative standard error $\approx 0.85$ ), while the power-optimal loss paired with the matched coefficient achieves a relative standard error of $\approx 0.51$ . When between-cell variation is high ( $S_{macro} = 0.50$ ), the gap narrows, but the optimal method remains superior.
Model Capacity: Increasing model size (from 4 to 255 leaves) does not compensate for a misaligned training objective. A model trained on unit-level MSE fails to improve its between-cell correlation ( $\rho_m$ ) regardless of size because the greedy splitting criterion prioritizes within-cell variance reduction. Conversely, the power-optimal loss effectively utilizes increased capacity to improve $\rho_m$ .
Coefficient Matching: The benefit of the matched coefficient ( $\theta^*$ ) is most pronounced when model capacity is limited. As models grow larger and fit both levels well, the difference between unit-level and matched coefficients diminishes.

Significance and Claims

The paper claims that the proposed framework provides a theoretically grounded, implementable solution to maximize statistical power in switchback experiments without requiring changes to the experimental design itself.

Practicality: The method requires only the calculation of two design constants ( $\bar{n}$ and $cv^2$ ) from the cell-size distribution to re-weight the training loss and analysis coefficient.
Efficiency: It allows practitioners to extract maximum power from existing data, particularly for metrics with high within-cell noise and high cell density, where standard methods are most inefficient.
Robustness: The approach is robust to the unknown composition of variance; the power-optimal method performs consistently well across different variance regimes, whereas standard methods degrade significantly when within-cell noise dominates.

The author concludes that this alignment is necessary because the "standard default" of unit-level MSE is fundamentally incompatible with the variance structure of clustered, time-windowed experiments. By re-weighting the learning objective to match the design's variance decomposition, the method ensures that model capacity is spent on the signal that actually drives statistical power.