📊 statistics

Micro-randomized Trials with Categorical Treatments and Binary Proximal Outcome: Causal Effect Estimation and Sample Size Calculation

This paper addresses the challenges of micro-randomized trials with categorical treatments and binary proximal outcomes by defining the causal excursion effect, proposing the EMEE-catA estimator, and deriving a robust sample size formula to ensure adequate power for comparing treatment levels, as demonstrated through simulations and a real-world application.

Original authors: Jeremy Lin, Tianchen Qian

Published 2026-08-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jeremy Lin, Tianchen Qian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship trying to get your crew to eat their vegetables. You have a fleet of smart devices on their wrists that can send little nudges: a funny cartoon, a stern reminder, or a friendly high-five. But you don't know which nudge works best, or if the crew is even looking at their wrists when you send them. This is the world of mobile health (mHealth): using phones and wearables to help people change their habits, like drinking less alcohol or exercising more.

To figure out what works, scientists use a special kind of experiment called a Micro-Randomized Trial (MRT). Think of it like a massive, high-speed game of "choose your own adventure." Instead of assigning a person to one strategy for the whole month, the computer flips a coin hundreds of times a day. At every single moment a decision is made, it randomly picks a different nudge to send. This happens so fast and so often that researchers can see, almost instantly, if a specific nudge made someone open their app or take a step. The big question they are trying to answer is: "Did this specific message, sent right now, cause the person to do the good thing?"

The tricky part is that sometimes the messages aren't just "yes" or "no." Sometimes you have a whole menu of options: a short message, a long story, a video, or no message at all. And the result you are looking for is often a simple "yes" or "no" (did they open the app? yes/no). Figuring out how to design these experiments so you have enough people to get a clear answer, without wasting time and money, is like trying to bake a cake where you don't know exactly how many eggs you need until you've already started mixing.

This paper is about baking that cake correctly. The authors, Jeremy Lin and Tianchen Qian, are statisticians who built a new recipe for these mobile health experiments. They created a tool to help researchers figure out exactly how many people they need to recruit to make sure their experiment actually works.

Here is the problem they solved: In the past, these tools only worked well when there were two choices (like "send a message" vs. "don't send a message"). But in the real world, scientists often want to test many different types of messages at once. Imagine a menu with a "funny joke," a "serious warning," and a "motivational quote." The old tools couldn't handle this menu properly, especially when the result is a simple "yes/no" (like "did they open the app?").

The authors invented a new way to measure the effect of these multiple choices. They call their new method EMEE-catA. It's like a super-powered calculator that can look at a messy history of thousands of random nudges and tell you, "Hey, the funny joke actually made people open the app 3.6 times more often than doing nothing!" Even better, this calculator is tough. It doesn't break if the researchers guess the wrong details about how people usually behave. It's designed to be robust, meaning it gives the right answer even when the real world is messy and unpredictable.

But the real magic is the sample size formula. This is a mathematical rule that tells a researcher: "If you want to be 80% sure you can tell the difference between your 'funny joke' and your 'serious warning,' you need exactly 1,197 people." Before this paper, researchers were often guessing this number, which could lead to experiments that were too small to find anything or too big and wasteful.

The authors tested their new recipe using data from a real study called Drink Less, which helped people cut down on alcohol. In that study, 349 people were nudged every day for 30 days. When the authors ran their new formula on this data, it confirmed that the study design was solid. They also ran thousands of computer simulations—virtual experiments where they pretended to be the researchers—to see if their formula would fail if the real world didn't follow the rules perfectly.

The results were promising. The formula worked great when the rules were followed. But even when the researchers broke the rules in their simulations (like when people's availability changed in weird ways or when the messages had delayed effects), the formula was surprisingly sturdy. It didn't always give the perfect number, but it usually gave a number that was close enough to ensure the experiment would succeed.

The paper also gives a set of "best practices" for anyone using this tool. It suggests that if you aren't sure about the details, it's better to be a little conservative. For example, if you think your message will work 50% of the time, but you aren't sure, the formula suggests planning for a scenario where it works 40% of the time. This ensures you don't run out of participants halfway through.

In short, this paper provides a new, reliable map for navigating the complex world of mobile health experiments. It allows scientists to design studies with multiple options and simple yes/no results without getting lost in the math. By using this new tool, researchers can be more confident that their experiments will actually tell them which digital nudges are the most effective at helping people change their lives.

Technical Summary: Micro-randomized Trials with Categorical Treatments and Binary Proximal Outcomes

Problem Statement
Micro-randomized trials (MRTs) are a standard experimental design for optimizing mobile health (mHealth) interventions by repeatedly randomizing participants among intervention options at multiple decision points. While previous literature has established methods for binary treatments (e.g., message vs. no message) and continuous outcomes, many practical mHealth applications involve categorical treatments with multiple levels (e.g., varying message contents or delivery strategies) and binary proximal outcomes (e.g., app engagement, behavior completion). Existing estimators for causal excursion effects (CEE) do not directly accommodate this specific combination. Furthermore, there is a lack of rigorous sample size calculation methods for detecting differential effects among multiple categorical treatment levels in MRTs with binary outcomes, which are critical for planning adequately powered studies.

Methodology
The paper addresses these gaps through three primary methodological developments:

Definition of Causal Excursion Effect (CEE) for Categorical Treatments:
The authors define the CEE on the log-relative risk scale for categorical treatments $A_t \in \{0, 1, \dots, K\}$ , where 0 is the reference level. The CEE contrasts the expected proximal outcome $Y_{t,\Delta}$ under an excursion of receiving treatment $k$ at time $t$ followed by no treatment for $\Delta-1$ subsequent points, versus receiving no treatment at time $t$ followed by no treatment for $\Delta-1$ points. This definition conditions on the availability of the decision point ( $I_t=1$ ) and marginalizes over the treatment history following the trial's randomization protocol.
Estimator: EMEE-catA:
The authors propose EMEE-catA (Estimator for Marginal Excursion Effects for categorical treatments with binary outcomes). This estimator extends the work of Qian et al. (2021) to the categorical setting.
- Model: It assumes a parametric model for the CEE, $CEE_{tk}(S_t) = f_t(S_t)^T \beta_k$ , where $S_t$ represents effect modifiers.
- Estimating Equation: The estimator solves a system of equations $P_n m(\alpha, \beta) = 0$ $P_{n} m (α, β) = 0$ . The estimating function $m(\alpha, \beta)$ $m (α, β)$ incorporates:
  - A stabilized inverse probability weight component ( $J_t$ ) to handle the marginal nature of the estimand and the requirement of no treatment in the subsequent $\Delta-1$ steps.
  - Centered treatment indicators $C_k(A_t) = \mathbb{1}(A_t=k) - \tilde{p}_t(k|S_t)$ to ensure orthogonality between the estimation of the causal parameters $\beta$ and the nuisance parameter $\alpha$ (which models the outcome under the reference treatment).
- Robustness: The estimator is shown to be consistent and asymptotically normal even if the nuisance outcome regression model ( $g_t(H_t)^T \alpha$ ) is misspecified, provided the CEE model is correct. This robustness is crucial given the high-dimensional history $H_t$ in MRTs.
Sample Size Formula:
The paper derives a sample size formula to guarantee statistical power for testing hypotheses of the form $H_0: L \times \text{MEE}_{1:K}(t) = 0$ (where MEE is the marginal CEE with $\Delta=1$ and $S_t=\emptyset$ ).
- Working Assumptions: The derivation relies on five working assumptions (WA): (a) parametric form of MEE, (b) known functional form of success probability under no treatment, (c) known availability probability, (d) no serial correlation in outcomes conditional on past treatments, and (e) exogenous availability.
- Test Statistic: A Wald-type test statistic is constructed based on the asymptotic normality of the estimator. To correct for small-sample bias, the rejection region utilizes a scaled F-distribution rather than a chi-squared distribution.
- Algorithm: An iterative algorithm (Algorithm 1) is provided to calculate the minimum sample size $n$ required to achieve a target power $(1-\beta)$ under a specific alternative hypothesis, utilizing the non-centrality parameter derived under the working assumptions.

Key Results

Theoretical Properties: Theorem 1 establishes the asymptotic normality of the EMEE-catA estimator. Theorem 2 demonstrates that under the working assumptions, the proposed sample size formula controls the Type I error rate at the nominal level and guarantees the desired power.
Simulation Studies: Extensive simulations (detailed in Supplementary Material) evaluate the operating characteristics of the methods:
- Type I Error: The Type I error rate remains controlled at the nominal level even when working assumptions are violated.
- Power: The sample size formula yields adequate power when all assumptions hold. When assumptions are violated, power performance varies:
  - Violations regarding the magnitude of the treatment effect (ATE) or average success probability (ASPN) significantly impact power (overestimation leads to under-powering; underestimation leads to over-powering).
  - Violations regarding the pattern (temporal variation) of effects or availability generally have less severe impacts, though specific patterns (e.g., linear vs. constant) can lead to under-powering in certain generative models.
  - Serial correlation and exogenous availability violations were found to be less critical to power than magnitude misspecifications.
Data Application: The methods were applied to the Drink Less MRT, a study optimizing an app to reduce alcohol consumption.
- The estimator successfully identified that both standard messages and messages from a bank of new messages significantly increased the probability of app opening compared to no notification.
- The sample size formula was used to retrospectively analyze the Drink Less design, determining that 1,197 participants would be required to detect the observed difference between the two active treatment levels with 80% power, assuming constant effects and availability.

Significance and Claims
The paper claims to extend the existing literature on MRTs by enabling granular comparisons among multiple categorical treatment levels for binary proximal outcomes, a common scenario in mHealth that was previously unsupported by specific causal inference and sample size tools.

The authors emphasize that their approach provides:

A robust estimator (EMEE-catA) that does not require correct specification of the complex nuisance outcome model, addressing the challenge of high-dimensional history in MRTs.
A practical sample size formula that controls Type I error and guarantees power under a set of working assumptions.
Practical guidance (summarized in Box 1) for researchers to input conservative estimates for effect magnitudes and average success probabilities to ensure adequate power, even if the specific temporal patterns of effects or availability deviate from the working assumptions.

The work is illustrated using real-world data from the Drink Less study, demonstrating the applicability of the proposed estimator and sample size calculator in planning and analyzing complex mHealth interventions. The authors note that while the framework assumes specific working assumptions for the derivation, simulations suggest the methods are robust to many violations, particularly regarding the temporal patterns of effects and availability, provided the magnitudes of key parameters are conservatively estimated.

Technical Summary: Micro-randomized Trials with Categorical Treatments and Binary Proximal Outcomes

More like this