📊 statistics

Orthogonal factorial designs for trials of therapist-delivered interventions: Randomising intervention-therapist combinations to patients

This paper proposes and evaluates a family of orthogonal factorial designs that randomize therapist-intervention combinations to patients, enabling the statistical separation of therapist effects from intervention effects in trials of therapist-delivered treatments.

Original authors: Rebecca EA Walwyn, Rosemary A Bailey, Arpan Singh, Neil Corrigan, Steven G Gilmour

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rebecca EA Walwyn, Rosemary A Bailey, Arpan Singh, Neil Corrigan, Steven G Gilmour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out which of two different cooking recipes (Intervention A vs. Intervention B) makes the best soup. In a typical experiment, you might hire two famous chefs (Therapists) and have Chef 1 cook only Recipe A and Chef 2 cook only Recipe B. If the soup from Chef 1 tastes better, you might not know if it's because the recipe is superior or simply because Chef 1 is a better cook.

This paper tackles that exact problem in the world of therapy trials. It proposes a new, more rigorous way to organize these experiments so we can tell the difference between a "good recipe" and a "good chef."

Here is the breakdown of their ideas using simple analogies:

1. The Core Problem: The "Chef vs. Recipe" Mix-up

In many therapy studies, researchers treat the therapy (the recipe) as the main thing they are testing, but they treat the therapist (the chef) as just a background detail. The paper argues that therapists are actually a major part of the "treatment." If you don't account for them properly, your results are muddy.

2. The Proposed Solution: The "Crossed Kitchen"

The authors suggest a design where every chef tries every recipe.

Old Way: Chef A only makes Recipe A; Chef B only makes Recipe B.
New Way: Chef A makes both Recipe A and Recipe B. Chef B also makes both Recipe A and Recipe B.

By having every chef try every recipe, you can mathematically separate the "Chef Effect" (how good the person is) from the "Recipe Effect" (how good the therapy is). This is called a Crossed Factorial Design.

3. The Three Kitchen Setups (The Designs)

The paper outlines three ways to organize this "Crossed Kitchen" depending on how complex the trial is:

The Simple Kitchen (Completely Randomised): Imagine one big kitchen with no shifts. You have a list of all possible "Chef + Recipe" combinations. You hand these out to patients completely at random. It's like shuffling a deck of cards and dealing them out.
The Shift Kitchen (Randomised Block): In real life, chefs don't work 24/7; they work in shifts (or "batches"). Maybe Chef A is only available in the morning, and Chef B in the afternoon. The paper suggests organizing the trial by these time blocks. You ensure that within every shift, every chef gets a chance to cook every recipe. This accounts for the fact that a chef might be tired at the end of the day or learning as they go.
The Multi-Location Kitchen (Multicentre Randomised Block): Now imagine the trial happens in several different cities (Centres). In each city, there are local chefs who can't travel to other cities. The design ensures that within each city, and within each time shift, the "Chef + Recipe" combinations are still mixed up fairly.

4. The Math: How to Taste the Soup (Analysis)

Once the data is collected, you can't just use a standard calculator to see who won. Because the design is so specific, the math needs to be just as specific.

The "Hasse Diagram": The authors use special flowcharts (Hasse diagrams) to map out exactly how the chefs, recipes, time, and locations fit together. Think of this as a blueprint for the kitchen to ensure no ingredient is double-counted.
The "Satterthwaite" Adjustment: When you mix so many variables (chefs, recipes, time, locations), the standard rules for calculating "statistical significance" (the odds that the result happened by chance) get tricky. The paper recommends using a specific mathematical adjustment (Satterthwaite's method) to make sure the final verdict is accurate.

5. The Simulation: A "Dry Run"

Before telling doctors to change how they run trials, the authors ran a computer simulation (a "dry run") with 10,000 fake trials. They tested their new method against the old, messy methods.

What they found:

Fairness: When they randomly assigned the "Chef + Recipe" combos, they could accurately measure how much of the result was due to the chef and how much was due to the recipe.
The Danger of Guessing: If they tried to assign chefs to recipes after seeing which recipe the patient got (or if they didn't hide the recipe from the chef), the results got skewed. It was like a chef tasting the soup before serving it and changing the recipe to match the customer's mood. This made the study unreliable.
The Verdict: The new method of randomizing the combination of therapist and intervention together is the most reliable way to get a true answer.

Summary

The paper argues that to truly know if a therapy works, we must stop treating therapists as invisible background noise. Instead, we should treat them as a key ingredient in the recipe. By designing trials where every therapist tries every therapy, and by using specific mathematical tools to analyze the results, we can get a clearer, fairer picture of what actually helps patients.

The Bottom Line: Don't just test the therapy; test the team (therapist + therapy) together, and use the right math to figure out who did what.

Technical Summary: Orthogonal Factorial Designs for Trials of Therapist-Delivered Interventions

Problem Statement
In individually-randomised parallel-group trials evaluating therapist-delivered interventions (e.g., psychotherapy), treatment-related clustering is a recognized issue where patient outcomes are correlated within therapists. While standard practice acknowledges this clustering for sample size and analysis, the primary focus remains on estimating intervention effects, treating therapists merely as a source of variance rather than a treatment factor. Disentangling the specific effects of therapists from the effects of interventions remains a challenge. Existing literature often addresses nested or partially nested designs (where therapists deliver only one intervention) but has left the literature on "crossed" designs (where each therapist delivers all interventions) relatively underdeveloped. Furthermore, there is a disconnect between the Design of Experiments (DoE) literature, which typically assumes fixed treatment factors, and clinical trial methodology, which often treats therapists as random effects.

Methodology
The authors propose a family of orthogonal factorial experimental designs that treat both the intervention and the therapist as treatment factors. In this framework, the combination of a specific therapist and a specific intervention constitutes the "treatment," which is randomised to patients. The paper adapts the classical DoE approach described by Bailey [42] to accommodate therapists as a random treatment factor.

The authors outline three specific design structures:

Completely Randomised Factorial Design: A single-centre trial where interventions and therapists are constant over time. Patients are randomised to all combinations of interventions and therapists.
Randomised Block Factorial Design: Accounts for "therapist learning" or time effects by introducing a blocking factor, "Batches" (time periods). Treatments are randomised to patients within each batch.
Multicentre Randomised Block Factorial Design: Extends the previous design to multiple centres, where therapists are nested within centres, and batches are crossed with centres.

For each design, the authors provide:

Hasse Diagrams: To visualise the relationships between unit and treatment factors, distinguishing between fixed and random effects.
Analysis of Variance (ANOVA): Detailed tables defining strata, sources of variation, degrees of freedom, and expected mean squares. The authors note that in block designs, the structure is not strictly orthogonal under standard definitions due to "dependent random effects" (where the supremum of random factors aliases with fixed factors). Consequently, they propose using linear combinations of mean squares and Satterthwaite's approximation to estimate appropriate residual degrees of freedom for testing fixed intervention effects.
Regression Models: Equivalent linear mixed models (random coefficient models) are presented for each design, linking variance components (between-therapist variance, between-therapist variance in intervention effect, and patient-level error) to the ANOVA eigenvalues.
Implementation: Specific R code using aov and lmer is provided to execute these analyses.

Simulation Study
A simulation study was conducted to evaluate the performance of five randomisation methods across the proposed designs:

Non-random therapist assignment after intervention randomisation.
Non-random therapist assignment before intervention randomisation.
Blocking intervention randomisation by therapist.
Proposed Method: Randomising the therapist-intervention combination to patients.
Randomising interventions, then randomising therapists to patients (blocked by intervention).

The study varied baseline covariates predictive of outcomes to simulate scenarios where therapist assignment might be influenced by patient characteristics or knowledge of the intervention.

Key Results

Bias and Variance Estimation: The intervention effect ( $\delta_1$ $δ_{1}$ ) was estimated without bias across all methods. However, the estimation of variance components was highly sensitive to the randomisation method.
- Non-random assignment of therapists (Methods 1–3) led to substantial overestimation of between-therapist variance ( $\sigma^2_{u1}$ ) and between-therapist variance in the intervention effect ( $\sigma^2_{v1}$ ) when covariates were imbalanced.
- Randomising therapist-intervention combinations (Method 4) or randomising therapists to patients (Method 5) facilitated unbiased estimation of these variance components.
Standard Error and Power: When non-random assignment caused an inflated estimate of the between-therapist variance in the intervention effect, the standard error of the intervention effect increased, leading to a substantial rise in Type II error (reduced power).
Type I Error: The Type I error rate remained at 5% for most methods. However, a specific scenario (Method 5, where therapists were randomised after interventions) showed inflated Type I error when a covariate influenced therapist assignment. This occurred because the post-hoc balancing of the covariate created an imbalance in the number of patients per treatment combination, which was not adjusted for in the analysis.
Boundary Estimates: The study noted that variance components were frequently estimated as zero (boundary estimates), which affects the degrees of freedom available for testing the intervention effect.

Significance and Claims
The paper claims to be the first to apply a formal Design of Experiments approach to the design of trials for therapist-delivered interventions where therapists are treated as a random treatment factor in a crossed design. The authors argue that this approach:

Provides a rigorous framework for separating therapist effects from intervention effects.
Offers a family of orthogonal designs that allow for the estimation of therapist learning (via blocking) and centre effects.
Demonstrates through simulation that purposeful randomisation of therapist-intervention combinations is superior to sequential or non-random assignment for unbiased variance estimation and maintaining statistical power.
Highlights the necessity of using Satterthwaite's approximation for degrees of freedom in these complex designs and suggests that ANOVA-based analysis may be preferable to standard regression to avoid unintended consequences of boundary estimates on degrees of freedom.

The authors conclude that while their proposed designs are theoretically ideal, practical constraints (such as therapist availability and turnover) require further methodological research to develop flexible randomisation methods. They emphasise that more purposeful trial design has the potential to generate better evidence for complex interventions in psychotherapy, physiotherapy, rehabilitation, and surgery.