📊 statistics

Lord's 'paradox' explained: the 50-year warning on the use of 'change scores' in observational data

This paper resolves Lord's 50-year-old paradox by demonstrating that neither change-score comparisons nor baseline-adjusted follow-up comparisons can reliably estimate causal effects in observational data due to distinct biases, thereby highlighting the critical need for well-defined research questions and advanced causal inference methods like g-methods.

Original authors: Peter W. G. Tennant, Georgia D. Tomova, Eleanor J. Murray, Kellyn F. Arnold, Matthew P. Fox, Mark S. Gilthorpe

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Peter W. G. Tennant, Georgia D. Tomova, Eleanor J. Murray, Kellyn F. Arnold, Matthew P. Fox, Mark S. Gilthorpe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Mystery: Two Statisticians, Two Answers

Imagine a university wants to know if the food in the dining hall makes students gain weight, or if boys and girls gain weight differently. They weigh everyone in September (the start of the term) and again in June (the end of the term).

They hire two statisticians to look at the data.

Statistician A looks at the difference in weight. They see that, on average, boys and girls weighed the same in September and weighed the same in June. Their conclusion: "Nothing changed. The diet and sex don't matter."
Statistician B looks at the final weight but adjusts for how heavy the students were to begin with. They see that for students who started at the same weight, the boys ended up heavier than the girls. Their conclusion: "Boys gained significantly more weight than girls, even when they started at the same size."

This is Lord's Paradox. How can two smart people look at the exact same numbers and come to completely opposite conclusions?

The Paper's Solution: It's Not a Paradox, It's a Trap

The authors of this paper argue that this isn't a magical mystery. It's actually a warning sign. Both statisticians are using methods that are flawed in observational data (data where people aren't randomly assigned to groups, like boys vs. girls).

The paper explains that "change" is a tricky concept. To understand it, imagine a variable (like weight) changing for three different reasons:

The "Habit" Factor (Endogenous Change): Things that happen automatically because of where you started. If you are already heavy, you might naturally stay heavy or gain a little just by existing.
The "Noise" Factor (Random Change): Random fluctuations, like weighing yourself on a wobbly scale or having a bad night's sleep.
The "Real" Factor (Exogenous Change): The actual effect of something new, like the diet or the exposure we are studying.

The Goal: We want to measure the "Real" factor. We want to know: Did the diet actually cause the weight change, independent of how heavy they were to start with?

Why Statistician A Failed (The "Change Score" Trap)

Statistician A used a Change Score. This is like calculating: Final Weight - Starting Weight.

The Analogy: Imagine you are trying to see how much a plant grew. You measure the plant at the start and the end, then subtract the start from the end.

The Problem: If the "exposure" (like being a boy) caused the starting weight to be higher, this method breaks.
The Paper's Explanation: When you subtract the starting weight, you accidentally subtract the effect of the exposure itself. If being a boy makes you start heavier, and you subtract that starting weight, you are essentially "canceling out" the very thing you are trying to measure.
The Result: Statistician A got a result of "zero change" because the math canceled out the real effect. The paper says this method is dangerous in observational studies because it often hides the truth or even gives the wrong answer (like saying a medicine works when it actually doesn't, or vice versa).

Why Statistician B Was Also Flawed (The "Adjustment" Trap)

Statistician B used ANCOVA (Follow-up conditional on baseline). This is like saying, "Let's compare boys and girls, but only if they started at the exact same weight."

The Analogy: Imagine you are trying to see if a new fertilizer works. You look at two plants that are the same height. But, your ruler is slightly bent (measurement error).

The Problem: In real life, our "starting weight" measurements aren't perfect. They have "noise" (like the wobbly scale).
The Paper's Explanation: When you try to adjust for a starting measurement that is slightly wrong, the math gets messy. It tends to overestimate the effect. It looks like the boys gained more weight than they actually did because the "noise" in the starting measurement messed up the adjustment.
The Result: Statistician B found a big difference, but the paper suggests this difference might be exaggerated because of measurement errors and other hidden factors (like physical activity) that weren't accounted for.

The Verdict: A 50-Year Warning

The paper concludes that neither method is perfect for observational data.

Change Scores (Statistician A) tend to underestimate or hide the effect, especially if the exposure affects the starting point.
Adjustment Methods (Statistician B) tend to overestimate the effect due to measurement errors and hidden confounders.

The "Perfect" Scenario:
The paper notes that if this were a Randomized Controlled Trial (like a drug trial where people are randomly assigned to groups), both methods would work fine and give the same answer. The paradox only happens in "observational" settings where groups (like boys vs. girls) already exist and have pre-existing differences.

The Takeaway for Everyone

The main lesson of this 50-year-old puzzle is: Be very careful when analyzing "change" in real-world data.

Define your question clearly: What exactly are you trying to measure?
Don't trust simple subtraction: Just taking the difference between "before" and "after" is often misleading if the groups started differently.
Don't trust simple adjustments: Just "adjusting" for the starting point isn't a magic fix if your measurements are imperfect or if there are hidden factors (like exercise habits) influencing the results.

The paper doesn't offer a single "magic bullet" solution for all situations, but it warns scientists that the two most common ways of looking at change are often biased in opposite directions, leading to the confusing "paradox" we see in Lord's example.

Technical Summary: Lord's 'Paradox' Explained: The 50-Year Warning on the Use of 'Change Scores' in Observational Data

Problem Statement
Frederick Lord's 1967 conundrum, known as Lord's 'paradox', highlights a persistent issue in observational data science: two ostensibly reasonable statistical approaches to estimating the effect of an exposure on the 'change' in an outcome can yield radically different, and sometimes sign-discordant, results. The first approach (Approach 1) compares mean 'change scores' (the difference between follow-up and baseline outcomes, $\Delta Y = Y_1 - Y_0$ ) between exposure groups. The second approach (Approach 2) compares follow-up outcomes between groups while conditioning on the baseline outcome (e.g., Analysis of Covariance, ANCOVA). While the literature has long debated whether these methods answer different questions or if one is superior, this paper argues that the paradox arises because both methods are susceptible to distinct, opposing biases when applied to observational data where the exposure is not randomized.

Methodology and Conceptual Framework
The authors resolve the paradox by deconstructing the definition of 'change' and mapping the causal pathways involved in longitudinal data. They propose that a variable's change over time ( $Y_1 - Y_0$ ) is driven by three distinct components:

Endogenous Change: Autocorrelation from the baseline instance ( $Y_0$ ), representing pre-determined processes.
Random Change: Transient random variation, including classical measurement error.
Exogenous Change: Non-random, non-endogenous change ( $C_Y$ ) that is potentially modifiable by baseline variables.

The authors posit that the true target estimand in most causal inquiries is the effect of an exposure ( $X$ ) on exogenous change. They utilize Directed Acyclic Graphs (DAGs) to illustrate how the relationship between the exposure and the baseline outcome ( $Y_0$ ) dictates the appropriate estimand. Specifically, when $X$ causes $Y_0$ (as in Lord's original example where sex influences baseline weight), the target is the controlled direct effect of $X$ on $Y_1$ at fixed levels of $Y_0$ .

To demonstrate the mechanics of the bias, the authors derive the mathematical relationship between the two approaches. They show that a change score analysis effectively subtracts a term $[1 - \beta_1]Y_0$ from the follow-up outcome. In observational settings where $X$ is associated with $Y_0$ , this subtraction term introduces bias unless strict assumptions (parallel trends and no anticipation) are met. Conversely, the follow-up conditional on baseline approach (ANCOVA) attempts to isolate exogenous change by subtracting a linear function of $Y_0$ . However, this approach is susceptible to bias if $Y_0$ is imperfectly measured (leading to regression dilution and inflated estimates) or if the relationship is nonlinear.

The authors validate these theoretical claims through simulated data replicating Lord's original scenario (sex, baseline weight, follow-up weight). They simulate scenarios with and without transient classical measurement error and time-varying confounding (physical activity).

Key Results
The simulation results and theoretical analysis yield the following findings:

Opposing Biases: In the absence of randomization, the two approaches produce estimates biased in opposite directions. Change score analyses tend to produce diluted or null estimates (or even opposite-sign coefficients) when the exposure causes or is caused by the baseline outcome. Follow-up conditional on baseline analyses tend to produce inflated estimates when the baseline outcome is measured with error.
The Role of Measurement Error: Imperfect measurement of the baseline outcome is a critical source of bias for the ANCOVA approach. The authors demonstrate that using multiple measures of the baseline outcome can reduce this bias, though it may not eliminate it entirely.
Time-Varying Confounding: When a time-varying confounder (e.g., physical activity) exists that is caused by the exposure and affects both baseline and follow-up outcomes, simple conditioning on the baseline outcome fails. In such cases, standard ANCOVA or change score models yield biased estimates, whereas g-methods (e.g., g-computation) are required to recover the true causal effect.
Equivalence under Specific Conditions: The authors note that change score analyses and follow-up conditional on baseline analyses are mathematically equivalent when the change score model also conditions on the baseline outcome, provided the same assumptions hold.

Significance and Conclusions
The paper concludes that Lord's 'paradox' is not a statistical curiosity but a fundamental warning regarding the use of change scores in observational data. The authors assert that neither approach is universally "correct" for estimating the effect of an exposure on exogenous change without satisfying specific, often untestable, assumptions.

The primary significance of the work lies in its clarification that:

Change scores are generally inappropriate for observational data where the exposure influences the baseline outcome (or vice versa), as they fail to isolate exogenous change due to the inherent bias in the subtraction term.
ANCOVA is not a panacea; it is highly sensitive to measurement error in the baseline variable and time-varying confounding.
Well-defined research questions are essential. Researchers must clearly define whether they are interested in total effects, direct effects, or exogenous change, and recognize that standard "change" analyses often fail to estimate these targets in non-randomized settings.

The authors recommend that change score analyses be reserved primarily for natural experiments or randomized trials where the exposure is exogenous. In observational settings, researchers should prioritize well-defined causal models, utilize multiple baseline measures to mitigate measurement error, and employ advanced methods like g-methods to address time-varying confounding.