📊 statistics

Cohen's f or Mean Standardized Differences? Assessing Covariate Balance with Multivalued Treatments

This paper introduces a Stata command to extend Cohen's f and standardized mean differences (SMD) for assessing covariate balance in multivalued treatments, demonstrating through simulations that Cohen's f tracks estimation bias as effectively as mean absolute SMD while providing a theoretically grounded alternative to the currently prevalent but less optimal maximum SMD approach.

Original authors: Ariel Linden

Published 2026-08-12

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Ariel Linden

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: "Did the new medicine actually cause the patient to get better, or were they just naturally healthier to begin with?" In the world of medical research, scientists often compare different groups of people—like those taking Drug A, Drug B, or a placebo—to find the truth. But there's a catch: before they can compare the results, they have to make sure the groups are fair. If the "Drug A" group happens to be full of young, fit athletes while the "Placebo" group is full of elderly people with colds, the comparison is broken. This is called covariate balance.

For a long time, when scientists compared just two groups, they had a perfect tool to check fairness called the Standardized Mean Difference (SMD). Think of it like a "fairness ruler" that measures how far apart the average ages or health scores of two groups are. If the number is small (like 0.1), the groups are balanced; if it's big, they are not. But what happens when you have three or more groups? Suddenly, the old ruler doesn't work. You can't just measure the distance between Group A and B, then B and C, and call it a day. You need a single, "omnibus" score—a master gauge that tells you if the whole room of groups is fair at once. Until now, researchers have been guessing how to combine those pairwise measurements, or using a statistical tool called Cohen's f that was originally designed for a different kind of math problem. This paper asks a simple, crucial question: When comparing three or more treatments, which tool is the best "fairness ruler"?

The Great Fairness Ruler Showdown

In this study, Dr. Ariel Linden set up a massive digital simulation lab to test the two main contenders for the title of "Best Multi-Group Fairness Ruler." On one side, we have the Status Quo, which is basically the "Average of All Differences" method. Imagine you have three teams: Red, Blue, and Green. The Status Quo method measures the gap between Red and Blue, then Blue and Green, then Red and Green, and then either averages those gaps or just picks the biggest one to report. It's like a teacher grading a class by looking at every pair of students and saying, "Okay, the average difference in their heights is small," or "Well, the tallest and shortest kid are very different, so the class is unbalanced."

On the other side is Cohen's f, a classic statistical tool that has been around for decades but usually used for analyzing variance in experiments. Dr. Linden realized this tool could be repurposed as a "fairness ruler" for multiple groups. Instead of just averaging the gaps, Cohen's f calculates a "root-mean-square" of the differences. To use a playful metaphor: if the Status Quo method is like taking the average speed of a group of runners, Cohen's f is more like calculating the total energy of the group. It pays extra attention to the runners who are really far ahead or behind, but it weighs them based on how many people are in each group.

The Simulation Race

To see which ruler actually works, Dr. Linden didn't just look at real data; he created a virtual world. He generated thousands of fake patients with different traits (some normal, some skewed, some yes/no traits) and assigned them to 3, 4, or 6 different treatment groups. He then introduced "confounding"—a fancy word for unfairness—by making sure certain groups got sicker or healthier patients on purpose. He tested these groups under two conditions: one where the groups were perfectly balanced (like a random lottery) and one where they were skewed (like a rigged game).

He then ran the groups through a "propensity score weighting" process. Imagine this as a digital scale that tries to re-balance the groups by giving more weight to the underrepresented people and less to the overrepresented ones, effectively making the groups look fair again. The goal was to see: Which ruler (Cohen's f or the Average/Max SMD) was better at predicting if the final treatment results were actually biased?

The Results: A Tie for First Place, But with a Twist

The simulation results were fascinating. When Dr. Linden checked how well each ruler predicted the actual "bias" (the error in the final medical conclusion), Cohen's f and the Average SMD tied for first place.

The Correlation: Both tools were incredibly good at tracking the bias. When the groups were unbalanced, both numbers went up. When the groups were balanced, both went down. The correlation between the ruler's score and the actual error was about 0.93 for both when looking at all the data together. Even when looking only at the "correctly weighted" groups, they both hovered around 0.79.
The Loser: The "Maximum SMD" (the method that just picks the single worst pair of groups and ignores the rest) was the weakest. It was the least reliable at predicting the overall bias. It's like a teacher who only cares about the one student who failed the test and ignores the fact that the rest of the class did great; it's too sensitive to one bad pair and misses the bigger picture.

So, does this mean Cohen's f is the magic bullet that replaces everything? Not exactly. The paper explicitly rules out the idea that Cohen's f is "better" in terms of accuracy. It performs just as well as the old method. However, the paper makes a very important discovery about scale.

The "Apples and Oranges" Problem

Here is the most critical finding: You cannot compare the numbers from Cohen's f directly to the numbers from the Standardized Mean Difference (SMD).

Imagine you have two thermometers. One measures temperature in Celsius, and the other in Fahrenheit. If you see "20" on the Celsius thermometer, you know it's a warm day. But if you see "20" on the Fahrenheit thermometer, it's freezing! You can't just say, "Oh, 20 is the same on both."

The paper proves mathematically that Cohen's f and the Average SMD are like those different thermometers.

If you have 3 groups, Cohen's f will be about 0.63 times the size of the Average SMD.
If you have 6 groups, Cohen's f will be about 0.77 times the size of the Average SMD.

This means if a researcher is used to seeing an SMD of 0.10 and thinking, "Okay, that's a small, acceptable imbalance," they cannot look at a Cohen's f of 0.10 and say the same thing. In fact, a Cohen's f of 0.10 might actually represent a larger imbalance than an SMD of 0.10, depending on how many groups you have. The paper explicitly warns against applying the old "0.10 rule" directly to Cohen's f.

The "Group Size" Trap

There is one more twist. Cohen's f has a built-in feature that the old method doesn't: it weighs the groups by their size.

The Big Group: If the biggest group (say, 80% of the patients) is slightly unbalanced, Cohen's f will shout about it loudly.
The Small Group: If a tiny group (say, 5% of the patients) is wildly unbalanced, Cohen's f might whisper about it, because it's "weighted" down by the small number of people.

The paper illustrates this with a real-world example of a heart failure study. Before weighting, the biggest imbalance was between the huge "Control" group and a small "Remote Monitoring" group. The old method (Average SMD) saw this as a big problem (0.134). Cohen's f saw it too, but because the Control group was so huge, the number looked smaller (0.065). Both agreed the groups were unbalanced, but the numbers looked different.

The author argues that this weighting is a "substantive choice." If you care about the entire population, Cohen's f is great because it focuses on the groups that matter most (the big ones). But if you care about a rare treatment that only a few people get, Cohen's f might hide the fact that those few people are being treated unfairly.

The Verdict

So, what should a researcher do?

Don't panic: Cohen's f is a valid, theoretically sound tool for checking balance in multi-group studies. It tracks bias just as well as the old methods.
Don't mix up the rulers: Never use the old "0.10" cutoff for Cohen's f. The numbers are on a different scale.
Look under the hood: The paper recommends that if you use Cohen's f, you must also report the breakdown of the numbers. You need to see the "per-level" scores (how each specific group is doing) and the "pairwise" scores (how each pair of groups is doing). If you only look at the single "Cohen's f" number, you might miss a specific group that is in trouble.

In the end, the paper suggests that Cohen's f is a powerful new addition to the detective's toolkit, offering a mathematically elegant way to look at multiple groups at once. But like any good tool, it comes with instructions: know how to read the scale, and always check the details before you declare the case solved.

Technical Summary: Assessing Covariate Balance with Multivalued Treatments

Problem Statement
In comparative-effectiveness research involving multivalued treatments ( $k \ge 3$ groups), there is no established omnibus standard for assessing covariate balance. The prevailing practice involves calculating pairwise standardized mean differences (SMD) for all group combinations and reporting either the mean or the maximum of these values. While effective for model selection, this aggregation lacks a direct theoretical link to Cohen's $d$ (the standard for two-group comparisons) and offers no principled way to extend the established "negligible imbalance" threshold (e.g., $|SMD| < 0.1$) to $k > 2$ groups. Furthermore, these aggregate SMD metrics have not been formally compared against alternative effect-size generalizations, such as Cohen's $f$ , regarding their ability to predict downstream treatment-effect bias.

Methodology
The paper proposes and evaluates Cohen's $f$ as an omnibus balance statistic for multivalued treatments, extending it to arbitrary weighted and covariate-adjusted models.

Statistical Framework:
- Cohen's $f$ : Defined as the weighted root-mean-square of each group's standardized deviation from the pooled mean: $f = \sqrt{\sum w_j d_j^2}$ , where $w_j$ are group weights and $d_j$ are standardized deviations.
- Implementation: The author extends the calculation of $f$ beyond simple one-way ANOVA to fitted models (e.g., generalized propensity score weighting) using the Stata command esizereg. This allows $f$ to be derived from adjusted predictive margins rather than raw sample means, ensuring validity under covariate adjustment and weighting.
- Decomposition: The method provides a per-level decomposition ( $d_j$ ) and pairwise comparisons ( $d_{jj'}$ ) to identify specific sources of imbalance.
- Theoretical Relationship: The author derives an exact mathematical relationship showing that $f$ is a size-weighted quadratic function of pairwise SMDs, whereas the mean absolute SMD ( $|d|$ ) is a linear, unweighted average. They prove that under equal group weights, the ratio $f/|d|$ has a specific minimum value determined by $k$ and the spacing of group means.
Simulation Study:
- Design: A simulation study was conducted with $k \in \{3, 4, 6\}$ treatment groups and sample sizes $N \in \{200, 500, 2000, 10000\}$ .
- Conditions: Data were generated under varying confounding strengths ( $\gamma$ ) and group balance scenarios (equal vs. unequal). Propensity scores were estimated under both correctly specified and misspecified models (omitting an interaction term).
- Validation: The performance of Cohen's $f$ was compared against the mean absolute SMD ( $|d|$ ) and maximum absolute SMD ( $|d|_{max}$ ) by correlating these statistics with the actual bias of downstream treatment-effect estimates.
- Robustness: A secondary scenario tested a nonlinear/interaction outcome model to ensure the findings were not artifacts of a linear outcome structure.

Key Results

Bias Tracking: Cohen's $f$ tracks downstream estimation bias comparably to the mean absolute SMD ( $|d|$ ). In the pooled analysis, the correlation between $f$ and bias was $r = 0.93$ , nearly identical to $|d|$ ( $r = 0.94$ ). Both outperformed the maximum absolute SMD ( $|d|_{max}$ ), which showed the weakest correlation ( $r \approx 0.92$ ). This ranking held true across correctly specified, misspecified, and unweighted arms, as well as in the nonlinear outcome scenario.
Scale Discrepancy: $f$ and $|d|$ are not numerically comparable. The ratio $f/|d|$ is systematically lower than 1 and depends on the number of groups ( $k$ ) and group size balance. For example, at $k=3$ , the ratio is approximately 0.63, rising to 0.77 at $k=6$ . Consequently, conventional SMD thresholds (e.g., 0.1) cannot be directly applied to $f$ .
Weighting Sensitivity: The relationship between $f$ and $|d|$ is influenced by group sizes. Because $f$ weights deviations by group size ( $w_j$ ), imbalance in large groups contributes more to the omnibus statistic than imbalance in small groups. Conversely, $|d|$ treats all pairs equally.
Robustness: The comparative performance of $f$ and $|d|$ remained stable even when the outcome model included nonlinear terms and interactions not captured by first-moment balance diagnostics.

Significance and Recommendations
The paper concludes that Cohen's $f$ is a theoretically grounded, valid omnibus statistic for assessing covariate balance in multivalued treatments, offering a direct generalization of the two-group $d$ framework.

Reporting Standards: The author recommends reporting $f$ alongside its per-level and pairwise decompositions. Relying solely on the omnibus $f$ may mask severe imbalance in specific pairs, particularly if that imbalance occurs in smaller groups where the size-weighting of $f$ dampens its contribution.
Thresholds: Researchers should not apply standard SMD thresholds (e.g., 0.1) directly to $f$ . Because the scale differs systematically, investigators must derive their own balance thresholds based on the specific relationship between $f$ and bias in their study design, rather than importing thresholds calibrated for pairwise SMDs.
Diagnostic Choice: While $f$ and $|d|$ perform similarly in tracking bias, they offer different perspectives. $f$ is sensitive to the overall pattern of imbalance weighted by group size, while $|d|$ provides an unweighted average. $|d|_{max}$ is the least reliable for tracking overall bias due to higher sampling variance, though it remains the most direct detector of a single worst-case pair.

The study emphasizes that while $f$ provides a unified, effect-size-based summary of balance, the choice of diagnostic should be transparent, and the specific substantive importance of treatment contrasts (e.g., rare treatment arms) should guide whether size-weighted or unweighted metrics are prioritized.