📊 statistics

Detecting and Quantifying Circularity Bias When Composite Quality Ratings Are Used as Covariates: A Methodological Demonstration Using CMS Hospital Excess Readmissions Data

This paper demonstrates that including composite quality ratings as covariates in regression models introduces a circularity bias that systematically attenuates the estimated effects of structural predictors, a methodological hazard confirmed through CMS hospital readmission data and Monte Carlo simulations.

Original authors: Ali Akram

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Akram

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Don't Use the Answer Key to Grade the Test

Imagine you are a teacher trying to figure out if Student A (a for-profit hospital) performs differently than Student B (a non-profit hospital) on a specific math test (hospital readmissions).

You want to be fair, so you decide to adjust the scores based on how "smart" the students are overall. You pull out a report card that gives them a Star Rating based on their grades in Math, Science, History, and Art.

Here is the trap: The "Math Grade" is actually part of that Star Rating.

If you include the Star Rating in your calculation to "adjust" for the students' overall ability, you are accidentally using the Math Grade to explain the Math Grade. It's like asking, "How well did they do on the Math test?" and then saying, "Well, they got a high Star Rating, which includes their Math score, so they must have done okay."

This creates a circular logic loop. Because the Star Rating already contains the answer you are trying to measure, using it as a "control" makes the difference between the two students look much smaller than it actually is. You end up underestimating the real gap.

What the Author Did

The author, Ali Akram, investigated this problem using real data from the U.S. government (CMS) regarding hospital readmissions.

The Setup: He looked at 2,446 hospitals. He wanted to see if For-Profit hospitals had higher readmission rates (more patients coming back) than Non-Profit hospitals.
The Mistake: Many researchers try to be thorough by adding the "CMS Overall Star Rating" to their math models to "adjust" for quality.
The Problem: The Star Rating is built partly from the readmission numbers. So, adding it to the model is like putting the answer key on the right side of the equation.
The Result:
- Without the Star Rating: For-profit hospitals had a readmission rate that was 0.0196 higher than non-profits. This is a clear, noticeable difference.
- With the Star Rating: The difference shrank to 0.0113.
- The Takeaway: By including the Star Rating, the researchers accidentally hid nearly half (48%) of the real difference. The "adjustment" made the problem look less severe than it was.

The "Magic Trick" to Detect This

The author proposes a simple test to catch this error, which he calls a Diagnostic:

Step 1: Run your analysis without the Star Rating.
Step 2: Run the analysis with the Star Rating.
Step 3: Compare the results.

If the number you are interested in (the difference between for-profit and non-profit) suddenly shrinks significantly when you add the Star Rating, you have found the "circularity bias." It's a red flag telling you: "Stop! You are using the outcome to explain the outcome."

The Simulation: Proving It's Not a Fluke

To prove this wasn't just a weird coincidence with this specific data, the author ran a computer simulation (a "Monte Carlo" test).

He created a fake world where he knew the true answer: For-profit hospitals were definitely 20 points worse.
He then ran the math with and without the fake "Star Rating."
Result: When he added the fake Star Rating, the math automatically cut the true difference in half (down to 10 points).
Conclusion: This proves the bias is a mathematical certainty, not just a fluke of the hospital data.

Another Finding: Geography Doesn't Matter Much

The paper also looked at whether hospitals in different parts of the country (North, South, East, West) performed differently.

The Statistic: The math said the differences between regions were "statistically significant" (meaning they weren't just random noise).
The Reality Check: The author used a tool called "variance decomposition" (think of it as a pie chart of where the differences come from).
The Result: 98.9% of the differences happened inside the regions (between individual hospitals), and only 1.1% happened between the regions.
The Lesson: Even though the math says "Region matters," in the real world, it barely matters at all. A hospital's location explains almost none of the variation in readmission rates.

The Bottom Line

Don't use the Star Rating as a control variable if you are studying the specific metric (like readmissions) that is inside that Star Rating. It will make your results look weaker and less important than they really are.
Use the "Two-Model Test": Always check if adding a composite rating changes your main result drastically. If it does, you are likely falling into a circular trap.
Context Matters: Just because a result is "statistically significant" doesn't mean it's "important." In this case, geography was statistically significant but practically irrelevant.

Important Note: The author clarifies that removing the Star Rating doesn't make the study a "perfect" proof of cause-and-effect. There are still other factors (like patient health) that aren't accounted for. However, removing the Star Rating gives a cleaner, more accurate description of the relationship between ownership and readmissions, free from this specific mathematical error.

Technical Summary: Detecting and Quantifying Circularity Bias in Composite Quality Ratings

Problem Statement
Publicly reported, risk-adjusted quality measures are increasingly utilized to compare healthcare providers and allocate payments. A common methodological practice involves including composite quality ratings (e.g., CMS overall star ratings) as covariates in regression models to adjust for overall quality when studying specific outcomes (e.g., hospital readmissions). This paper identifies a critical methodological hazard in this practice: circularity bias (a form of over-adjustment).

When a composite rating $R$ is constructed, in part, as a function of the outcome of interest $Y$ (e.g., the star rating includes readmission performance), $R$ becomes a descendant of the outcome. Including such a rating as a regressor in a model where $Y$ is the dependent variable effectively places a proxy for the outcome on the right-hand side of the equation. Consequently, the rating coefficient absorbs part of the mechanical relationship between the outcome and itself, causing the coefficients of genuine structural predictors (e.g., hospital ownership) to be systematically attenuated toward zero. This is distinct from confounding adjustment or collider bias; it is a structural error where the analyst conditions on a post-treatment variable that is partially determined by the outcome.

Methodology
The paper proposes a two-pronged methodological approach: a diagnostic test and a variance decomposition.

The Circularity Diagnostic:
- Procedure: Estimate the structural model twice on the same subsample:
  - Model A: Regress the outcome $Y$ on structural predictors $X$ (excluding the composite rating).
  - Model B: Regress $Y$ on $X$ including the composite rating $R$ .
- Interpretation: Compare the focal coefficient of $X$ between the two models. A material movement toward zero in Model B indicates attenuation. While attenuation can also result from legitimate confounding adjustment, the diagnosis relies on structural knowledge: if the added variable is a descendant of the outcome (as composite ratings often are), the attenuation represents bias, and the unadjusted Model A provides the less biased estimate of the structural association.
Variance Decomposition:
- To complement the diagnostic, the authors perform a between- versus within-group variance decomposition. This partitions the total sum of squares of the outcome to determine how much variation is attributable to group-level characteristics (e.g., census regions) versus within-group variation. This guards against over-interpreting statistically significant but substantively small geographic coefficients.
Data and Simulation:
- Empirical Data: The study utilizes CMS data for Fiscal Year 2026, covering 2,446 acute care hospitals. The outcome is the discharge-weighted excess readmission ratio (a risk-adjusted measure of 30-day readmissions). The focal predictor is hospital ownership (for-profit vs. nonprofit).
- Monte Carlo Simulation: To confirm the systematic nature of the bias, the authors simulated data with a known true effect of a binary indicator on an outcome. A composite rating was constructed to embed the outcome. The simulation compared estimators with and without the rating to verify that the inclusion of the descendant rating systematically biases the estimate toward zero.

Key Results

Empirical Findings: In the primary model excluding the CMS star rating (Model A), for-profit ownership was associated with a higher excess readmission ratio compared to nonprofit ownership ( $+0.0196$ , $p < 0.001$ ).
Quantification of Bias: When the star rating (which embeds the readmission outcome) was included (Model B), the coefficient for for-profit ownership attenuated by 48% to $+0.0113$ . The star rating itself was a strong predictor ($-0.0209$ per star), explaining 19.8% of the variance, but this improvement was largely mechanical due to the rating's construction.
Simulation Confirmation: The Monte Carlo simulation, with a known true effect of $+0.020$ , recovered this effect almost exactly when the rating was excluded. When the outcome-embedding rating was included, the estimate biased toward zero (mean $+0.0099$ ), resulting in a 50% mean attenuation. This closely matched the 48% attenuation observed in the real CMS data, confirming the bias is systematic.
Variance Decomposition: Only 1.1% of the variation in the excess readmission ratio lay between census regions, while 98.9% was within regions. This indicates that while regional coefficients may be statistically significant, regional membership accounts for negligible substantive dispersion in the measure.
Robustness: The positive association between for-profit ownership and excess readmissions was stable across six specific clinical conditions (e.g., heart failure, pneumonia), though the magnitude varied.

Significance and Claims
The paper claims three primary contributions:

Formalization of the Hazard: It explicitly identifies and formalizes the "descendant-of-the-outcome" problem in the context of publicly reported composite ratings, distinguishing it from standard confounding or collider bias.
A Simple Diagnostic: It proposes a low-cost, transparent diagnostic (comparing models with and without the rating) that allows analysts to detect and quantify this specific over-adjustment bias without requiring additional data.
Clarification of Variation: Through variance decomposition, it demonstrates that statistically significant grouping coefficients (e.g., by region) may explain almost none of the actual variation in quality measures, cautioning against equating statistical significance with substantive importance.

Limitations and Scope
The authors maintain a modest stance regarding their findings:

Not a Causal Estimate: The "unadjusted" estimate (excluding the rating) is less biased regarding circularity, but it is not a causal estimate of ownership effects. It remains subject to standard confounding by patient mix, as the CMS excess readmission ratio is risk-adjusted within peer groups but not at the hospital level.
Composite Nature: The study acknowledges the irony that the outcome itself is a composite of six conditions, though the weighting is transparent and does not embed the structural predictors, thus avoiding the specific circularity issue being studied.
Generalizability: The diagnostic and decomposition tools are presented as transferable to any setting where a composite rating embedding the outcome is considered as a covariate (e.g., nursing homes, dialysis facilities).

The paper concludes that including composite ratings that embed the outcome as covariates is not a neutral adjustment but a source of attenuation bias. The proposed tools allow researchers to detect this bias and reframe the interpretation of structural associations and geographic variations in quality data.