On the choice of using raw or demographically-corrected scores

Technical Summary: On the Choice of Using Raw or Demographically-Corrected Scores

Problem Statement
Demographic corrections (e.g., adjusting for age and education) are standard practice in neuropsychology and health sciences, particularly for cognitive screening tests like the Mini-Mental State Examination (MMSE). These corrections aim to normalize scores against reference norms to ensure fairness and prevent overdiagnosis in vulnerable demographic groups. However, empirical evidence and some theoretical arguments suggest that such corrections may reduce classification accuracy (sensitivity and specificity) compared to using raw scores. The debate centers on whether the potential loss in accuracy is justified by gains in fairness or demographic insensitivity. Existing literature often relies on restrictive parametric assumptions, specific simulation models, or empirical observations that lack a general theoretical framework to explain when and why raw scores might outperform corrected ones.

Methodology
The authors develop a general, non-parametric framework to compare the classification accuracy of raw scores ($X$) versus demographically-corrected scores ($Z$).

• Formalism: The study defines demographic correction as a transformation $Z = g_V(X)$ based on the conditional distribution of raw scores in the non-impaired population ($D=0$). They distinguish between different types of corrections, with a focus on Type I corrections (e.g., z-score standardization) which rely on the distribution of $X$ given $V$ and $D=0$.
• Assumptions: The analysis relies on three key assumptions:
  1. Well-designed test: The likelihood ratio of the raw score is decreasing (implying concave covariate-specific ROC curves).
  2. Well-designed correction: The corrected score $Z$ is conditionally independent of demographics $V$ given the non-impaired status ($Z \perp \perp V | D=0$).
  3. Probabilistic co-monotonicity: The association between demographics and the raw score in the non-impaired group moves in the same direction as the association between demographics and the disease status ($D$) conditional on a specific raw score threshold.
• Causal Interpretation: The authors employ a causal DAG (Directed Acyclic Graph) framework to interpret these assumptions. They model demographics ($V$) as having direct effects on both the test score ($X$) and the disease status ($D$). They analyze how conditioning on $X$ creates a "collider" path that may alter the observed association between $V$ and $D$.
• Fairness Analysis: The paper formalizes the concept of "demographic insensitivity" (both strong and weak) using counterfactual reasoning. They define a score as insensitive if the decision rule's performance (True/False Positive Rates) remains invariant under hypothetical interventions on the sensitive attributes (e.g., changing the test's sensitivity to education or sensory abilities).
• Empirical Application: The theoretical results are applied to the OASIS-3 dataset (1,317 participants) using MMSE scores. The authors estimate the superiority condition ($\tau(t)$) using non-parametric methods and parametric hierarchical models to test the validity of Assumption 3. They also test for demographic insensitivity in both raw and age-corrected scores.

Key Contributions

1. Sufficient Conditions for Raw Score Superiority: The paper provides formal, sufficient conditions (Theorem 1) under which raw scores yield higher marginal classification accuracy (higher TPR for a fixed FPR) than demographically-corrected scores. This holds without relying on restrictive parametric models or vague notions of "strong" associations.
2. Causal Interpretation of Correction Detriment: The authors clarify that corrections become detrimental when demographic variables are strong predictors of the condition of interest (e.g., age is a strong risk factor for dementia) and the correction removes the variation in the raw score that is causally linked to the disease risk.
3. Formalization of Fairness: The paper distinguishes between "equalizing error rates" (which corrections often achieve for False Positive Rates) and "demographic insensitivity" (where decision rules are invariant to demographic interventions). They demonstrate that standard corrections do not guarantee demographic insensitivity, particularly regarding True Positive Rates.
4. Empirical Validation: Using OASIS-3 data, the study provides evidence supporting the theoretical claim that raw MMSE scores have superior classification accuracy over age-corrected scores and demonstrates that age-corrected scores are not demographically insensitive among cognitively impaired individuals.

Results

• Theoretical: Under the assumption that the test is well-designed and the correction achieves independence of the score from demographics in the non-impaired group, raw scores outperform corrected scores if the demographic variable's effect on the raw score (in the healthy population) and its effect on the disease risk (conditional on the score) are co-monotone.
• OASIS-3 Application:
  • Accuracy: Non-parametric estimates of the superiority bound $\tau(t)$ were positive for most thresholds, though confidence intervals were wide. However, parametric analysis (assuming a hierarchical linear model) strongly rejected the null hypothesis of non-superiority ($p < 0.05$), providing statistical evidence that raw MMSE scores have better classification accuracy than age-corrected scores across all FPRs.
  • Insensitivity: The study rejected the hypothesis that raw MMSE scores are age-insensitive. Crucially, it found that even age-corrected scores (z-scores) were not demographically insensitive among individuals with cognitive impairment ($D=1$), failing to achieve the desired fairness property for the target population.

Significance and Claims
The authors claim to bring clarity to the debate on demographic corrections by establishing that raw scores can be theoretically and empirically superior in classification accuracy without relying on restrictive data-generating models. They argue that the intuition that "corrections are fair" is flawed because standard corrections do not ensure demographic insensitivity in the decision-making process, particularly for those with the condition.

The paper concludes that while demographic corrections may equalize False Positive Rates across groups, they can be detrimental to the overall diagnostic utility of screening tests when demographics are strong risk factors for the condition. The authors emphasize that their findings apply broadly to health sciences where demographic corrections are used (e.g., kidney function, lung function, neurological diagnoses), suggesting that the trade-off between accuracy and perceived fairness requires careful, context-specific evaluation rather than routine application of corrections.