Towards a unified approach to formal risk of bias assessments for causal and descriptive inference

1. Problem Statement

The paper addresses a critical gap in statistical practice: the fragmentation of "risk of bias" (RoB) assessments between causal inference (e.g., randomized controlled trials, quasi-experiments) and descriptive inference (e.g., survey sampling, population monitoring).

• Current State: In medical research, RoB tools (like Cochrane RoB2 and ROBINS-I) are well-established but primarily focus on internal validity (systematic errors within the study design that threaten causal identification). Conversely, descriptive inference fields (like survey sampling) often struggle with external validity (generalizability and transportability), particularly when using non-probability samples.
• The Core Issue: Researchers frequently treat statistical models as the sole source of uncertainty. However, "invisible" uncertainties—such as unmodelled selection bias, non-response mechanisms, and the mismatch between a study sample and a target population—often undermine conclusions.
• The Gap: There is no unified framework that treats bias as a property of the inferential claim rather than just the estimator. Current practices often fail to explicitly assess the validity of extending in-sample estimates (whether causal effects or descriptive means) to a broader target population, especially when statistical adjustments (e.g., weighting) rely on untestable assumptions (e.g., "missing at random").

2. Methodology and Theoretical Framework

The authors do not propose a new dataset or a specific statistical algorithm. Instead, they employ a conceptual and logical synthesis of existing statistical theories to propose a unified framework.

• Statistical Argumentation: The paper reframes applied statistics as "argumentation" (citing Hodges, 1996). It argues that the validity of a statistical conclusion depends on the "burden of proof" required for the specific inferential goal (causal vs. descriptive).
• Unifying Logic: The authors map the logical structures of causal and descriptive inference to show they share identical vulnerabilities:
  • Causal Inference: Requires internal validity (identifying the causal effect within the study) and external validity (generalizing/transporting that effect to a target population).
  • Descriptive Inference: Requires identifying the population parameter within the source frame and ensuring the realized sample represents that frame (selection bias) and that the frame represents the target population (coverage bias).
• Integration of Tools: The paper synthesizes tools from:
  • Causal Inference: Propensity scores, causal graphs (DAGs), and transportability theory (e.g., Hartman, 2021).
  • Survey Sampling: Design-based vs. model-based inference, non-probability sampling adjustments, and "ignorability" diagnostics (e.g., Meng, 2022).
• Conceptual Distinction: The authors distinguish between:
  • Identification: Establishing the estimand within the study setting (Internal Validity).
  • Generalization/Transportability: Extending the estimand to a target population (External Validity).
  • They argue that both steps are susceptible to systematic error and require qualitative assessment.

3. Key Contributions

The paper makes several significant theoretical and practical contributions:

1. Unified Definition of Risk of Bias: It proposes that "risk of bias" should not be limited to internal validity. Instead, it must encompass external validity biases (generalizability/transportability) for all inferential claims. Bias is defined as a systematic deviation from the truth relative to a specific inferential goal.
2. Logical Equivalence of Causal and Descriptive Problems: The authors demonstrate that the problem of generalizing a treatment effect from a trial to a population is logically identical to the problem of generalizing a survey mean from a non-probability sample to a population. Both require assumptions about the relationship between the sample, the source frame, and the target population.
3. The "Invisible" Uncertainty: The paper highlights that model-based adjustments (like weighting or shrinkage) often rely on assumptions (e.g., ignorability) that cannot be empirically guaranteed. Therefore, qualitative assessment of these assumptions is as crucial as the quantitative adjustment itself.
4. Proposed Reporting Standards: The authors argue for the mandatory inclusion of qualitative RoB assessments in research reporting. They suggest that journals and funders should require authors to explicitly disclose:
   • The estimand and target population.
   • The study frame and realized sample characteristics.
   • Assumed selection/non-response and missingness mechanisms.
   • Key measurement limitations affecting generalizability.

4. Results and Findings

As a theoretical paper, the "results" are the logical conclusions drawn from the synthesis of literature:

• Shared Vulnerability: Both causal and descriptive fields suffer from the same "unmodelled error" when attempting to generalize findings. The tools used to assess bias in one field (e.g., sensitivity analysis for weighting in causal inference) are directly applicable to the other (e.g., assessing representativeness in survey sampling).
• Limitations of Current Tools: Existing RoB tools in medicine are often too narrow, focusing on internal validity and ignoring the "transportability" of results. Conversely, descriptive fields often lack formalized frameworks to assess the credibility of non-probability samples.
• Ethical Imperative: The paper concludes that failing to assess and report these biases is not just a methodological flaw but an ethical issue. Researchers have the resources to assess these risks; failing to do so leads to "incredible certitude" (Manski, 2011) and potentially harmful policy decisions based on flawed generalizations.

5. Significance and Implications

The paper has profound implications for the future of statistical practice across disciplines:

• Paradigm Shift: It moves the focus from "fixing the model" to "evaluating the argument." It suggests that no model is perfect, and the credibility of a conclusion depends on the transparency of its limitations.
• Policy and Publishing: The authors call for a "top-down" approach where funding agencies and journals mandate the use of unified RoB frameworks (similar to PRISMA or GRADE in medicine) for all research involving generalization, not just medical trials.
• Interdisciplinary Learning: It encourages cross-pollination between causal inference and survey sampling communities, suggesting that causal inference can learn from survey sampling's handling of non-probability data, and vice versa.
• Research Ethics: By framing bias assessment as a duty to the public and a requirement for scientific falsifiability, the paper elevates the discussion of uncertainty from a technical footnote to a core component of research integrity.

In summary, Pescott et al. argue that to truly understand the value of statistical research, consumers must be presented with a clear, qualitative assessment of the "invisible" uncertainties regarding how well a study's sample and design support its claims about the wider world. This requires a unified, mandatory approach to risk-of-bias assessment that bridges the divide between causal and descriptive inference.