How can the use of different modes of survey data collection introduce bias? A simple introduction to mode effects using directed acyclic graphs (DAGs)

Technical Summary: Bias in Mixed-Mode Survey Data Collection

Problem Statement
The paper addresses the methodological challenges introduced by the increasing adoption of mixed-mode data collection (e.g., combining face-to-face interviews, telephone surveys, and web-based questionnaires) in epidemiology. While mixed-mode designs offer benefits such as reduced costs and higher participation rates, they introduce two distinct sources of bias:

1. Mode Effects: Systematic measurement error where the survey mode influences how respondents answer questions (e.g., social desirability bias in interviewer-led modes vs. satisficing in self-completion modes).
2. Mode Selection: Differences in sample composition where individuals self-select into specific modes based on their characteristics (e.g., education level, internet access, or the exposure/outcome variables themselves).

The authors argue that while mode effects are often recognized, the consequences of attempting to correct them via standard statistical adjustments (like conditioning on mode) are frequently underappreciated. Specifically, naïve conditioning on survey mode in the presence of mode selection can induce collider bias, potentially distorting exposure-outcome relationships more severely than the original measurement error.

Methodology
The paper employs Directed Acyclic Graphs (DAGs) as a primary methodological tool to visualize and analyze the causal structures underlying mixed-mode surveys.

• Notation: The authors distinguish between latent (true) variables ($X$) and observed measures ($X^*$) obtained via survey.
• Scenario Modeling: The authors construct a series of DAGs to depict various causal configurations:
  • Scenarios where mode acts solely as a source of measurement variation (a competing exposure or confounder of measured variables).
  • Scenarios where mode acts as a collider because the latent exposure and/or outcome influence the choice of survey mode.
  • Complex longitudinal scenarios involving repeated measures and mode switching over time.
• Evaluation of Solutions: The paper theoretically evaluates the validity of three common statistical approaches—conditioning, imputation, and quantitative bias analysis—under these different causal structures.

Key Contributions and Results
The analysis yields several critical insights regarding the interaction between mode effects and mode selection:

1. Conditioning on Mode:

   • Valid Scenario: If mode introduces measurement variation but is not influenced by the exposure or outcome (i.e., no mode selection), conditioning on mode (e.g., via stratification or covariate adjustment) effectively removes the measurement bias.
   • Invalid Scenario: If the latent exposure or outcome influences the choice of mode (mode selection), mode becomes a collider. Conditioning on mode in this context opens a back-door path between the exposure and outcome, introducing collider bias. The paper notes that this bias can dominate the reduction in measurement error, leading to net negative consequences.

2. Imputation Approaches:

   • The paper critiques multiple imputation strategies that treat data from non-reference modes as "missing." This approach implicitly conditions on mode (by discarding non-reference data), thereby violating the "Missing At Random" (MAR) assumption if mode selection is driven by the variables of interest. This leads to the same collider bias issues as direct conditioning.

3. Longitudinal Complexity:

   • In longitudinal settings, mode selection can occur at multiple time points. Switching modes between waves can induce apparent changes in variables that are actually artifacts of measurement differences. Attempting to correct this by conditioning on time-varying modes risks opening complex collider paths, particularly if unmeasured determinants of mode selection (e.g., socio-economic position) exist.

4. Quantitative Bias Analysis (QBA):

   • The authors identify QBA as a more robust alternative when mode selection is present. Unlike conditioning or imputation, QBA (including calibration and sensitivity analysis) does not require the assumption that mode selection is absent or fully controlled. It allows researchers to estimate the magnitude of bias using external validation data or hypothesized effect sizes without making strong, often untestable, assumptions about the selection mechanism.

Significance and Claims
The paper positions itself as a "simple introduction" to these challenges, aiming to bridge the gap between survey methodology literature and epidemiological practice. Its primary claims are:

• Underappreciation of Risk: The risk of introducing collider bias through naïve conditioning on survey mode is likely underappreciated in epidemiology, particularly as mixed-mode designs become standard.
• Context Dependency: There is no universal solution. The appropriate handling of mode depends entirely on the specific causal structure (i.e., whether mode selection exists and its relationship to the exposure/outcome).
• Recommendation for Causal Thinking: Researchers should utilize DAGs to explicitly map out the causal relationships between exposure, outcome, and survey mode before selecting an analytical strategy.
• Shift in Best Practice: When meaningful mode selection is suspected, standard adjustment methods should be avoided. Instead, researchers should prioritize quantitative bias analyses to calibrate estimates or assess the sensitivity of their findings to potential mode effects, rather than relying on conditioning or imputation which may introduce new biases.

The authors conclude that while preventing mode effects through design is ideal, statistical solutions must be chosen with a rigorous understanding of the underlying selection mechanisms to avoid compromising the internal validity of survey-based epidemiological studies.