Towards Improving the External Validity of Software Engineering Experiments with Transportability Methods

1. Problem Statement

Controlled experiments are the gold standard in Software Engineering (SE) for establishing causal relationships. However, they suffer from a fundamental trade-off:

• Internal Validity vs. External Validity: While randomization ensures internal validity (eliminating confounding factors), achieving external validity (generalizability to the target population) is difficult.
• Sampling Bias: Recruiting a representative sample of professional software engineers is expensive and logistically challenging. Consequently, SE experiments often rely on small samples of students or junior developers.
• Covariate Shift: This leads to a mismatch between the experimental sample (e.g., students with low experience) and the target population (e.g., industry professionals with varying experience). If a covariate (like experience) moderates the treatment effect, the results derived from the biased sample will not accurately reflect the true Average Treatment Effect (ATE) in the target population.
• Underutilized Data: SE researchers possess vast amounts of observational data (e.g., from repositories, logs, surveys) that could characterize the target population, but these are rarely combined with experimental data to correct for bias.

Transportability methods, a framework from statistical causal inference, offer a solution to "transport" results from a limited experimental sample to a broader target population by combining experimental and observational data. Despite their potential, these methods have not been adopted in SE.

2. Methodology

The paper proposes a framework for applying transportability methods to SE experiments.

A. Theoretical Framework

The authors define the problem using a Directed Acyclic Graph (DAG) where:

• $A$: Treatment (e.g., using GenAI).
• $Y$: Outcome (e.g., defects detected).
• $X$: Covariate (e.g., experience level) which acts as a treatment effect modifier.
• $S$: Trial eligibility (selection into the experiment).

The goal is to estimate the target population ATE ($\tau$) using the experimental sample ATE ($\tau_1$) and the distribution of $X$ in the target population.

B. Preconditions (Assumptions)

For transportability to work, seven assumptions must hold (adapted from Colnet et al.):

1. Consistency: Observed outcomes match potential outcomes.
2. Randomization: Treatment assignment is independent of potential outcomes within the trial.
3. Ignorability on Trial Participation: Outcome is unaffected by participation status when controlling for covariates.
4. Mean Exchangeability: Average treatment effects are the same across groups with identical observed characteristics.
5. Sample Ignorability for Treatment Effects (A5): The treatment effect is independent of trial eligibility given covariates $X$. (Crucial: $X$ must affect selection).
6. Transportability of Conditional ATE (A6): The conditional ATE is the same in the sample and the target population for every stratum of $X$.
7. Positivity (A7): Every subject in the target population has a non-zero probability of being in the sample (no stratum of $X$ is completely missing from the experiment).

C. Estimation Methods

The paper highlights two primary classes of estimators:

1. Reweighting (IPSW - Inverse Probability of Sampling Weighting):
   • Weights experimental data points based on the inverse probability of being selected into the experiment.
   • Subjects with low trial eligibility (e.g., senior engineers in a student-heavy sample) are up-weighted to represent the target population.
2. Regression (Plug-in g-formula):
   • Fits separate regression models for the treatment and control groups based on covariates $X$.
   • Predicts outcomes for the entire target population distribution of $X$ using these models and calculates the difference.

3. Simulation Study

To validate the approach, the authors conducted a computer simulation:

• Setup:
  • Target Population: 1,000 subjects with experience ($X$) following a negative-binomial distribution.
  • Selection Bias: Trial eligibility ($S$) decreased as experience increased (simulating the difficulty of recruiting senior engineers).
  • Treatment: Using GenAI ($A=1$) vs. Not using ($A=0$).
  • Ground Truth: The true ATE was set to 16.7, with a non-linear moderation effect where the benefit of GenAI decreased as experience increased.
• Comparison: The simulation compared four estimation methods:
  1. Naïve Mean Difference.
  2. Linear Regression with Interaction.
  3. IPSW (Transportability).
  4. Plug-in g-formula (Transportability).
• Results:
  • Naïve Mean Difference: Vastly overestimated the ATE (bias due to sample skew toward low experience).
  • Linear Regression: Improved but still overestimated because it assumed a linear interaction, whereas the simulation used a non-linear one.
  • IPSW: Correctly centered around the true ATE but showed high variance/instability.
  • Plug-in g-formula: Most accurate and robust. It successfully recovered the true ATE (16.7) by leveraging the target population's covariate distribution to correct for the non-linear moderation.

4. Key Contributions

1. Introduction of Transportability to SE: The first formal proposal to apply causal transportability methods to Software Engineering experiments.
2. Prerequisites & Guidelines: A clear definition of the necessary assumptions (A1–A7) and a roadmap for researchers to verify them.
3. Simulation Evidence: Empirical demonstration that transportability methods (specifically the g-formula) can recover true causal effects in the presence of covariate shift, outperforming standard regression and naïve analysis.
4. Application Scenarios: Identification of three key areas in SE where this applies:
   • Participant Experience: Validating student subjects against industry professionals.
   • System Properties: Generalizing results from small/student projects to large industrial systems.
   • Task Complexity: Generalizing from simplified experimental tasks to complex real-world workflows.
5. Roadmap for Adoption: A practical guide for the SE community to operationalize these methods.

5. Roadmap for Adoption

The authors propose a four-step path for SE researchers:

1. Understand Moderators: Identify which covariates (e.g., experience, domain knowledge) moderate the treatment effect and explicitly model them (e.g., via DAGs).
2. Operationalize Covariates: Ensure covariates are measured validly (e.g., defining "experience" beyond just "years worked").
3. Collect Observational Data: Gather large-scale observational data (surveys, logs) to characterize the distribution of these covariates in the target population.
4. Transport Results: Apply IPSW or g-formula estimators to combine the experimental results with the observational covariate distribution, thereby correcting for selection bias without needing more expensive experiments.

6. Significance

This paper addresses a critical bottleneck in SE research: the gap between controlled experiments and real-world applicability.

• Scientific Rigor: It moves the field beyond anecdotal debates about "students vs. professionals" toward a statistical framework for quantifying and correcting generalizability.
• Resource Efficiency: It allows researchers to maximize the value of existing, smaller experiments by leveraging abundant observational data, reducing the need for prohibitively expensive large-scale industry trials.
• Practical Impact: By improving external validity, the findings of SE experiments become more reliable and actionable for industry practitioners, ultimately bridging the gap between academic research and software engineering practice.