Technical Summary: Applications of Causality in Software Testing

Problem Statement

As software systems grow in complexity through interacting components, adaptive behaviors, and data-driven elements, traditional testing faces significant challenges. These include the absence of reliable oracles, intricate input-output behaviors, and the difficulty of interpreting outcomes under confounding effects and spurious correlations. A fundamental issue is the inability to distinguish genuine causal relationships from mere statistical associations. Traditional testing and debugging often rely on correlational signals from execution traces, which can be distorted by hidden confounders, leading to conclusions that lack interpretability and reliability. While causal inference offers a principled framework for reasoning about cause-and-effect rather than correlation, its adoption in software testing remains fragmented across different tasks and research communities, lacking a systematic analysis of how it is positioned, what techniques are used, and where the gaps lie.

Methodology

This study employs a rapid review methodology following the guidelines of Cartaxo et al. and complemented by backward and forward snowballing techniques recommended by Wohlin.

• Search Strategy: Queries were conducted across ACM Digital Library, IEEE Xplore, Scopus, and SpringerLink for peer-reviewed studies published between 2010 and 2025. Search strings combined terms related to software testing/verification/validation with causality/causal inference/causal graphs.
• Selection Process: Initial queries yielded 373 unique papers. After screening titles, abstracts, and full texts against inclusion/exclusion criteria, 14 studies were retained. Snowballing added 13 more, resulting in a final dataset of 27 primary studies.
• Analytical Framework: The authors synthesized the literature using a layered causal inference pipeline grounded in Pearl's Structural Causal Model (SCM) framework. This taxonomy organizes techniques into four stages:
  1. Causal Representation
  2. Causal Structure Discovery
  3. Causal Identification
  4. Causal Effect Estimation
• Data Extraction: Two authors independently extracted data regarding the functional role of causal inference, testing phases, techniques, assumptions, and limitations, resolving disagreements through discussion.

Key Contributions

The paper makes four primary contributions:

1. Systematic Synthesis: It presents the first structured rapid review of 27 primary studies at the intersection of causal inference and software testing.
2. Layered Taxonomy: It proposes a four-layer perspective (representation, discovery, identification, estimation) and maps these to specific testing phases (design, execution, interpretation, debugging).
3. Risk Analysis: It identifies recurring cross-layer methodological risks that affect validity and scalability.
4. Research Agenda: It outlines key directions for advancing causal methods in software testing based on current patterns and gaps.

Results

1. Positioning of Causal Inference (RQ1)

• Functional Roles: Causal inference primarily serves as an enhancement mechanism for existing techniques (e.g., improving fault localization). A smaller subset uses it as a methodological foundation for the testing framework, and a limited number reformulate testing itself as a causal inference problem.
• Testing Phases: Integration is most common in Result Interpretation and Debugging, where causal reasoning helps distinguish spurious correlations and perform root cause analysis. Fewer studies integrate causal inference into Test Design or Test Execution.

2. Techniques and Assumptions (RQ2)

The review categorizes techniques into the four-layer pipeline:

• Representation: Uses Structural Causal Models (SCMs), Directed Acyclic Graphs (DAGs), and Program Dependence Graphs (PDGs) to encode dependencies.
• Discovery: Employs constraint-based, score-based, and intervention-enhanced methods to infer graphs from data when structures are not predefined.
• Identification: Utilizes graphical criteria (backdoor, d-separation) and explicit interventions (input perturbation, fault injection, mutation) to define valid estimands.
• Estimation: Applies statistical and ML methods (Regression, Double Machine Learning, Instrumental Variables, Counterfactuals) to quantify effects like Average Treatment Effect (ATE) or Conditional Average Treatment Effect (CATE).

Underlying Assumptions: The application of these techniques relies on strict assumptions, including causal model adequacy, structural stability, faithfulness, no unmeasured confounding, and treatment well-definedness.

3. Challenges and Limitations (RQ3)

The review identifies four layers of risks:

• Representation Risk: Model dependence and abstraction gaps. Translating program semantics to causal variables often leads to misspecified graphs due to omitted variables or incorrect mapping of dependence to causality.
• Discovery Risk: Structural uncertainty and scalability issues. Automatic discovery struggles with sparse failures, cyclic dependencies, and high computational overhead in large systems.
• Identification Risk: Unverifiable assumptions and intervention constraints. Hidden states and partially observable executions make confounding control difficult. Furthermore, required interventions (e.g., fault injection) may be infeasible or unsafe.
• Estimation Risk: Limited support and model sensitivity. Sparse failure cases and imbalanced data lead to high-variance or biased estimates. Results are highly sensitive to modeling choices and data quality.
• Generalizability: Many studies rely on small benchmarks or simulations, leaving applicability to large real-world systems uncertain.

4. Research Gaps and Directions (RQ4)

The authors propose a research agenda focusing on:

• Automated Representation: Developing systematic mappings from program structure to causal variables to reduce modeling errors.
• Scalable Discovery: Creating methods that handle limited failure cases and noisy data in large systems, potentially by integrating domain knowledge.
• Practical Interventions: Designing feasible and safe intervention strategies for real environments.
• Robust Estimation: Improving estimation under incomplete data and hidden confounders through uncertainty detection and adaptive data collection.
• Validation: Establishing broader benchmarks, clearer uncertainty reporting, and better integration into debugging workflows.

Significance

The paper claims that while causal inference offers a principled framework for software testing, its practical impact is currently constrained by model misspecification, strong identification assumptions, scalability limitations, and a lack of real-world validation. By organizing existing work through a layered causal inference perspective, this review provides a structured foundation for future empirical and methodological work. It aims to unify disparate contributions and guide the community toward developing more robust, scalable, and empirically grounded causal testing methodologies.