💻 computer science

Can Causality Cure Confusion Caused By Correlation (in Software Analytics)?

This study investigates whether integrating causality-aware split criteria into symbolic models can enhance their stability and robustness in software analytics compared to traditional correlation-based methods, while evaluating the trade-offs in predictive performance and comparing these automated results against human expert judgments.

Original authors: Amirali Rayegan, Tim Menzies

Published 2026-02-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amirali Rayegan, Tim Menzies

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car. You have a huge pile of data showing what happened in the past: "When the driver turned left, the car went left," or "When it rained, the driver slowed down."

The robot builds a rulebook (a Decision Tree) to make decisions. But here is the problem: The robot is currently using Correlation as its guide. It sees that "Rain" and "Slowing Down" happen together, so it assumes Rain causes the slowing. But what if the real reason is that a traffic light turned red, and that caused both the rain (a coincidence) and the slowing?

This is the core confusion the paper addresses: Just because two things happen together doesn't mean one causes the other.

The Problem: The "Shaky" Rulebook

The authors point out that current software tools (like the robot's rulebook) are incredibly unstable.

The Analogy: Imagine you ask a group of 20 different chefs to write a recipe for "Perfect Soup" using the same bag of ingredients. If they are just looking at what ingredients usually appear together (correlation), you might get 20 completely different recipes. One chef says "Add salt because it's always with pepper," another says "Add salt because it's always with onions."
The Result: If you change the data slightly (maybe you add one more data point or shuffle the order), the robot's rulebook changes drastically. It might flip from saying "Feature A is the boss" to "Feature B is the boss." This makes engineers lose trust in the software because the answers keep changing.

The Proposed Solution: The "Causal" Detective

The researchers want to upgrade the robot's brain. Instead of just looking at what happens together, they want the robot to act like a detective looking for Causality.

The Analogy: Instead of just seeing that "Ice Cream Sales" and "Shark Attacks" both go up in July (Correlation), the detective asks, "Is there a hidden third thing?" The detective finds the real cause: Summer Heat. The heat causes people to buy ice cream, and the heat causes people to swim (leading to shark attacks).
The Method: The paper proposes a new way to build the robot's rulebook. Before the robot makes a rule, it checks: "If I remove the 'Summer Heat' variable, does the link between Ice Cream and Sharks disappear?" If yes, the robot ignores that link as a fake correlation. It only keeps rules that look like true cause-and-effect.

The Experiment: Testing the New Brain

The researchers tested this idea using over 120 different real-world software problems (like tuning cloud servers, predicting software bugs, or optimizing video encoding).

They compared three things:

The Old Robot: Uses standard correlation (looks for patterns).
The New Robot: Uses "Causal" logic (looks for causes and filters out fake links).
Human Experts: Real software engineers asked to guess what causes what.

The Questions they asked:

RQ1 (Humans vs. Robots): Are human experts actually stable? Do they all agree on what causes what, or do they argue just as much as the robots?
RQ2 (Stability): If we give the robot the "Causal" brain, will its rulebook stop changing every time we tweak the data? Will it be more reliable?
RQ3 (The Trade-off): Will being "causal" make the robot worse at its actual job (like driving the car or optimizing the server)? Sometimes, being too picky about causes might make you miss good shortcuts.

The Big Picture

The paper is essentially asking: "Can we cure the confusion caused by fake patterns by teaching our software to think about cause and effect?"

If the answer is "Yes," then software engineers can finally trust their tools to give consistent, reliable advice, rather than getting a different answer every time they run the program. It's about moving from "It usually happens this way" to "This is why it happens," making software analytics more trustworthy and less shaky.

Based on the provided text, here is a detailed technical summary of the paper "Can Causality Cure Confusion Caused By Correlation (in Software Analytics)?" by Amirali Rayegan and Tim Menzies.

1. Problem Statement

Software analytics relies heavily on symbolic models (e.g., decision trees) for explainable tasks like defect prediction and configuration tuning. However, these models face a critical stability crisis:

Correlation vs. Causation: Traditional models use correlational split criteria (variance reduction, information gain) that identify statistical associations but cannot distinguish between direct causation ( $X \to Y$ ), reverse causation ( $Y \to X$ ), or confounding ( $Z \to X, Z \to Y$ ).
Instability: Both correlation-based models and existing causal discovery algorithms suffer from high instability. Minor perturbations in data sampling or preprocessing lead to radically different model structures and explanations.
The Gap: While causal discovery algorithms attempt to fix this, they are often NP-hard and rely on heuristics that also produce unstable graphs. Conversely, correlation-based models are unstable because they elevate spurious, confounded features to decision rules.
Core Question: Can incorporating causality-aware split criteria into symbolic models improve their stability and robustness without sacrificing predictive or optimization performance?

2. Methodology

The study utilizes a preregistered bootstrap-ensemble protocol to evaluate stability across 120+ multi-objective optimization tasks from the MOOT repository.

A. Experimental Setup

Baseline Model (EZR): A lightweight, label-efficient framework that builds decision trees using correlation-based splitting (variance reduction). It optimizes for the "win score" (distance-to-heaven metric).
Proposed Model (Causal-Tree): A modified symbolic model that replaces variance reduction with causality-aware split criteria.
- Split Criterion: Uses Conditional Entropy ( $H(Y|X)$ ) normalized by total entropy. Lower scores indicate attributes that better explain the target, suggesting a causal link.
- Confounder Filtering (Backdoor Protection): Before tree construction, the system applies Conditional Mutual Information checks. If a feature's association with the target disappears when conditioning on another variable (matching the backdoor pattern $Z \to [X, Y]$ ), it is flagged as a confounder and removed.
Human Benchmark: A survey of domain experts (Subject Matter Experts - SMEs) to establish a baseline for human causal judgment stability.

B. Research Questions (RQs)

RQ1 (Human vs. Automated Stability): Do human experts reach stable conclusions about causal relationships, and how does this compare to correlation-based automated models?
RQ2 (Stability Impact): Does using causality-aware split criteria produce more stable model structures and optimization outcomes compared to traditional correlation-based trees?
RQ3 (Performance Trade-offs): Does the gain in stability come at the cost of predictive accuracy or optimization performance?

C. Evaluation Metrics

Stability: Measured via variance in win-score assignments and Gini Impurity of feature selection across 20 bootstrap iterations.
Performance: Measured by the distance-to-heaven (d2h) metric (lower is better) and Win Score (0–100).
Statistical Tests: Kolmogorov-Smirnov (KS) test for distribution differences and Cliff's Delta for effect size.

3. Key Contributions

Causal Splitting Framework: The paper proposes a novel method to integrate causal reasoning into decision trees by combining conditional entropy for splitting and conditional mutual information for confounder filtering.
Empirical Stability Benchmark: It establishes a rigorous protocol to measure the stability of symbolic models against human expert judgments, moving beyond simple accuracy metrics.
Comprehensive Dataset Scope: The study evaluates the approach across 120+ diverse datasets (software configuration, cloud tuning, project health, financial, and behavioral data), ensuring generalizability.
Differentiation from Causal Discovery: Unlike full causal graph discovery (which is unstable), this approach embeds causal logic directly into the tree-building process to create "causal trees" that are more robust to data perturbations.

4. Results (Expected/Planned)

Note: As this is a Registered Report (study design phase), the paper outlines the analysis plan rather than final empirical results. However, the expected outcomes are defined as follows:

RQ1: The study anticipates comparing the variance of human expert judgments against the variance of EZR trees. The goal is to determine if human intuition is inherently more stable than current correlational heuristics.
RQ2: The hypothesis is that causality-aware trees will exhibit lower variance in feature selection and optimization outcomes across bootstrap samples compared to EZR, indicating higher robustness.
RQ3: The study aims to prove that increased stability does not significantly degrade optimization performance (d2h scores). If causal trees are both more stable and equally performant, they represent a superior approach for explainable AI in SE.

5. Significance

Trust and Reproducibility: By addressing the instability of current software analytics models, this work aims to restore practitioner trust in data-driven explanations.
Actionable Insights: Moving from "seeing" correlations to "doing" interventions (causal reasoning) allows software engineers to make more reliable configuration changes and defect predictions.
Methodological Shift: It challenges the dominance of pure correlation-based learners in SE, proposing a hybrid approach that leverages causal filtering to prune spurious relationships, leading to more coherent and reliable decision rules.
Human-Centered Analytics: By benchmarking against human experts, the study bridges the gap between automated machine learning and domain intuition, ensuring that automated explanations align with human causal understanding.

In summary, the paper proposes that causality-aware splitting can "cure" the confusion caused by correlation by filtering out spurious associations, thereby creating symbolic models that are not only interpretable but also stable, robust, and reliable for real-world software engineering tasks.