💻 computer science

Is this Build Failure Related to my Patch? An Empirical Study of Unrelated Build Failures in Continuous Integration

This empirical study analyzes 77,354 CI build failures across seven Apache projects to quantify the developer effort wasted on unrelated failures and demonstrates that semi-supervised Positive and Unlabeled (PU) learning models, utilizing features like CI latency and error patterns, can effectively predict such non-actionable failures to help developers prioritize their debugging efforts.

Original authors: Andie Huang, Daniel Alencar da Costa, Grant Dick, Mariam El Mezouar

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andie Huang, Daniel Alencar da Costa, Grant Dick, Mariam El Mezouar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef working in a very busy, chaotic kitchen (this is the Continuous Integration environment). Every few minutes, a new order comes in (a code push), and the kitchen automatically starts cooking a test meal to see if the new ingredients work.

Sometimes, the test meal burns or tastes terrible. Usually, this means the chef who just added the new ingredients made a mistake. But sometimes, the fire alarm goes off because the previous chef left the stove on, or the oven broke, or someone else dropped a tray in the next room. This is what the paper calls an "Unrelated Build Failure."

The problem is that the chef who just added the new ingredients doesn't know why the meal failed. They spend hours (the paper says a median of 4 hours) frantically checking their own spices and knives, trying to prove, "It wasn't me!" This wastes a lot of time and causes stress.

What the Researchers Did

The authors (a team of researchers) decided to investigate this kitchen chaos. They looked at 77,354 "burnt meals" (build failures) from 7 large open-source software projects (like Apache Hadoop and HBase).

The Detective Work: They manually read through thousands of comments left by developers after a failure. They looked for phrases like "this isn't related to my change" or "unrelated." They found about 10,300 cases where developers explicitly said, "This failure wasn't my fault."
The Interview: They picked a smaller, representative sample of 371 of these "not my fault" cases and analyzed them like a detective analyzing a crime scene. They asked: Why did the developers say this wasn't their fault?
- The Findings: The most common reasons were:
  - Unrelated Tests: The test that failed was actually checking something from a different part of the kitchen, not the new ingredient.
  - External Interference: Something outside the kitchen changed (like a supplier delivering bad flour to everyone).
  - Non-reproducible: The fire happened once, but when they tried to make the meal again, it was fine (maybe a random power surge).
  - The "Unspecified" Group: A huge chunk (35%) of the time, developers just said "It's not me" without explaining why.

The Solution: A "Smart Assistant"

Since developers can't always explain why a failure is unrelated immediately, the researchers built a Smart Assistant (a machine learning model) to guess for them.

They used a special technique called PU Learning (Positive-Unlabeled Learning).

The Analogy: Imagine you are trying to teach a dog to find a specific type of ball. You have a few balls you know are the right kind (the Positive examples). But you have a huge pile of mixed balls where you don't know which are the right kind and which are wrong (the Unlabeled pile). You can't just say "everything else is a wrong ball" because some of them might actually be the right kind, you just haven't checked them yet.
How it worked: The researchers fed their model the "known unrelated failures" and the "unknown pile." The model learned to spot patterns that suggest a failure is likely unrelated, even without a clear label.

How Good Was the Assistant?

The model was tested on the same 7 projects.

It was very good at precision (when it said "This isn't your fault," it was usually right, about 70% to 88% of the time).
It was good at recall (it found most of the unrelated failures, though it missed some).
It significantly outperformed random guessing or simple rules.

The "Clues" the Model Used

The researchers found three main clues that helped the model decide if a failure was unrelated:

The Time Gap (CI Latency): If a developer pushed code and waited a long time before triggering the build, it's more likely that someone else broke the kitchen in the meantime.
The "Deja Vu" Error: If the error message looks exactly like one that happened recently, it's probably a repeat of an old problem, not a new one caused by the current chef.
The Chatter: If there are a lot of comments on the issue before the failure happened, it suggests the problem is complex and likely involves other people's work, not just the current push.

The Bottom Line

The paper concludes that by using this "Smart Assistant," developers can get a quick hint: "There's a high chance this failure isn't your fault."

This doesn't mean they can ignore the problem, but it tells them, "Don't spend 4 hours checking your own code. Maybe check the oven or ask the other chef." This helps them stop wasting time on false alarms and get back to cooking (coding) faster.

Important Note: The paper focuses entirely on identifying these failures in software projects. It does not claim this method works for medical diagnosis, financial trading, or any other field outside of software development. It is strictly a tool for software teams to manage their own "kitchen" chaos.

Technical Summary: Is this Build Failure Related to my Patch? An Empirical Study of Unrelated Build Failures in Continuous Integration

1. Problem Statement

In Continuous Integration (CI) environments, multiple developers often push code concurrently. This concurrency can lead to unrelated build failures: legitimate build failures that occur during the CI process but are not caused by the current code push. Unlike flaky tests (which are non-deterministic), these failures are real but stem from factors such as cascading breakages from previous pushes, configuration issues, external dependencies, or environmental instability.

The core problem is that developers must spend significant time investigating these failures to determine if they are responsible. The authors' empirical data reveals that developers invest a median of 4 hours to confirm whether a failure is unrelated to their changes. This "time misspent" creates a burden, potentially leading to "notification fatigue" and reduced adherence to CI best practices if developers perceive builds as constantly failing due to factors outside their control.

2. Methodology

The study employs a mixed-methods approach involving data mining, qualitative analysis, and semi-supervised machine learning across seven Apache Software Foundation (ASF) projects (AMBARI, HADOOP, HBASE, HDDS, HDFS, HIVE, YARN).

2.1 Data Collection and Heuristic Labeling

Dataset: The authors extracted 77,354 CI build failures from JIRA issue reports.
Heuristic-Based Labeling (HL): To identify potential unrelated failures, they developed a keyword-based heuristic. They searched for specific terms in developer comments following a build failure (e.g., "unrelated," "not relevant," "irrelevant," "not connected").
Initial Set: This process identified 10,316 potentially unrelated build failures.
Sample Curation: From this set, a statistically representative sample of 371 truly unrelated failures was manually curated (95% confidence level, 5% confidence interval) through a rigorous document analysis process involving multiple coders.

2.2 Document Analysis (Qualitative)

The authors performed a thematic analysis on the 371 curated failures to understand why developers deemed them unrelated. This involved:

Treating CI logs and developer comments as digital documents.
Inductive coding to generate themes.
Inter-rater reliability checks (Cohen's Kappa scores of 0.76–0.77).

2.3 Feature Engineering

Based on the qualitative findings, the authors extracted 33 features from three dimensions:

Issue Report Level: Priority, parallel issues, cross-project links, duplicate status, etc.
Comment Level: Number of prior comments, shared error messages with previous failures, number of similar failures.
Push Level: CI latency (time from patch submission to build trigger), presence of code/config patches, lines added/modified/deleted in source and config files.

Preprocessing: Collinear features were removed using Spearman's rank correlation ( $|\rho| \ge 0.7$ ) and redundancy analysis ( $R^2 > 0.9$ ) on a per-project basis.

2.4 Modeling: Positive-Unlabeled (PU) Learning

Recognizing that the dataset is imbalanced and that "negative" instances (failures related to the push) cannot be reliably labeled (as developers may not explicitly state a failure is related), the authors adopted a Positive-Unlabeled (PU) learning framework.

Positive Class: Manually confirmed unrelated failures (371 samples, extended to 700 for training).
Unlabeled Class: All other build failures (potentially containing both related and unrelated failures).
Algorithms: Two variants of the Elkan & Noto classifier were implemented:
1. Classic Elkan & Noto: Assumes unlabeled data is a mixture of positive and negative.
2. Weighted Elkan & Noto: Assigns weights to unlabeled examples to account for the possibility that they contain positive instances.
Assumption: The study validates the SCAR (Selected Completely At Random) assumption, ensuring that the labeled positive instances are a representative sample of all positive instances.
Evaluation: Models were evaluated using 10-fold cross-validation and a "leave-one-project-out" cross-project validation strategy. Metrics included Precision, Recall, F1-score, and AUC.

3. Key Contributions

3.1 Empirical Characterization

Time Cost: Quantified the median time developers spend verifying unrelated failures as 4 hours.
Root Causes: Identified seven themes explaining why failures are deemed unrelated:
1. Unrelated Tests (20%): Failures in test classes unrelated to the current push.
2. Unspecified by Developer (34.86%): Developers state "unrelated" without context.
3. External Interference (19.73%): Issues with third-party libraries or dependencies.
4. Non-reproducible (17.03%): Transient infrastructure issues (network, disk, timeouts).
5. Concurrent Commits, Configuration Issues, and Environment Impact.

3.2 Predictive Modeling

Developed the first semi-supervised approach to automatically predict unrelated build failures.
Demonstrated that Weighted Elkan & Noto models consistently outperform baselines (Random, Constant Positive, and Heuristic-based Error Message matching).
Performance:
- AUC: Ranged from 0.63 to 0.97 across projects.
- Precision: Ranged from 0.70 to 0.88.
- Recall: Ranged from 0.30 to 1.00.
- F1-Score: Ranged from 0.44 to 0.91.
Generalization: The models maintained robust performance in cross-project validation, with the Weighted variant showing significant improvements in precision (13–24% gains) over the Classic variant.

3.3 Feature Importance

Permutation feature importance analysis revealed the most influential indicators for unrelated failures:

CI Latency: The time between patch submission and the build-triggering push. Longer latency increases the likelihood of unrelated failures.
Similar Error Messages: Whether the current failure shares an error message with recent failures (indicating cascading breakages).
Number of Prior Comments: The volume of discussion preceding the failure, which may indicate complex, multi-issue contexts.

4. Significance and Claims

The paper claims that its primary contribution is providing a semi-supervised framework to assist developers in triaging build failures. By predicting the probability that a failure is unrelated to the current push, the tool aims to:

Reduce Developer Effort: Allow developers to skip investigating code changes when a failure is likely unrelated, focusing instead on infrastructure or re-running the build.
Improve Decision Making: Help teams decide whether to proceed with merging a contribution despite a build failure.
Identify Systemic Issues: Repeated predictions of unrelated failures can signal misconfigurations, infrastructure network issues, or dependency problems that need systematic monitoring.

The authors explicitly state that their goal is not to achieve perfect generalizability across all software domains immediately, but to demonstrate the viability of this approach in identifying unrelated failures within mature CI ecosystems. They emphasize that their work complements existing research on flaky tests and general build outcome prediction by targeting a specific, often overlooked class of failures that confuse developers in concurrent development settings.