🤖 machine learning

Bug-Report-Driven Fault Localization: Industrial Benchmarking and Lesson Learned at ABB Robotics

This study demonstrates that traditional machine learning models utilizing term frequency-inverse document features can outperform fine-tuned transformer-based language models in industrial fault localization when relying solely on textual bug reports from ABB Robotics, challenging the assumption that advanced AI models universally excel in domain-specific maintenance contexts.

Original authors: Pernilla Hall, Anton Ununger, Riccardo Rubei, Alessio Bucaioni

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pernilla Hall, Anton Ununger, Riccardo Rubei, Alessio Bucaioni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Finding a Needle in a Haystack

Imagine you work in a massive, ancient library (the software system) that has been growing for decades. Every day, people write notes on sticky pads (bug reports) saying, "Something is wrong here!" but they don't know exactly where in the library the problem is.

Usually, a librarian (a developer) has to read the note, guess which section of the library it might be in, and then physically walk through thousands of shelves to find the broken book. This takes a long time and costs a lot of money.

The Goal: The researchers wanted to see if a computer could read those sticky notes and instantly say, "I think the problem is in the 'Robot Arm' section," so the librarian doesn't have to search the whole building.

The Challenge: No Blueprints, Just Notes

In many modern computer studies, the AI gets to look at the library's blueprints, the books themselves, and even see people walking around (code, execution traces, logs).

But in the real industrial world (like at ABB Robotics, where this study happened), the AI cannot see the blueprints or the books. It is strictly forbidden from looking at the source code for security reasons. The AI only has the text of the sticky note. It has to guess the location based only on the words written by the person who found the bug.

The Experiment: The "Guessing Game"

The researchers gathered about 5 years of these sticky notes (1,867 reports) from a real ABB Robotics system. They linked each note to the actual place in the code where the fix was made. This gave them a "answer key" to test their AI.

They set up a contest between two types of "guessers":

The Old-School Detectives (Traditional Models): These are simpler algorithms (Logistic Regression, SVM, Random Forest) that look for specific keywords. If the note says "motor," they guess "motor section." They use a technique called TF-IDF, which is like counting how unique and important specific words are in a sentence.
The Super-Intelligent Readers (AI Transformers): These are fancy, modern Large Language Models (RoBERTa and Distil-RoBERTa). They are known for understanding deep context and nuance, like a human who understands that "the thing that spins" might mean "motor" even if the word "motor" isn't there.

The Twist: Because there weren't enough notes to train the Super-Intelligent Readers properly, the researchers also tried "Data Augmentation." Think of this as a teacher taking a student's note and rewriting it in different ways (using synonyms, swapping word order) to create more practice examples without changing the meaning.

The Results: The Simple Detective Wins

The results were surprising to many in the tech world:

The Old-School Detectives won. The simple models that just looked for keywords (TF-IDF) performed better than the fancy Super-Intelligent Readers.
Why? The library (the software system) uses very specific, technical jargon. The simple models were excellent at spotting these specific keywords. The fancy AI models, which are trained on general internet data, got a bit confused by the specialized language and didn't have enough specific examples to learn the "secret code" of this particular library.
The Power of Practice: When the researchers used the "Data Augmentation" (rewriting the notes to create more practice), the Random Forest model (one of the Old-School Detectives) got even better. It became the strongest performer overall.
The Fancy Readers: The Super-Intelligent Readers did okay, but they didn't beat the simple keyword counters. In fact, they sometimes struggled more because the dataset was small and the "classes" (types of bugs) were very unbalanced (some sections had hundreds of bugs, others had only a few).

The Bottom Line: What This Means for Industry

The study concludes that for industrial maintenance, you don't always need the most expensive, complex AI.

Accuracy: The best simple model could guess the correct section of the library as the top choice about 53% of the time. If you let it give a shortlist of the top 5 guesses, it was right about 86% of the time.
Practicality: This is huge. If a developer knows the problem is in one of 5 specific sections instead of searching the whole building, they save hours of work.
Safety & Cost: Because the simple models don't need to access the source code or connect to the cloud, they are safer (no data leaks) and cheaper to run.

Summary Analogy

Imagine you are trying to find a specific recipe in a giant cookbook based only on a customer's description: "It tasted like burnt sugar and the texture was weird."

The Fancy AI tries to understand the feeling of burnt sugar and the concept of texture. It's smart, but it might overthink it.
The Simple Model just looks for the words "burnt" and "sugar" and immediately points to the "Caramel" chapter.

In this specific industrial kitchen, the Simple Model was faster and more accurate because the customers used very specific words that always pointed to the same chapter. The researchers found that for this job, simple is better than complex.

1. Problem Statement

In industrial software maintenance, identifying the location of a fault (fault localization) is a time-consuming and costly bottleneck. Developers often rely on textual bug reports (titles and descriptions) during the triage phase, lacking access to execution traces, dynamic logs, or full code context. While Machine Learning (ML) and Large Language Models (LLMs) have shown promise in open-source contexts, their applicability in confidential, proprietary industrial environments remains under-explored.

The core challenge addressed is: Can natural language processing (NLP) models, using only the text of bug reports, accurately predict the specific software components (subfolders) where a fix is required, without access to source code or runtime artifacts?

2. Methodology

Data Source and Context

Dataset: Proprietary data from ABB Robotics (Västerås, Sweden), spanning five years of resolved bug reports.
Volume: 660 unique bug reports linked to verified code commits.
Labels: The target was defined at the subfolder (component) level rather than the individual file level. This abstraction aligns with developer workflows, reduces label sparsity, and respects architectural coherence.
Constraints: The study was conducted under strict confidentiality (on-premise execution), with no access to source code, call graphs, or execution traces during inference.

Experimental Pipeline

The study framed fault localization as a multi-label text classification and ranking problem. The pipeline included:

Preprocessing: Text normalization (lowercasing, de-camelcasing, lemmatization, stop-word removal) and removal of non-textual metadata (e.g., severity, assignee) to ensure a "text-only" scenario.
Feature Extraction:
- Traditional ML: TF-IDF (Term Frequency–Inverse Document Frequency) vectors and Sentence Embeddings (using all-mpnet-base-v2).
- Transformers: Native tokenization for RoBERTa-Base and Distil-RoBERTa.
Data Augmentation: To address class imbalance and limited data, the authors applied synonym replacement (WordNet-based) and random word swapping to the training set. Strategies included augmenting the entire set vs. targeting only under-represented labels.
Models Evaluated:
- Traditional: Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF).
- Deep Learning: Fine-tuned RoBERTa-Base and Distil-RoBERTa.
Evaluation Metrics: Ranking-centric metrics were used to simulate developer decision-making:
- Top- $k$ Accuracy (Hit@k) for $k \in \{1, 3, 5, 10\}$ .
- Recall@k, Mean Average Precision (MAP), and Mean Reciprocal Rank (MRR).

3. Key Contributions

Industrial Benchmark: A large-scale evaluation of text-only fault localization on a proprietary, real-world industrial system, contrasting with previous studies reliant on open-source data or auxiliary artifacts (like stack traces).
Model Comparison: A controlled comparison between lightweight, interpretable models (LR, SVM, RF) and fine-tuned Transformer-based LLMs under identical industrial constraints.
Empirical Evidence on LLMs: Findings challenging the assumption that LLMs universally outperform classical methods in data-constrained, domain-specific industrial settings.
Actionable Guidance: Practical recommendations for industry adoption, identifying when simple models suffice versus when complex models add marginal value.
Reproducibility: A public replication package (excluding proprietary data) containing scripts, configurations, and hyperparameters.

4. Key Results

Performance Comparison (RQ1 & RQ2)

Traditional Models vs. LLMs: Contrary to expectations, traditional models using TF-IDF features consistently outperformed or matched the fine-tuned RoBERTa models.
- Best Performer: Logistic Regression (LR) with TF-IDF and Random Swap augmentation achieved the highest MRR (0.6655) and tied for the best Top-1 Accuracy (52.63%).
- Random Forest (RF): With Full Synonym Replacement augmentation, RF achieved the highest MAP (0.6171) and Recall@1 (42.65%).
- SVM: Performed best on Top-5 Accuracy (85.53%) and Recall@5 (78.51%) without augmentation.
LLM Performance: Neither RoBERTa-Base nor Distil-RoBERTa surpassed the strongest TF-IDF baselines. Distil-RoBERTa generalized slightly better than RoBERTa-Base but still trailed behind the linear models.
Feature Importance: TF-IDF features significantly outperformed Sentence Embeddings for traditional models. This suggests that in this specific domain, precise lexical cues and domain-specific terminology are more predictive than generalized semantic similarity.

Impact of Data Augmentation

Augmentation had a model-dependent effect. It substantially improved Random Forest performance but offered only marginal or slightly negative gains for LR and SVM (likely because perturbations diluted discriminative keywords).
LLMs benefited from augmentation but did not close the gap with the best TF-IDF baselines.

Quantitative Highlights

Top-1 Accuracy: ~53% (meaning the first recommended component was correct in over half the cases).
Top-5 Accuracy: Up to ~86% (developers can find the correct component within the top 5 recommendations in the vast majority of cases).
MAP/MRR: ~0.61–0.66, indicating strong ranking quality.

5. Significance and Implications

Cost-Effectiveness & Deployability: The study demonstrates that lightweight, locally executable models (LR/SVM with TF-IDF) are sufficient for industrial fault localization. They require no source code access, no external APIs, and minimal computational resources, making them ideal for confidential environments.
Challenging the "LLM Hype": In data-constrained, domain-specific industrial contexts with high class imbalance, complex Transformer models do not necessarily provide a performance advantage over well-tuned classical models. The "black box" nature and computational cost of LLMs may not be justified when simpler models achieve comparable or superior results.
Practical Adoption: The results suggest a pragmatic adoption path:
1. Start with LR or SVM using TF-IDF for rapid triage.
2. Introduce Random Forest with augmentation if label imbalance becomes a limiting factor.
3. Integrate ranked lists directly into bug trackers to narrow the search space, acting as an assistive tool rather than a fully automated locator.
Validity: While limited to a single system (threat to external validity), the rigorous on-premise setup and realistic constraints provide a highly credible benchmark for industrial AI adoption.

In conclusion, the paper provides empirical evidence that historical bug reports can be systematically leveraged for scalable, low-cost, AI-assisted fault localization, validating the use of interpretable, text-only models as a robust complement to traditional debugging practices in industry.