🤖 machine learning

Evaluating deep learning models for fault diagnosis of a rotating machinery with epistemic and aleatoric uncertainty

This paper presents the first comprehensive comparative study of uncertainty-aware deep learning models for rotating machinery fault diagnosis, demonstrating that deep ensemble architectures outperform other state-of-the-art methods in detecting both epistemic and aleatoric uncertainties while offering shorter inference times.

Original authors: Reza Jalayer, Masoud Jalayer, Andrea Mor, Carlotta Orsenigo, Carlo Vercellis

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Reza Jalayer, Masoud Jalayer, Andrea Mor, Carlotta Orsenigo, Carlo Vercellis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a mechanic trying to diagnose a problem with a giant, spinning industrial motor. You have a computer program (a "Deep Learning Model") that has been trained on thousands of recordings of this motor running perfectly, plus recordings of it with three specific, known problems (like a broken ball bearing or a cracked inner race).

The problem is: What happens when the motor makes a sound the computer has never heard before? Or, what if the microphone recording the sound is just really staticky?

This paper is like a massive "stress test" for different types of computer programs to see which one is best at saying, "Hey, I don't know what this is, or the data is too messy to trust!" instead of confidently guessing the wrong answer.

Here is the breakdown of the study using simple analogies:

1. The Two Types of "Confusion"

The researchers tested the computers against two different kinds of confusion:

Epistemic Uncertainty (The "Blind Spot"): Imagine the computer was trained only on dogs and cats. If you show it a picture of a hamster, it doesn't know it's a hamster. It might guess "dog" or "cat" confidently, but it's wrong. In the factory, this happens when a brand new type of fault appears that the computer has never seen. The computer lacks the knowledge to identify it.
Aleatoric Uncertainty (The "Static"): Imagine the computer is trying to listen to a song, but someone is screaming next to the microphone. The data is there, but it's noisy. The computer might get confused because the signal is dirty. In the factory, this is caused by electrical interference or a shaky sensor.

2. The Contenders: Three Different "Detectives"

The paper compared three different ways of building these computer programs to see which detective is best at spotting the "unknowns" or the "noise":

The "Gambler" (ConvLSTM-D): This model tries to guess the answer, but every time it makes a guess, it randomly forgets a few facts (like a student who closes their eyes and guesses). It does this 10 times. If it gets 10 different answers, it knows it's confused.
The "Probabilist" (Bayesian Neural Network): This model doesn't just have one set of facts; it has a whole library of possible versions of itself. It asks all of them for an opinion. If the library members disagree, it knows it's uncertain.
The "Council" (Deep Ensemble): This is the star of the show. Instead of one detective, they hire a team of four different detectives. They all look at the same problem independently. If the team agrees, it's a safe bet. If they all start arguing, the system knows, "Wait, something is weird here."

3. The Rules of the Game

To test them, the researchers created two scenarios:

The "Unknown Fault" Test: They trained the computers on 5 types of motor problems, then secretly tested them on the 6th type (which they had never seen).
The "Noise" Test: They took clean recordings and added different kinds of static (like the hiss of a radio, the crackle of a spark, or random static) to see if the computers could still tell the difference between a healthy motor and a broken one.

They also tested two different "alarm bells" (thresholds) to decide when to raise the red flag:

The "Panic Button" (Threshold 1): This is a very sensitive alarm. It screams "DANGER" at the slightest hint of confusion. It catches almost all the bad stuff, but it also screams at a lot of normal, healthy motors (false alarms).
The "Balanced Judge" (Threshold 2): This is a new method the authors invented. It tries to find the perfect middle ground where it catches the bad stuff without screaming at the good stuff too much.

4. The Results: Who Won?

When facing "Unknown Faults" (Epistemic): The Deep Ensemble (The Council) was the clear winner. It was the best at saying, "I don't know this," without guessing wrong. The other two models often confidently guessed the wrong fault.
When facing "Noise" (Aleatoric): The Deep Ensemble was again the winner. Interestingly, the more noise there was, the easier it was for the computers to realize something was wrong (because the noise made the data look very strange). However, when the noise was very low (just a little bit of static), it was very hard for any model to tell the difference. Even then, the Deep Ensemble handled the situation better than the others.
Speed: The Deep Ensemble took longer to train (learning the ropes) because they had to teach four detectives instead of one. However, once they were trained, they were actually faster at making a diagnosis than the other models. Why? Because the other models had to run their "guessing game" 10 times for every single check, while the Deep Ensemble just asked its four members once.

5. The Bottom Line

If you are building a system to monitor machines in the real world, the paper suggests you should use the Deep Ensemble approach.

It is the most reliable at spotting new, unknown problems.
It is the most robust when the sensors are noisy.
It is faster to run once it's set up.

The authors also suggest that if you are in a super-critical situation (like a nuclear plant), you should use the "Panic Button" alarm (Threshold 1) to catch everything, even if it means checking a few false alarms. If you are in a less critical situation, use the "Balanced Judge" (Threshold 2) to save time and effort.

In short: When you need a machine to tell you when it's sick, or when it's confused by a new disease, or when the microphone is broken, hiring a team of diverse experts (Deep Ensemble) works better than relying on a single expert who tries to guess or a single expert who is unsure of their own facts.

Technical Summary: Evaluating Deep Learning Models for Fault Diagnosis with Epistemic and Aleatoric Uncertainty

Problem Statement
Rotating machinery, such as motors and turbines, is critical to industrial operations but prone to faults that can cause significant financial loss and safety hazards. While Deep Learning (DL) has improved fault diagnosis by automatically extracting features, most existing methods operate under the assumption that training and future prediction data share the same distribution. In practice, this assumption is often violated by Out-of-Distribution (OOD) data. OOD data arises from two primary sources of uncertainty:

Epistemic Uncertainty (Knowledge Uncertainty): Caused by a lack of knowledge, such as encountering unseen fault types not present in the training set.
Aleatoric Uncertainty (Data Uncertainty): Caused by inherent data variability, such as sensor noise (Gaussian or non-Gaussian) or measurement errors.

Current DL models often fail to quantify this uncertainty, leading to unreliable predictions where the model may confidently misclassify unseen faults or noisy data. There is a need for a comprehensive comparison of uncertainty-aware DL architectures to determine which methods effectively distinguish between in-distribution (ID) and OOD data under varying conditions.

Methodology
The authors conducted a comparative study using the Case Western Reserve University (CWRU) bearing dataset. The study evaluated three state-of-the-art uncertainty-aware DL architectures:

ConvLSTM-D: A Convolutional LSTM with Bernoulli dropout, utilizing Monte Carlo (MC) dropout at inference time to generate prediction distributions.
Bayesian Neural Network (BNN): A network where weights are treated as random variables following a Gaussian distribution, generating outcome distributions through multiple weight samples.
Deep Ensemble: Two configurations were tested:
- De1: Four identical base learners trained in parallel.
- De2: Four base learners with mixed architectures (two identical to De1, two combining Conv+Pool with LSTM).

Experimental Design:

Epistemic Scenarios: Created via a hold-out strategy where one fault class was excluded from training and used as OOD data during validation/testing. Six scenarios were generated, one for each class.
Aleatoric Scenarios: Created by injecting noise into the data. Four noise types were tested: Gaussian, Impulse, Rayleigh, and Weibull. These were applied at three intensity levels (Signal-to-Noise Ratios of -5 dB, 0 dB, and 5 dB).
Uncertainty Quantification: Entropy was used as the metric for uncertainty. Two thresholds were applied to distinguish ID from OOD data:
- $\tau_1$ (Uncertainty Outlier Detection): Based on the Inter-Quartile Range (IQR) of entropy values from ID validation data.
- $\tau_2$ (Uncertainty F1 Score): A novel threshold proposed in this paper, optimized to maximize the F1-score on the validation set by balancing True Positives (OOD detected) and True Negatives (ID correctly identified).

Key Results

Epistemic Uncertainty: All models could detect a portion of OOD data, but Deep Ensemble models (De1 and De2) demonstrated superior performance. De2, in particular, showed exceptional capability in detecting OOD data regardless of the threshold used. ConvLSTM-D and BNN showed notable limitations, especially in specific scenarios (e.g., excluding Class 0 or Class 2).
- Threshold Trade-off: Using $\tau_1$ resulted in higher OOD detection rates but also higher false positives (misclassifying ID data as untrustworthy). $\tau_2$ offered a more pragmatic balance, reducing the need for extensive human re-evaluation of misclassified ID data.
Aleatoric Uncertainty: Noise level significantly impacted performance.
- High Noise (Strong): OOD detection improved significantly across all models as noise amplified prediction entropy. Deep Ensembles maintained superiority, detecting 80–100% of OOD data.
- Low Noise (Slight): Performance declined sharply for all models as ID and OOD entropy distributions overlapped. However, Deep Ensembles still outperformed ConvLSTM-D and BNN, exhibiting a milder degradation in performance.
- Noise Type: Weibull and Rayleigh noises were generally more detectable by Deep Ensembles under strong noise conditions.
Computational Efficiency:
- Training: Deep Ensembles (De1/De2) required more training time due to parallel training of multiple base learners.
- Inference: Deep Ensembles were significantly faster (approx. 3x) during the prediction phase compared to ConvLSTM-D and BNN, which require $K=10$ repeated forward passes for MC sampling.

Significance and Contributions
The paper claims to be the first comprehensive comparative study of state-of-the-art uncertainty-aware DL architectures specifically for rotating machinery fault diagnosis, addressing both epistemic and diverse forms of aleatoric uncertainty (including non-Gaussian noise).

Key contributions include:

Empirical Guidance: The study identifies Deep Ensemble architectures (specifically De2) as the preferred choice for real-world deployment. They offer a superior balance of high OOD detection accuracy, robustness against varying noise levels, and faster inference times.
Novel Thresholding: The introduction of the $\tau_2$ (F1-score based) threshold provides a practical alternative to the standard outlier detection method ( $\tau_1$ ), helping practitioners minimize false positives and reduce the cost of human intervention in post-processing.
Real-World Alignment: By testing non-Gaussian noise types (Impulse, Rayleigh, Weibull) alongside Gaussian noise, the study bridges the gap between theoretical uncertainty analysis and the complex noise environments found in actual industrial sensor data.

Conclusion
The authors conclude that while uncertainty-aware DL is essential for reliable fault diagnosis, the choice of architecture and threshold is critical. Deep Ensembles are recommended for their robustness and efficiency. The selection of the entropy threshold should depend on the application's criticality: a conservative threshold ( $\tau_1$ ) for high-stakes environments where missing OOD data is unacceptable, and a balanced threshold ( $\tau_2$ ) for scenarios where minimizing false alarms is prioritized. Future work is suggested to extend these findings to other datasets (e.g., Paderborn University) and to investigate active learning approaches involving domain experts.