Learning Rewards, Not Labels: Adversarial Inverse Reinforcement Learning for Machinery Fault Detection

This paper proposes an Adversarial Inverse Reinforcement Learning framework for machinery fault detection that learns reward dynamics from healthy operational sequences to generate anomaly scores, thereby enabling robust, label-free fault diagnosis by leveraging RL's sequential decision-making capabilities.

The Gist

Imagine you are a mechanic trying to figure out when a car engine is about to break.

The Old Way (The "Guessing Game")
Most current computer systems for detecting machine faults work like a game of "Spot the Difference" with a single photo. You show the computer one snapshot of the engine, and it guesses, "Is this broken? Yes or No?"
The problem? Machines don't break instantly. They get sick slowly over time, like a person catching a cold. By looking at just one snapshot, the computer misses the story of how the engine is changing. It's like trying to diagnose a disease by looking at a single photo of a patient's face without knowing their medical history.

The New Way (The "Storyteller")
This paper proposes a smarter approach called Adversarial Inverse Reinforcement Learning (AIRL). Instead of guessing, the computer learns to tell a story.

Here is how it works, broken down into simple metaphors:

1. Learning from the "Perfect Student"

Imagine you want to teach a robot how to drive a car perfectly. Usually, you'd have to write a manual saying, "If you see a red light, stop. If you see a pothole, slow down." This is hard because you have to guess every possible rule.

In this paper, the researchers do something different. They don't write rules. Instead, they show the robot only videos of a perfect, healthy engine running smoothly for a long time. They say, "This is what 'good' looks like. Learn the rhythm of it."

The robot becomes a student who memorizes the "perfect dance" of a healthy machine.

2. The "Imposter" and the "Judge"

Once the robot has learned the perfect dance, the researchers set up a game between two AI characters:

• The Generator (The Imposter): This AI tries to create fake engine movements. It tries to mimic the healthy engine but eventually starts to make tiny mistakes (simulating a fault).
• The Discriminator (The Judge): This AI is the expert. Its job is to watch the movements and say, "Is this the real healthy engine, or is this the Imposter faking it?"

As they play this game over and over, the Judge gets incredibly good at spotting the tiniest, subtle differences between a healthy engine and one that is starting to fail. It learns the reward of being healthy.

3. The "Health Score"

Once the Judge is trained, it doesn't need to see a broken machine to know something is wrong. It just watches the engine run.

• If the engine moves exactly like the "Perfect Student" it learned, the Judge gives it a high score (Great job! You are healthy!).
• If the engine starts to stumble, even slightly, the Judge's score drops.

This score acts like a fever thermometer. You don't need to know what virus the machine has; you just need to know its "temperature" is rising.

4. Why This is a Big Deal

The researchers tested this on real data from helicopter gearboxes and industrial machines.

• The Old Methods: Many existing systems screamed "ALARM!" too early (false alarms) or waited too long until the machine was already broken. One popular method (called a "Contextual Bandit") failed completely because it was too busy looking at single snapshots to notice the slow decline.
• The New Method: Their AI spotted the trouble early—before the machine actually broke, but after the very first subtle signs of fatigue appeared. It was more accurate than the "Challenge Winner" of a recent competition and much better than standard tools.

The Bottom Line

This paper is about teaching machines to listen to the story of how a machine ages, rather than just taking a quick photo. By learning what "healthy" looks like over time, the AI can spot the first whisper of a problem long before it becomes a shout, saving factories from expensive breakdowns without needing a human to label every single mistake.

Technical Summary

1. Problem Statement

Machinery Fault Detection (MFD) is critical for industrial reliability but faces a significant bottleneck: the scarcity of labeled fault data in real-world settings.

• Limitations of Supervised Learning: While dominant (approx. 81% of studies), it struggles due to the lack of extensive fault labels.
• Limitations of Existing RL Approaches: Most current Reinforcement Learning (RL) methods for MFD reduce the problem to Contextual Bandits (CB). In these setups:
  • Sensor samples are treated as independent states.
  • Agents perform one-shot classifications.
  • The discount factor is set to zero ($\gamma=0$), effectively ignoring the temporal structure and sequential nature of machine degradation.
  • This simplification fails to capture the gradual accumulation of fatigue damage.

The authors propose a paradigm shift: moving from static classification to sequential decision-making by formulating MFD as an Offline Inverse Reinforcement Learning (IRL) problem.

2. Methodology

The core innovation is the application of Adversarial Inverse Reinforcement Learning (AIRL) to learn the reward dynamics of healthy machine operations directly from data, bypassing the need for manual reward engineering or fault labels.

A. State Transition Construction (State-Only Imitation Learning)

Since industrial datasets often lack recorded control inputs (actions), the authors adopt a State-Only Imitation Learning (SOIL) formulation:

• Data Segmentation: Normalized vibration signals are segmented into fixed-length windows.
• Proxy Action: In the absence of explicit control actions, the system's natural temporal evolution is treated as a "proxy action."
  • State ($s_t$): Current window.
  • Action ($a_t$): The transition to the next window ($x_{t+1}$).
• This allows the AIRL discriminator to evaluate the plausibility of the transition dynamics ($s_t \to s_{t+1}$) against the distribution of healthy "expert" trajectories.

B. Adversarial Reward Learning

The framework utilizes a GAN-like structure with two components:

1. Generator ($\pi$): Trained to mimic the dynamics of the healthy expert.
2. Discriminator ($D$): Trained to distinguish between transitions from the healthy expert distribution and those generated by the policy.

The discriminator is structurally constrained to recover a meaningful reward function ($r_\theta$):
$$D(s, a, s') = \sigma(r_\theta(s, a) + \gamma V_\phi(s') - V_\phi(s) - \log \pi(a|s))$$

• Here, $\sigma$ is the sigmoid function, and $V_\phi$ represents the value function.
• This constraint forces the learned term $r_\theta(s, a)$ to act as a robust reward function (or health score), disentangled from the system dynamics. High rewards indicate alignment with healthy dynamics; low rewards signal deviations.

C. Anomaly Scoring

Once trained, the model generates an anomaly score for a trajectory $\tau$ by inverting the average discriminator confidence:
$$\text{Score}(\tau) = 1 - \frac{1}{T} \sum_{t=0}^{T} D(s_t, a_t, s_{t+1})$$

• High Score: Indicates a deviation from normal behavior (fault).
• Fault Onset Detection: Determined by applying dynamic thresholding methods (e.g., Otsu's method, K-means) to the anomaly score.

3. Key Contributions

1. First Application of AIRL to MFD: Introduces the first framework applying Adversarial Inverse Reinforcement Learning specifically for machinery fault detection.
2. Sequential Modeling: Moves beyond Contextual Bandits by respecting the temporal structure of machine degradation, utilizing the discount factor to model the accumulation of damage.
3. Label-Free Learning: Eliminates the need for fault labels or manual reward engineering by learning reward dynamics solely from healthy operational sequences.
4. Interpretability: The learned reward function serves as an interpretable "health score," providing a clear signal of deviation from normal operation.

4. Experimental Results

The framework was evaluated on three run-to-failure benchmark datasets:

• HUMS2023 (Helicopter gearbox fatigue)
• IMS
• XJTU-SY

Key Findings on HUMS2023:

• Detection Timing: The AIRL model detected the fault onset on Day 22 (File #163).
  • This is earlier than the official Ground Truth (Day 24) and the Challenge Winner (Day 23).
  • It falls between the FRESH filter (Day 22, File #127) and the Challenge Winner, providing a valuable early warning window.
• Comparison with Baselines:
  • Standard Baselines (IF, OCSVM, AE): Flagged anomalies prematurely (Day 21), leading to false positives.
  • Sequential Baselines (LSTM-AE, LSTM-VAE): Improved precision but still triggered earlier than AIRL.
  • Contextual Bandit (CTQN) Baseline: Failed completely, classifying the entire test set as normal. This confirms that without modeling state transitions ($\gamma=0$), agents cannot perceive gradual fatigue.
• Post-Detection Consistency (PDC): AIRL maintained a steady anomaly rate (~65%) after fault onset, demonstrating superior stability compared to other methods.

5. Significance and Conclusion

This work bridges the gap between Reinforcement Learning and Industrial Fault Detection by aligning RL's sequential reasoning with the temporal structure of machinery degradation.

• Robustness: The method proves that learning the dynamics of health is superior to merely classifying isolated observations.
• Practicality: It offers a solution for data-driven industrial settings where fault labels are unavailable, enabling early and robust fault detection without manual reward engineering.
• Future Directions: The authors plan to extend the framework to multi-sensor fusion and incorporate uncertainty-aware thresholding to further reduce false alarms in variable operating conditions.

In summary, this paper demonstrates that treating MFD as a sequential IRL problem, rather than a static classification task, yields earlier, more reliable, and more interpretable fault detection results.