🤖 AI

FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation

This paper introduces FISHER, a foundation model for multi-modal industrial signal representation that addresses data heterogeneity through novel sub-band modeling and teacher-student self-distillation, achieving superior diagnostic accuracy and versatility across 19 datasets while significantly outperforming larger state-of-the-art models.

Original authors: Pingyi Fan, Anbai Jiang, Shuwei Zhang, Xinhu Zheng, Zhiqiang Lv, Bing Han, Wenrui Liang, Junjie Li, Wei-Qiang Zhang, Yanmin Qian, Xie Chen, Jia Liu

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pingyi Fan, Anbai Jiang, Shuwei Zhang, Xinhu Zheng, Zhiqiang Lv, Bing Han, Wenrui Liang, Junjie Li, Wei-Qiang Zhang, Yanmin Qian, Xie Chen, Jia Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a factory floor as a giant, noisy orchestra. Every machine—pumps, gears, motors, transformers—plays its own instrument. Sometimes, they play perfectly in tune (normal operation), and sometimes, a violin string snaps or a drumstick breaks (a fault).

For years, trying to listen to this orchestra to find the broken instrument has been a nightmare for engineers. Here is why:

The "M5" Problem: The paper calls the mess of data the "M5 problem." It means the data comes in Many forms (sound, vibration, electricity), at Many different speeds (sampling rates), with Many different scales, for Many different tasks, and usually with Missing information about what a "broken" machine actually sounds like.
The Old Way: Previously, engineers had to build a tiny, specialized "listener" for every single machine. If you wanted to check a pump, you needed one model. For a gear, a different one. If the pump changed its speed, the model broke. It was like hiring a different translator for every single word in a book.

Enter FISHER: The Universal Translator

The authors created FISHER, a "Foundation Model." Think of this not as a specialized mechanic, but as a super-smart, universal translator that can understand the language of any machine, no matter what it is or how fast it's spinning.

Here is how FISHER works, using some simple analogies:

1. The "Lego Block" Strategy (Solving the Speed Problem)
Imagine you are looking at a picture of a sunset.

Old models tried to force every picture to be the same size. If the picture was high-resolution (fast sampling rate), they would squish it down to fit a small frame, losing all the tiny details of the clouds.
FISHER is different. It realizes that a high-resolution picture is just a low-resolution picture plus some extra Lego blocks on top.
Instead of squishing the image, FISHER breaks the sound wave into "sub-bands" (like Lego strips). It analyzes the low-frequency strips first, then adds the high-frequency strips on top. This way, it never loses the fine details, even if the machine is recording at a super-fast speed. It adapts to the speed naturally, without needing to resize the data.

2. The "Music School" Training (The Surprise Twist)
You might think a model designed to fix factory machines needs to be trained on factory machines.

The Paper's Claim: Surprisingly, the authors found that training on industrial data was actually worse than training on music and general audio.
The Analogy: Imagine you want to teach a student to recognize a broken car engine.
- Option A: Show them 10 hours of a car engine that mostly sounds the same (stationary). They get bored and learn very little.
- Option B: Show them 10,000 hours of jazz, rock, and classical music. The music has wild changes in tempo, rhythm, and pitch.
- The Result: The student trained on music learns to listen for patterns and changes much better. When they finally hear the car engine, they can instantly spot the "off-note" because they are so good at listening to complex sounds. The paper found that music and audio data provided the perfect "temporal variability" to teach the model how to listen, which it then applied to the factory floor.

3. The "No-Training" Superpower
Usually, AI models need to be "fine-tuned" (re-trained) for every new job.

FISHER's Trick: The authors tested FISHER on 19 different datasets covering sounds, vibrations, voltage, and current. They used a "K-Nearest Neighbor" method, which is like saying, "Does this new sound look like the sounds I've already seen?"
The Result: FISHER didn't need to be re-trained at all. It just looked at the new data and said, "I know what this is." It beat 24 other top-tier models, often using a fraction of the computer power (up to 16 times smaller) to achieve better accuracy.

The "RMIS" Benchmark

To prove this wasn't a fluke, the authors built a new testing ground called RMIS (Representation of M5 Industrial Signals). It's like a standardized "driver's license test" for industrial AI.

It includes 19 different datasets.
It tests two main skills: Anomaly Detection (Is this sound weird?) and Fault Diagnosis (What exactly is broken?).
The Score: FISHER got the highest score (62.23%), beating the next best model by a clear margin.

The Bottom Line

The paper claims that by treating high-speed data as "extra layers of information" rather than a problem to be fixed, and by training on diverse music/audio instead of boring factory data, we can build a single, small, efficient model that understands the "language" of any industrial machine. It's a universal listener that works out of the box, saving engineers from building a new model for every single machine in the factory.

Technical Summary: FISHER – A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation

1. Problem Definition: The M5 Challenge

The paper identifies that industrial signal analysis is currently hindered by severe data heterogeneity, characterized as the M5 problem. Existing solutions rely on specialized models trained on small-scale datasets, which lack robustness and scalability across different machines, modalities, and tasks. The M5 problem is defined by five specific challenges:

Multi-modal: Signals vary across sound, vibration, voltage, and current.
Multi-sampling-rate: Sensor sampling rates vary widely (3 kHz to 100 kHz) to balance cost and Nyquist requirements.
Multi-scale: Signal characteristics differ based on operating mechanisms (sliding, rotating, static) and working conditions.
Multi-task: Downstream tasks include anomaly detection, fault diagnosis, and remaining useful life (RUL) estimation.
Minim fault: Fault data is scarce, and class distributions are often imbalanced.

The authors argue that the primary bottleneck is not data quantity but data quality. Industrial signals are often stationary and form disjoint clusters in the data space, making them insufficient for training general-purpose foundation models via large-scale pre-training alone.

2. Methodology: The FISHER Architecture

To address the M5 problem, the authors propose FISHER (Foundation model for multi-modal Industrial Signal compreHEnsive Representation). The methodology is built upon a prioritized roadmap focusing on three pillars: special model design for sampling rates, large-scale pre-training with high-quality data, and high-quality representation for fault detection.

A. Sub-band Modeling (Addressing Multi-sampling-rate)

The core innovation of FISHER is a novel sub-band modeling approach that treats sampling rate increments as concatenated sub-band information, eliminating the need for resampling.

STFT Conversion: Signals are converted to Short-Time Fourier Transform (STFT) spectrograms with fixed time-frequency resolution ( $\Delta f = 1/t_{win}$ ), ensuring constant frequency resolution regardless of the input sampling rate.
Band Splitting: The spectrogram is split into sub-bands of fixed bandwidth ( $w$ ). These sub-bands are treated as independent building blocks.
Concatenation: The model processes each sub-band individually using a Vision Transformer (ViT) encoder. The final signal representation is formed by concatenating the [CLS] embeddings of all sub-bands.
Benefit: This allows the model to utilize the full signal bandwidth. Higher sampling rates simply add more informative high-frequency sub-bands rather than distorting the signal through resampling.

B. SSL Pre-training Strategy

FISHER employs a teacher-student self-distillation framework:

Architecture: A student encoder ( $E_{stu}$ ) and a teacher encoder ( $E_{tea}$ ), where the teacher is an Exponential Moving Average (EMA) of the student. Both utilize a ViT structure.
Masking Scheme: The student processes masked sub-bands (using inverse block masking and mask cloning to generate multiple views), while the teacher processes the complete sub-band.
Loss Function: The student is trained to align its representations with the teacher at two levels:
1. Sub-band level: Aligning [CLS] tokens.
2. Patch level: Aligning patch embeddings via a student decoder ( $D_{stu}$ ).
Data Source: Contrary to the intuition that domain-specific data is required, FISHER is pre-trained on audio and music data (Audioset, Freesound, MTG-Jamendo, Music4all, totaling 17k hours). The authors argue these datasets offer greater temporal variability and similar inductive bias, which is more beneficial for pre-training than stationary industrial signals.

3. The RMIS Benchmark

To comprehensively evaluate foundation models, the authors introduce the RMIS (Representation of M5 Industrial Signals) benchmark.

Composition: It comprises 19 sub-datasets across four modalities (Sound, Vibration, Voltage, Current).
Tasks: It covers Anomaly Detection (6 datasets, e.g., DCASE challenges) and Fault Diagnosis (13 datasets, e.g., bearing, gear, and motor faults).
Evaluation Protocol: All models are evaluated using k-Nearest Neighbor (KNN) inference without fine-tuning to test zero-shot generalization. For datasets without official splits, a "sealed" train-test split is used to prevent information leakage from highly stationary signal segments.

4. Experimental Results

FISHER was evaluated against 24 state-of-the-art (SOTA) encoders, including speech, audio, music, Large Audio Language Models (LALM), and time-series encoders.

Performance: FISHER achieved a 62.23% overall RMIS score, surpassing the best baseline (ECHO) by at least 1.56% and outperforming all other baselines by a significant margin.
Efficiency: FISHER achieves these results with significantly smaller model sizes (up to 16x smaller than common 90M parameter encoders). The "small" variant (22M parameters) outperforms models with up to 2B parameters.
Versatility: The model demonstrated strong performance across both anomaly detection and fault diagnosis tasks, particularly excelling in high-sampling-rate fault diagnosis scenarios where baselines typically lose information during resampling.
Scaling: The performance gain is shown to scale with model size and the number of sub-bands utilized.

5. Key Findings and Significance

The paper draws two critical conclusions based on their experiments:

Seamless Adaptation to Sampling Rates: The ability to handle variable sampling rates natively (via sub-band modeling) without resampling is the key driver of generalization. High-frequency components are crucial for industrial signal analysis, and resampling often dilutes this information.
Data Curation for Pre-training: Audio and music data provide superior temporal variability for pre-training foundation models compared to stationary industrial signals. An industrial signal foundation model does not necessarily require training on industrial signals to achieve SOTA performance; in fact, adding industrial signals to the pre-training mix of audio/music data can sometimes degrade performance due to data quality issues.

Significance:
The authors claim FISHER represents a breakthrough in industrial signal analysis by demonstrating that a foundation model can generalize across modalities, machines, and tasks without fine-tuning. By solving the multi-sampling-rate problem and leveraging high-quality external data for pre-training, FISHER offers a scalable, efficient, and versatile solution that reduces the development burden of training specialized models for every unique industrial scenario. The work is open-sourced to benefit the community.