💻 computer science

Co-identification of Vibration, Structural Looseness, and Surface Damage in Industrial Equipment Operation Videos

This paper presents a joint discrimination algorithm that synergistically identifies vibration states, structural looseness, and surface defects in industrial equipment videos by integrating spatiotemporal features from a Video Swin Transformer, micro-displacement features from RAFT optical flow, and multi-task appearance analysis, thereby significantly reducing missed detections in composite anomaly scenarios compared to single-task models.

Original authors: Yukun Du, Zhihuang Chen, Quanle Liu, Yuang Dong

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yukun Du, Zhihuang Chen, Quanle Liu, Yuang Dong

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery inside a giant, noisy factory. Usually, when machines break, they don't just snap in half; they get sick slowly. First, they might start to shiver a tiny bit. Then, a screw might wiggle loose. Finally, a crack might appear on the metal skin. The tricky part is that these symptoms often happen at the same time, and they are very hard to spot if you only look at one single snapshot. It's like trying to tell if a car engine is failing just by looking at a single photo of the hood; you might miss the fact that the engine is shaking violently or that a bolt is slowly unscrewing. This is where computer vision comes in—a field where we teach computers to "see" and understand the world through cameras. But most computers are great at spotting static things (like a crack in a wall) or moving things (like a car driving by), but they struggle when a machine is doing both: vibrating while also getting damaged.

This is exactly the puzzle a team of researchers set out to solve. They wanted to build a super-smart system that could watch videos of industrial equipment—like pump pipes, valves, and metal shells—and spot three things happening all at once: weird vibrations, loose parts, and surface damage like cracks or rust. Instead of just taking a photo and guessing, their system acts like a detective who watches the whole movie, paying attention to how the machine moves, how much it wiggles, and what its surface looks like, all in one go.

The Detective's New Toolkit

The researchers created a system that treats industrial inspection videos like a complex story rather than a series of disconnected pictures. They realized that if you only look at a single frame, you might miss the subtle clues. For example, a pipe might look fine in one photo, but if you watch a 32-frame clip (which is just over a second of video at 25 frames per second), you might see it shivering slightly or a bracket slowly drifting away from its spot.

To catch these clues, the team built a "three-headed" brain for their computer:

The Rhythm Watcher (Video Swin Transformer): Imagine a dancer trying to learn a complex routine. This part of the system watches the whole dance sequence (the video frames) to understand the rhythm of the machine. It looks for patterns in how the machine moves over time, spotting if the vibration is a steady hum, a periodic wobble, or a chaotic shake. It's like listening to a song to tell if the singer is off-key, rather than just looking at their face in one photo.
The Wiggle Counter (RAFT Optical Flow): This is the system's way of measuring tiny movements that the human eye might miss. It tracks how every single pixel moves from one frame to the next. If a bolt is supposed to be still but it's drifting 0.5 millimeters, this module catches it. To make sure it's not just the camera shaking, the system ignores the background (like the wall or the floor) and focuses only on the machine parts. It's like trying to feel a tiny vibration in a car while ignoring the fact that the road is bumpy.
The Damage Spotter (Defect Localization): This part looks for the "scars" on the machine: cracks, rust, and corrosion. It's trained to find these tiny blemishes, even if they are smaller than a fingernail (less than 20 pixels wide).

Putting It All Together: The "Co-Identification" Magic

The real magic happens when these three heads talk to each other. In the past, engineers might have used one tool to check for vibrations and a different tool to check for cracks. But the researchers found that this often led to mistakes. If a machine had a tiny crack and a slight vibration, a single-tool system might think, "Oh, the vibration is just normal noise," and miss the crack. Or it might see the crack but think, "The machine isn't shaking, so it's fine," missing the fact that the crack is making the structure loose.

The new system combines everything. It asks: "Is the machine vibrating? Is it loose? Is there a crack?" and then looks for connections between them. They found that when a machine has a slight vibration and an early crack, the risk of it breaking is much higher than if you just looked at one symptom alone.

What They Found

The team tested their system on a huge library of videos—about 36 hours of footage from real pump stations and factories, which turned into roughly 72,000 training images and 11,000 test images. They labeled everything: normal operation, slight vibration, periodic resonance, abnormal shaking, stable structures, suspected looseness, cracks, and corrosion.

The results were impressive. The system got the vibration states right 92.1% of the time. When it came to finding cracks and corrosion, it was correct 88.9% of the time (measured by a standard score called mAP@0.5). Perhaps most importantly, when they compared their "all-in-one" system to the old "single-task" systems, they found that the new system missed about 16% fewer dangerous situations.

For example, if a pipe had a tiny crack and was shaking slightly, the old systems often missed it because the vibration wasn't "loud" enough to trigger an alarm, and the crack was too small to see clearly. The new system, however, connected the dots: "Hey, that tiny vibration combined with that tiny crack means this part is getting loose!"

They also created a "looseness score" to predict how likely a part is to fall apart. This score matched up with human experts' reviews 79% of the time (a correlation of 0.79). They discovered that the most important clues for looseness weren't just the cracks themselves, but the peak movement of the support areas and how much the local parts were drifting apart.

Why This Matters

This study suggests that we can't just look at machines in pieces; we have to watch them as a whole, dynamic system. By combining the rhythm of movement, the tiny drifts of parts, and the visual scars of damage, we can catch problems much earlier. While the system isn't perfect—it still struggles with very dark lighting, heavy reflections, or when parts are hidden behind other objects—it represents a significant step forward. It shows that by teaching computers to look at the "story" of a machine's operation rather than just a snapshot, we can keep industrial equipment safer and prevent small issues from turning into big disasters.

Technical Summary: Co-identification of Vibration, Structural Looseness, and Surface Damage in Industrial Equipment Operation Videos

Problem Statement
Industrial equipment deterioration is a continuous evolutionary process where micro-vibrations, connection stiffness degradation, and surface damage propagation often occur simultaneously. Traditional inspection methods typically address vibration status, crack detection, corrosion identification, and structural loosening assessment as separate tasks. This siloed approach fails to capture the correlations between motion abnormalities and visual defects, leading to misleading risk assessments. Specifically, when minor vibrations, incipient cracks, and localized loosening occur concurrently, single-frame image analysis or isolated vibration classification is insufficient to detect structural risks or modify motion status accurately. The challenge lies in the need to simultaneously extract optical-flow amplitude, displacement variance, texture gradients, and target-area stability features from continuous video sequences to enable comprehensive composite anomaly detection.

Methodology
The authors propose a multi-task learning framework designed to jointly discriminate motion states, structural responses, and surface damage. The architecture integrates three primary modules:

Spatio-Temporal Feature Extraction (Video Swin Transformer):
This module processes continuous frame sequences (32-frame windows) to address low vibration amplitudes, small defect areas, and strong background interference. Using a 3D spatio-temporal block segmentation method with $2 \times 4 \times 4$ patches, it captures movement rhythms (5–40 Hz) and spatial texture variations across four hierarchical encoding levels. It generates multi-scale features (96 to 768 dimensions) to model operational status changes.
Micro-Displacement Estimation (RAFT Optical Flow):
To quantify subtle structural movements, the system employs the RAFT (Recurrent All-Pairs Field Transforms) module. It constructs a global 4D correlation volume to match features between adjacent frames at 1/8 resolution, refined through 12 GRU iterations. A critical component is global motion compensation, where static background points (e.g., walls, fixed floors) are tracked to estimate and subtract camera shake, isolating local structural movement. The module outputs horizontal/vertical displacement and amplitude maps, recording oscillation cycles and edge drift traces around supports and connections.
Surface Defect Localization and Structural Looseness Scoring:
A parallel structure consisting of a detection head and a regression head processes the fused features:
- Detection Head: Outputs class probabilities, confidence scores, and bounding boxes for cracks, corrosion, and edge spalling. It utilizes edge gradient weights to handle thin cracks (<10px width).
- Regression Head: Calculates a continuous structural looseness score ( $S_l$ ) by combining RAFT optical flow statistics (amplitude, variance, direction dispersion) with spatial-temporal coding and defect features (distance to joint boundary, defect confidence). The score is defined by a weighted sum of these normalized features.

Joint Classification and Loss Function
The model fuses spatiotemporal features ( $F_t$ ), optical flow statistical features ( $F_m$ ), and defect region features ( $F_d$ ) into a shared 256-dimensional space using gated attention weights. The system outputs four vibration states, two structural states, defect bounding boxes, and a looseness score. To balance gradients across tasks, an uncertainty-weighted loss function is employed, which includes classification loss, bounding box overlap (CIoU), confidence loss, and a smooth L1 loss for the looseness score. A cross-task consistency constraint is applied to composite anomaly clips to prevent the underestimation of risks when weak vibrations and early surface defects coexist.

Key Contributions

Integrated Framework: The study introduces a novel approach that synergistically detects motion anomalies and surface damage, moving beyond single-task isolation to a joint discrimination model.
Micro-Displacement Quantification: By integrating RAFT optical flow with global motion compensation, the system effectively separates camera jitter from genuine structural micro-displacements (0.5–3.0 mm range), providing quantitative inputs for looseness scoring.
Composite Anomaly Handling: The model specifically addresses the "composite degradation" scenario where slight vibrations accompany early-stage cracks or corrosion, a condition where traditional single-task models often fail.

Experimental Results
The system was evaluated on a dataset comprising approximately 36 hours of industrial inspection videos (pump station pipelines, valve connections, brackets, metal shells), resulting in ~72,000 training and ~11,000 test images.

Vibration Recognition: Achieved an accuracy of 92.1% across four states (normal, slight vibration, periodic resonance, abnormal shaking).
Defect Detection: Attained an mAP@0.5 of 88.9% for crack and corrosion detection.
Looseness Scoring: The structural looseness score showed a Spearman correlation coefficient of 0.79 with manual review results.
Composite Anomaly Performance: Compared to single-task models, the joint model reduced the missed detection rate for composite anomalies (slight vibration + early cracks) by approximately 16% (from 27.5% to 11.5%).
Feature Importance: Ablation studies confirmed that peak optical flow amplitude in support areas, local displacement variance, and texture changes around defects are highly correlated with increased structural loosening risk.

Significance and Claims
The paper claims that the proposed method enhances the integrity of equipment state recognition, particularly in composite anomaly scenarios where single visual inspection or isolated vibration analysis is inadequate. By synchronously labeling and learning vibration states, structural looseness, and surface defects, the model provides a more comprehensive basis for early risk detection and maintenance decision-making. The authors note that while the method demonstrates strong consistency and reduced false negatives, it remains subject to constraints such as lighting conditions, occlusion, and strong reflections. Future work is suggested to incorporate multi-source sensor data and online calibration to further improve engineering generalization.

The Detective's New Toolkit

Putting It All Together: The "Co-Identification" Magic

What They Found

Why This Matters

Technical Summary: Co-identification of Vibration, Structural Looseness, and Surface Damage in Industrial Equipment Operation Videos

More like this