⚡ electrical engineering

Thermal-history and morphology-conditioned multimodal fusion for robotic MAG weld-quality classification

The authors propose HMCF-Net, a multimodal fusion network that enhances robotic MAG weld-quality classification accuracy by jointly conditioning visual, electrical, and acoustic features on calculated thermal history and explicit pool geometry descriptors, achieving 94.72% accuracy without requiring direct temperature measurements or extended observation windows.

Original authors: Bin Zhong, Kanghong Zhu, Runquan Xiao, Yiyang Zhao, Yingcai Wan, Huabin Chen

Published 2026-10-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bin Zhong, Kanghong Zhu, Runquan Xiao, Yiyang Zhao, Yingcai Wan, Huabin Chen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of heavy industry, robotic welding is a cornerstone of modern manufacturing, joining steel plates to build everything from ships to skyscrapers. For these machines to work reliably, they must know instantly if a weld is forming correctly or if a defect is taking shape. A perfect weld requires the molten metal to flow just right, fusing the pieces together without burning a hole through them or leaving a weak, incomplete bond. Traditionally, robots rely on cameras and microphones to "see" and "hear" the welding arc, but these sensors only capture a split second of the process. They miss the story of what happened just before that moment. A weld is not just a snapshot; it is the result of heat building up over time, and without knowing the history of that heat, a robot might misinterpret a sudden change in the metal's appearance as a defect when it is actually a normal part of the process.

Researchers at Shanghai Jiao Tong University and Xiamen CCRE Group have developed a new way to help robots understand this full story. They created a system that combines the robot's immediate sensory data with a calculated history of the heat input and a precise measurement of the molten metal's shape. By feeding this combined information into a smart computer model, they achieved a significant leap in accuracy when classifying the quality of robotic welds. The system does not need expensive temperature sensors or longer observation times; instead, it uses the power data already available to the robot to reconstruct the thermal history and pairs it with a clear geometric description of the molten pool.

The challenge the researchers tackled was that a robot's camera sees only the last few hundred milliseconds of a weld. In that brief window, the molten metal might look strange because of a change in the arc, but the robot cannot tell if that change is due to a new problem or simply the lingering effect of heat from seconds ago. To solve this, the team built a digital model that calculates how heat spreads through the metal plates based on the electrical power the robot has been using since the very start of the weld. This calculation acts as a memory, telling the system how much heat has accumulated in the metal before the current moment. Simultaneously, the system uses a computer vision tool to draw a precise outline of the molten pool in every camera frame, measuring its size, position, and how spread out the metal is.

These two pieces of information—the calculated heat history and the explicit shape of the pool—are then fed into a neural network alongside the raw video, sound, and electrical signals. The network uses this extra context to update its understanding of the current situation. It is like giving a driver not just a view of the road ahead, but also a map of the terrain they just drove over and a clear measurement of the car's current position. This allows the system to distinguish between a temporary fluctuation and a genuine defect. The researchers tested this approach on thousands of short time windows from real welding jobs, covering four different outcomes: a perfect weld, a weld that didn't penetrate deep enough, a weld that burned a hole through the metal, and a weld that drifted off course.

The results showed that adding this context made a measurable difference. The new system correctly identified the quality of the weld in nearly 95 percent of the test cases, outperforming other advanced methods that relied only on the immediate sensory data. Specifically, the combination of the heat history and the shape description helped the system recognize normal welding and incomplete penetration much more reliably. While the system still found some burn-through defects harder to identify than others, the overall improvement was clear. The study demonstrated that by explicitly telling the computer about the heat that built up before the camera started watching and the exact geometry of the pool it is seeing, the robot can make far better decisions about the quality of its work.

This approach offers a practical path forward for industrial automation. It proves that a robot does not need to be equipped with more complex or expensive hardware to see better; it simply needs to be taught to look at the data it already has in a smarter way. By calculating the thermal past and measuring the geometric present, the system bridges the gap between a fleeting observation and the full reality of the welding process. The researchers found that using both the history and the shape together was more effective than using either one alone, suggesting that these two types of information complement each other perfectly. This method allows for high-speed, high-accuracy quality control without slowing down the production line or requiring direct temperature measurements that are difficult to obtain in a harsh industrial environment.

The study concludes that integrating these calculated histories and explicit shapes into the robot's decision-making process is a powerful tool for improving weld quality. The system successfully classified welds into four categories with high confidence, showing that the robot can now distinguish between a normal welding process and various types of defects with greater reliability. While there is still room to improve the detection of specific defects like burn-through, the core finding is robust: a robot that understands the thermal history and the precise shape of the molten pool is a more competent welder. This work provides a clear blueprint for how future robotic systems can be made more aware of their own actions and the physical consequences of those actions, ensuring stronger and safer metal structures.

Technical Summary: Thermal-history and morphology-conditioned multimodal fusion for robotic MAG weld-quality classification

Problem Statement
In robotic Metal Active Gas (MAG) welding, reliable in-process quality recognition requires interpreting local sensor observations within the context of the conditions under which the weld is formed. While multimodal sensing (visual, electrical, acoustic) captures local process dynamics, short observation windows (e.g., 200 ms) fail to explicitly describe the accumulated thermal response preceding the current moment. Furthermore, existing fusion models often rely solely on learned features from observed signals, neglecting explicit geometric descriptors of the molten pool or calculated thermal history derived from input power. This limitation hinders the accurate classification of complex quality states, such as distinguishing between incomplete penetration and burn-through, which depend on joint gap, groove geometry, and thermal accumulation.

Methodology
The authors propose HMCF-Net, a thermal-history and morphology-conditioned multimodal fusion network designed for four-class weld-quality classification (normal, incomplete penetration, burn-through, and weld deviation). The architecture integrates three distinct information streams:

Multimodal Observation Encoding:
- Visual: A shared MobileNetV2 backbone extracts global and local visual features from every camera frame within a 200-ms window. Local features are derived from regions of interest (ROI) defined by Segment Anything Model 3 (SAM 3) masks.
- Signal: Electrical signals (current/voltage) and arc sound are encoded via causal 1D and 2D CNNs, respectively.
- Fusion: A time-constrained cross-modal attention mechanism allows visual tokens to interact with signal tokens available up to that specific frame. The sequence is aggregated via a Gated Recurrent Unit (GRU) to form a 96-dimensional window observation vector ( $z$ ).
Nominal Thermal-History Descriptor ( $H$ ):
- Instead of relying on direct temperature measurement, the model calculates a six-dimensional descriptor based on a time-varying finite-plate analytical model driven by recorded power (current and voltage).
- Using a double-ellipsoidal heat source model, the system computes the nominal temperature rise from the arc onset to the current time.
- The six features capture peak response intensity, transverse spread, longitudinal asymmetry, backside-to-surface ratio, rate of change, and cumulative energy input.
Explicit Molten-Pool Morphology Descriptor ( $g$ ):
- An eight-dimensional descriptor is extracted directly from SAM 3 masks of the same images used for visual encoding.
- These dimensionless features quantify pool area, bounding box extent, centroid position, occupancy, and spatial dispersion (variance) within a fixed image context.
Joint Conditioning Mechanism:
- The thermal ( $H$ ) and morphology ( $g$ ) descriptors are embedded and summed to form a joint condition vector ( $c$ ).
- This condition vector updates the multimodal observation representation ( $z$ ) via a conditioned residual mapping. A nonlinear residual ( $\Delta z$ ) is computed from the joint condition and the observation state, which is then added to the original observation via a bypass connection.
- The updated representation is passed to a linear classifier for quality prediction.

Key Contributions

Novel Fusion Architecture: The paper introduces a mechanism to jointly condition multimodal observations on both calculated thermal history and explicit geometric descriptors, rather than treating them as separate inputs or relying solely on learned temporal dynamics.
Explicit vs. Learned Features: The study demonstrates the value of combining learned visual features (appearance and context) with fixed-definition geometric descriptors (extent, position, dispersion) derived from the same images.
Calculated Thermal Context: The approach incorporates a physics-driven nominal thermal history (derived from power input and plate geometry) to provide context preceding the observation window, eliminating the need for direct temperature sensors or extended camera windows.
Residual Conditioning: The use of a residual update with a bypass ensures that the original multimodal representation is preserved while being refined by the process history and geometry.

Results
The model was evaluated on a dataset of 36 robotic MAG weld records (26 training, 10 validation), comprising 5,078 validation windows.

Performance: HMCF-Net achieved 94.72% accuracy, 93.04% balanced accuracy, and 93.46% macro-F1.
Comparisons: It outperformed alternative architectures (CNN-LSTM, CNN-Transformer, Cross-Attention Fusion) and a matched model using only the observation encoder (HMCF-Net observation-only). Specifically, the complete HMCF-Net exceeded the strongest alternative model (Cross-Attention Fusion) by 2.52 percentage points in macro-F1.
Ablation Studies: Matched ablations confirmed that both descriptors contribute independently and jointly. Adding explicit morphology improved macro-F1 by 1.27 points, and adding thermal history improved it by 1.07 points over the observation-only baseline. Their joint use yielded the highest performance, improving macro-F1 by 1.69 percentage points over the model without either descriptor.
Class Performance: The model showed significant gains in recognizing normal welding and weld deviation. While burn-through remained the most challenging class (82.43% recall), the joint conditioning reduced the total number of errors compared to ablation variants.

Significance and Claims
The paper claims that HMCF-Net demonstrates that jointly conditioning multimodal observations on calculated thermal history and explicit pool geometry improves overall weld-quality classification performance. The significance lies in achieving this without requiring direct temperature measurement or extending the camera observation window beyond the standard 200 ms. The results suggest that integrating accumulated process history and explicit geometric context with short-term sensor data provides a more robust representation for quality recognition than relying on learned temporal dynamics alone. The authors note that while the approach is effective, discrimination of burn-through remains a specific target for future improvement, and the method's transferability to independent plates and broader joint conditions requires further evaluation.

More like this