💻 computer science

Revisiting RIAWELC: Radiograph-Level Auditing Reveals Data Leakage and Class-Selective Domain Shift in Weld Defect Classification

This paper exposes severe data leakage and class-selective domain shift in the RIAWELC weld defect dataset caused by byte-identical duplicates and acquisition artifacts, demonstrating that radiograph-level auditing is essential to prevent deep learning models from mistaking memorization and artifacts for genuine generalization.

Original authors: beny beny, hendry hendry, ivanna k. timotius, budhi kristianto

Published 2026-10-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: beny beny, hendry hendry, ivanna k. timotius, budhi kristianto

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the heavy industries that build our bridges, ships, and pipelines, the safety of a structure often depends on a single weld. If a tiny crack or a hidden pocket of air forms inside that joint, it can lead to catastrophic failure. To find these hidden flaws without cutting the metal apart, inspectors use radiography, a technique similar to a medical X-ray that passes radiation through the weld to create an image of its interior. For decades, humans have studied these images, but the process is slow and relies on the inspector's experience. In recent years, scientists have turned to artificial intelligence to automate this task, training computer programs to recognize the visual signatures of cracks, gaps, and bubbles. The hope is that machines can inspect these critical joints faster and more consistently than people, ensuring that the infrastructure we rely on remains safe.

However, a new study suggests that the way these artificial intelligence models have been tested may have given a false sense of security. Researchers have discovered that a popular dataset used to train these systems contained a hidden flaw: the same source images were accidentally used for both teaching the computer and testing its knowledge. It is like giving a student a practice exam and then using the exact same questions for the final test; the student would score perfectly, not because they learned the material, but because they simply memorized the answers. When the researchers corrected this mistake by ensuring that no single source image appeared in both the training and testing groups, the performance of the artificial intelligence dropped significantly. This revealed that the high scores reported in previous studies were measuring memorization rather than true understanding.

The researchers, led by a team from Satya Wacana Christian University and Universitas Dinamika Bangsa, went a step further than just fixing the data. They audited the corrected dataset to see if there were other hidden problems that standard testing might miss. They found that even after removing the duplicate images, the artificial intelligence struggled with a specific, unusual image file that looked different from the rest. This file, named "bam5," came from a different source or was captured under different conditions than the thousands of other images in the set. While the computer could easily identify defects in the standard images, it failed miserably when trying to spot cracks in this specific, unusual file.

The investigation revealed that the problem was not with the computer's ability to learn, but with the nature of the image itself. The "bam5" file was noticeably blurrier than the others, likely due to a difference in how the X-ray was taken or processed. This blur did not affect all types of defects equally. The artificial intelligence remained perfect at finding large gaps and round bubbles, even in the blurry image. However, its ability to spot thin, hairline cracks collapsed, dropping to a success rate of only 38 percent. The study showed that the blur specifically hid the fine, delicate lines that define a crack, while leaving the larger, chunkier defects visible. This is a crucial distinction because cracks are often the most dangerous type of defect, and missing them could have serious consequences.

To prove that this was a real problem and not just a fluke, the researchers tested thirteen different types of artificial intelligence models, ranging from simple networks to complex modern systems. Every single one of them showed the same pattern: they performed well on the standard images but failed to detect cracks in the blurry "bam5" image. The researchers used a statistical tool to confirm that the blurry image was indeed different from the rest of the group, identifying the lack of sharpness as the primary factor that confused the computers. They found that the difference was so distinct that a simple statistical program could tell the blurry image apart from the standard ones with nearly perfect accuracy, almost every time.

This discovery highlights a critical gap in how industrial safety systems are currently evaluated. The study demonstrates that simply testing an artificial intelligence on a mix of images is not enough to guarantee it will work in the real world. If the testing data includes images that look slightly different from the ones the computer was trained on—perhaps because they were taken with a different machine or at a different time—the system might fail silently on the most dangerous defects. The researchers argue that future tests must be designed to check for these subtle differences, ensuring that the artificial intelligence can handle the messy reality of industrial inspection, not just the clean, uniform images found in a perfect dataset. By catching these hidden weaknesses before they are deployed, engineers can build systems that truly keep our infrastructure safe.

Problem Statement

The paper addresses two critical flaws in the evaluation of deep learning models for radiographic weld defect classification using the public RIAWELC dataset:

Data Leakage via Patch-Level Splitting: The original dataset splits data at the patch level rather than the source radiograph level. Since multiple patches are cropped from the same source radiograph, random splitting often places patches from the same source image into both training and testing sets. This allows models to "memorize" specific source characteristics (e.g., sensor noise, weld geometry) rather than learning generalizable defect features, leading to inflated accuracy metrics.
Hidden Domain Shift: Even after correcting for data leakage, patch-level evaluation can mask acquisition-driven domain shifts. The authors identified a specific radiograph, bam5, which exhibited anomalous naming and visual characteristics (blur). Under standard aggregate metrics, the impact of this specific domain shift was obscured, yet it caused severe, class-selective performance degradation that went undetected by previous studies.

Methodology

The authors propose a comprehensive radiograph-level auditing protocol and apply it to the RIAWELC dataset through the following workflow:

Dataset Reconstruction (RIAWELC-RG):
- Leakage Detection: The authors performed MD5 hashing on the original dataset, confirming that all 2,443 files in the original test set were byte-identical duplicates of training files.
- Radiograph-Level Split: They constructed a new dataset, RIAWELC-RG, by grouping patches by their source radiograph ID. All patches from a single source radiograph were assigned exclusively to either the training, validation, or testing set, ensuring zero overlap between subsets.
- Deduplication: The resulting dataset contains 21,964 unique patches from 29 distinct source radiographs.
Auditing Protocol:
To investigate the anomalous bam5 radiograph, the authors applied a four-component diagnostic pipeline:
- Per-Radiograph Performance ( $P_{perf}$ ): Analyzing classification accuracy for each source radiograph individually.
- Embedding Distance ( $P_{dist}$ ): Calculating the distance of bam5 patches from the centroid of the RRT-domain (the standard radiographs) in the feature space of a fully fine-tuned ConvNeXtV2-T model.
- Statistical Profiling ( $P_{stat}$ ): Computing intensity and texture statistics (e.g., Laplacian sharpness variance) to identify acquisition artifacts.
- Domain Classification ( $P_{clf}$ ): Training a Random Forest classifier to distinguish bam5 patches from RRT-domain patches using statistical features.
Generalization and Validation:
- Cross-Architecture Evaluation: The study evaluated 13 different architectures (including CNNs like ConvNeXtV2, MobileNet, and Transformers like ViT, DeiT, and XCiT) to ensure the findings were not architecture-specific.
- 5-Fold Cross-Validation: A radiograph-level 5-fold cross-validation was conducted to verify that the observed performance gaps were not artifacts of a single data partition.

Key Results

1. Confirmation of Data Leakage
The MD5 audit confirmed that the original RIAWELC test set was 100% duplicated from the training set. Consequently, previously reported high accuracies (e.g., 93.33%) measured memorization rather than generalization.

2. Class-Selective Degradation on bam5
When evaluated on the leakage-free RIAWELC-RG split, the inclusion of bam5 in the test set caused a significant drop in overall accuracy (e.g., from 0.9663 to 0.9277 for ConvNeXtV2-T). Crucially, this degradation was class-selective:

Crack: Accuracy dropped to 0.380.
Lack of Penetration (LoP) & Porosity: Accuracy remained at 1.000.
No Defect: High accuracy maintained.

3. Identification of the Shift Mechanism

Visual/Statistical Evidence: bam5 patches appeared uniformly blurrier than RRT patches. Statistical analysis revealed that bam5 had significantly lower Laplacian sharpness variance compared to the RRT domain.
Domain Separability: A Random Forest domain classifier trained on statistical features could distinguish bam5 from RRT patches with near-perfect accuracy (ROC-AUC = 0.9995).
Feature Importance: Laplacian sharpness variance was the dominant feature (contributing ~50.8% of importance) in separating the domains.
Mechanism: The authors conclude that bam5 represents a domain shift caused by acquisition blur. This blur disproportionately affects "crack" defects, which rely on fine, high-frequency linear structures, while coarser defects like LoP and porosity remain recognizable.

4. Generalization Across Architectures
The performance gap caused by bam5 was consistent across all 13 evaluated architectures (CNNs, hybrids, and Transformers). A paired t-test across the 13 models confirmed the gap was statistically significant ( $p < 10^{-6}$ ), proving the effect is a property of the data domain, not a specific model limitation.

5. Cross-Validation Findings
In 5-fold cross-validation, the "crack-class collapse" pattern reappeared on other radiographs (e.g., RRT-107R, RRT-29R), suggesting that while bam5 is a clear case study, class-selective domain shifts due to acquisition variations are a broader vulnerability in patch-based datasets.

Significance and Claims

The paper claims that radiograph-level evaluation must be paired with explicit domain-shift auditing to avoid mistaking acquisition artifacts for genuine generalization failure.

Beyond Leakage Correction: Simply removing duplicate patches (deduplication) is insufficient. Researchers must audit for "hard-domain" radiographs that introduce shifts invisible to aggregate metrics.
Class-Selectivity: Domain shifts in industrial radiography may not affect all classes equally; they can be highly specific to defect types that rely on fine spatial details (like cracks).
Protocol Replicability: The proposed auditing protocol (combining per-radiograph performance, embedding distance, statistical profiling, and domain classification) is not specific to RIAWELC or weld inspection. It can be applied to any patch-based industrial NDT dataset where source radiograph identifiers are recoverable.
Practical Implication: Future datasets must provide source-radiograph identifiers and acquisition metadata. Benchmark reports should include per-radiograph performance to reveal errors concentrated in specific domains, rather than relying solely on aggregate accuracy.

The authors emphasize that without these rigorous auditing steps, classifiers trained on datasets like RIAWELC risk failing in safety-critical industrial deployments where acquisition conditions may vary.

Problem Statement

Methodology

Key Results

Significance and Claims

More like this