💻 computer science

RobustDefect-LLM: Explainable and Robustness-Aware Industrial Surface Defect Classification with Decision Support and AI-Assisted Reporting

This paper introduces RobustDefect-LLM, an integrated industrial surface defect inspection framework that combines high-accuracy MobileNetV3-Large classification with confidence-based human review routing and AI-assisted reporting to deliver a robust, explainable, and traceable quality-control workflow, while explicitly acknowledging its performance sensitivity to severe image degradation.

Original authors: Nazlıcan Düşünmez, Halûk Gümüşkaya

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nazlıcan Düşünmez, Halûk Gümüşkaya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where factories are like giant, busy kitchens, and the chefs (robots) are trying to cook perfect meals. But sometimes, the food gets a little burnt, or a smudge appears on the plate. In the real world of manufacturing, these "smudges" are called surface defects, and they can ruin a product's reliability. For a long time, humans had to stare at every single item, squinting to find these tiny flaws. It was tiring, and humans get tired, which means mistakes happen. Enter Artificial Intelligence (AI), specifically a branch called "Deep Learning." Think of Deep Learning like a super-powered student who doesn't just memorize pictures of defects but actually learns what a scratch or a dent looks like by studying thousands of examples. However, there's a catch: these AI students are often "black boxes." They give you an answer ("This is a scratch!"), but they can't explain why they think that, or how sure they are. In a factory, you can't just trust a black box; you need to know if the AI is confident or if it's just guessing. That's where the new idea of "Explainable AI" comes in—it's like asking the student to point to exactly where they saw the scratch and say, "I'm 99% sure because I see this line here."

This paper introduces a new system called RobustDefect-LLM, which is like a complete, high-tech quality-control team for factories. Instead of just a single robot looking at a product, this system combines a sharp-eyed AI classifier, a "magnifying glass" that shows humans exactly where the AI is looking, a smart referee that decides when to trust the AI and when to call a human expert, and even a robot writer that drafts the final report. The researchers tested this system on a dataset of 1,799 images of steel surfaces with six different types of defects (like scratches, patches, and pits). They didn't just build one model; they tested four different types of AI "brains" (ResNet50, EfficientNet-B0, DenseNet121, and MobileNetV3-Large) to see which one was the best at spotting these flaws.

The results were quite impressive for the "best" candidate, a model called MobileNetV3-Large. On a clean, perfect set of test images, this model got it right 99.26% of the time. That's incredibly accurate! It was so good that it was statistically tied with another strong contender, DenseNet121, meaning there wasn't a huge difference between them, but MobileNetV3-Large was chosen because it was faster and lighter, making it easier to run on factory computers. The system also proved it could "see" what it was looking at. Using a technique called Grad-CAM, the system generated heatmaps (like a thermal camera view) that highlighted the exact spots on the steel where the defect was, giving human operators visual proof of the AI's decision.

But here is the most important part of the story: the system knows when it doesn't know. The researchers programmed a "smart referee" rule. If the AI isn't very confident (specifically, if its confidence score is below 0.90 or if it's confused between two similar defects), the system automatically stops and says, "HUMAN REVIEW." In their tests, this rule was very strict. It sent 87.78% of the images to a human to check, only letting 12.22% of the decisions happen automatically. Why so strict? Because the system is designed to be safe. When they looked at the two mistakes the AI made, both of them were caught by this "Human Review" rule! This means the system successfully prevented any wrong automatic decisions from happening in the test.

However, the paper is very honest about its limits. When the researchers simulated bad conditions—like making the images blurry, dark, or noisy—the AI's performance dropped significantly. At mild levels of "badness," accuracy fell to 87.78%, and at severe levels, it dropped below 40%. This tells us that while the system is great in a controlled lab, it's not yet ready to handle the messy, unpredictable reality of a real factory floor without more training. The system also doesn't know what a "perfect" piece of steel looks like; it only knows what the six specific defects look like, so it can't tell you if a piece is perfect, only if it has a specific flaw.

To tie it all together, the team built a full mobile app that lets factory workers take a picture of a steel sheet, see the AI's guess, look at the "heat map" to see where the AI is looking, and get a written report generated by a Large Language Model (LLM). This report is "controlled," meaning the AI can't just make things up; it has to stick to the facts the computer found. The whole process, from taking the picture to getting a report, is designed to be a partnership between the machine and the human, ensuring that the final decision is always safe, traceable, and explained. The authors suggest that while this is a huge step forward for making AI trustworthy in factories, it still needs more testing in real-world conditions before it can run a production line entirely on its own.

Technical Summary of RobustDefect-LLM

Problem Statement
Industrial surface inspection is critical for manufacturing reliability but remains vulnerable to human fatigue, subjectivity, and environmental variations. While deep learning has improved defect classification accuracy, many high-performing Convolutional Neural Networks (CNNs) operate as "black boxes," lacking the explainability, uncertainty awareness, and traceability required for practical deployment. Furthermore, existing solutions often focus solely on benchmark accuracy without integrating prediction, visual explanation, confidence-aware decision support, and structured reporting into a unified, deployable workflow. The paper addresses the gap in creating a trustworthy, end-to-end industrial inspection system that balances high predictive performance with operator-facing transparency and controlled AI-assisted decision-making.

Methodology
The proposed framework, RobustDefect-LLM, integrates four core functions: perception, explainability, decision support, and reporting.

Perception & Model Evaluation: The system evaluates four transfer-learning-based CNN architectures (ResNet50, EfficientNet-B0, DenseNet121, and MobileNetV3-Large) on the NEU-DET dataset, which contains 1,799 grayscale images across six steel defect classes (crazing, inclusion, patches, pitted surface, rolled-in scale, and scratches). All models were fine-tuned using identical train/validation/test splits (70%/15%/15%) and ImageNet normalization.
Explainability: Grad-CAM is employed to generate heatmaps highlighting image regions influencing predictions, providing visual evidence to operators without claiming causal proof.
Confidence-Aware Decision Support: A rule-based engine processes the predicted class, maximum confidence score ( $c$ ), and top-2 confidence margin ( $\Delta$ ). A conservative policy routes predictions to HUMAN REVIEW if $c < 0.90$ or $\Delta < 0.10$ . Only predictions meeting both thresholds proceed to severity-based automated actions (ACCEPT, REWORK, or REJECT). Defect severity is not predicted by the model but assigned via a configurable mapping based on the defect class.
Controlled AI-Assisted Reporting: A Large Language Model (Llama-3.1-8B-Instant via Groq API) generates natural language reports from structured inspection data. To ensure auditability, the system validates generated reports against the structured input; if inconsistencies are detected, a deterministic rule-based fallback report is used.
System Architecture: The framework is implemented as a full-stack prototype with a FastAPI backend, MongoDB for persistent storage, and a React Native mobile application for operator interaction.

Key Results

Classification Performance: MobileNetV3-Large achieved the highest test accuracy (99.26%) and macro F1-score (0.9926) on the held-out test set, with a 95% bootstrap confidence interval of [0.9815, 1.0000]. An exact McNemar test found no statistically significant difference between MobileNetV3-Large and DenseNet121 (p = 1.000).
Efficiency: MobileNetV3-Large demonstrated superior computational efficiency, requiring only 0.467 GFLOPs and achieving a CPU inference throughput of 16.66 FPS (0.060 s/image).
Robustness: Under synthetic image degradation (Gaussian blur, brightness variation, noise), accuracy dropped to 87.78% at mild intensity and below 40% at severe intensities, indicating sensitivity to significant image quality deterioration.
Decision Support: The conservative confidence thresholds resulted in an automatic coverage of 12.22% and a HUMAN REVIEW rate of 87.78%. Among the 33 automatically eligible cases, the system achieved 100% observed selective accuracy (95% CI: 89.43%–100.00%), successfully routing both observed classification errors to human review.
Reporting: Under nominal conditions, 100% of LLM-generated reports passed deterministic consistency checks with a mean latency of 1.66 s.

Significance and Claims
The paper asserts that its primary contribution is not the proposal of a novel classifier, but the integration of multiple complementary components into a unified, explainable, and deployable industrial workflow. The significance lies in:

System-Level Integration: Combining efficient transfer learning, Grad-CAM visualization, confidence-aware routing, deterministic decision logic, and controlled LLM reporting into a single mobile-enabled system.
Trustworthiness over Pure Accuracy: Prioritizing a workflow where uncertain predictions are explicitly routed to human operators, ensuring that high benchmark accuracy does not come at the cost of operational safety or auditability.
Practical Feasibility: Demonstrating that such a framework can be built using standard open-source tools (FastAPI, React Native, PyTorch) and public datasets, providing a blueprint for traceable and explainable AI in quality control.

Limitations and Future Directions
The authors maintain a modest stance regarding the current results, noting that:

The dataset (NEU-DET) is relatively small and lacks a "no-defect" class, limiting generalization to defect-free production.
Robustness is currently evaluated only under controlled synthetic degradation, not real-world factory conditions.
The current prototype is a local demonstration not yet optimized for large-scale concurrent deployment or edge integration.
Future work must focus on expanding datasets, adding defect-free classes, calibrating confidence thresholds for specific industrial risk profiles, and validating the system under real production conditions.

Technical Summary of RobustDefect-LLM

More like this