Technical Summary: RobustMAD – Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

Problem Statement

Multimodal Large Language Models (MLLMs) have expanded industrial anomaly detection from rigid vision-only systems to interactive, open-ended vision-language querying. However, their deployment in real-world smart factories is hindered by prohibitive computational demands and privacy risks associated with cloud-based inference. While Multimodal Small Language Models (MSLMs), typically under 4 billion parameters, offer a viable path for on-device deployment, their readiness for safety-critical industrial inspection remains unverified.

Current research suffers from a significant gap: existing benchmarks either focus on large models unsuitable for edge devices or fail to assess robustness against realistic, diverse, and imperfect user queries. Real-world industrial settings involve domain-intensive knowledge, dynamic image quality variations (e.g., motion blur, low light), ill-posed user queries, and open-ended reporting requirements. Standard benchmarks relying on controlled, multiple-choice formats do not capture the fragility of MSLMs when faced with these practical challenges, potentially leading to catastrophic false negatives in safety-critical inspections.

Methodology: The RobustMAD Benchmark

To address this gap, the authors introduce RobustMAD, the first deployment-motivated benchmark designed to comprehensively evaluate the robustness of MSLMs in industrial anomaly inspection.

Dataset Construction

• Source Data: The benchmark utilizes 410 representative images curated from the MVTec AD and VisA anomaly detection datasets, covering 20 object types (including general objects and domain-knowledge-intensive items like PCBs and transistors).
• Question Generation: The dataset comprises 4,510 multiple-choice questions (MCQs) and 7,380 open-ended questions. These are generated using a two-stage pipeline involving GPT-5 assisted brainstorming and rigorous human expert validation (560+ person-hours).
• Robustness Categories: The benchmark evaluates models across four knowledge-based robustness categories:
  1. General Object Understanding: Identifying object names, attributes, and functions.
  2. Stand-alone Anomaly Detection: Detecting and localizing defects from single images.
  3. Pairwise Anomaly Detection: Comparing a query image against a defect-free reference to assess anomalies.
  4. Unanswerable or Ill-posed Query Detection: Evaluating the model's ability to recognize and handle queries that cannot be answered based on the provided visual evidence.
• Visual Quality Perturbations: To simulate real-world deployment conditions, the benchmark re-evaluates all questions under two specific degradations: motion blur and low-lighting, applied to half of the image set each.

Evaluation Protocol

• Models Evaluated: The study benchmarks state-of-the-art MSLMs (e.g., Qwen3-VL-4B, MiniCPM-V 4.0, InternVL3.5-4B) against moderate-scale open-source models (5–8B) and proprietary efficient models (GPT-5 Nano, Gemini 3 Flash) as references.
• Metrics:
  • MCQs: Measured by standard accuracy.
  • Open-ended Responses: Evaluated using an LLM-as-a-Judge approach (GPT-5) with a multi-dimensional, user-centered scoring scheme. The scheme assesses:
    • Technical Accuracy: Factual correctness and absence of hallucinations.
    • Comprehensiveness: Coverage of all required details (e.g., defect location, type).
    • Relevance: Alignment with the query intent.
    • Style and Clarity: Readability and formatting.
  • Validation: The LLM judge's reliability was validated against independent human experts, achieving a 93.8% agreement rate.

Key Results

The evaluation reveals that while top-performing MSLMs show promising baseline competence, they exhibit critical robustness gaps that standard benchmarks obscure.

Performance Highlights

• Surprising Competence: Contrary to the assumption that larger models are strictly superior, top-tier MSLMs (specifically Qwen3-VL-4B-Instruct and MiniCPM-V 4.5) outperform larger models like Phi-4-Multimodal-Instruct (6B) and even the proprietary GPT-5 Nano in several categories. This suggests that architectural optimization and training data quality can outweigh sheer parameter scale for specific industrial tasks.
• Robustness Gaps: Despite high MCQ accuracy in object understanding (often >90%), performance drops significantly in anomaly detection and unanswerable query handling. In open-ended evaluations, even the best models struggle to achieve an average overall score above 3.0 (on a 5-point scale) for anomaly detection and unanswerable queries.
• Visual Sensitivity: Models show a measurable performance drop (0.8–2.4 percentage points in accuracy, and larger drops in open-ended scores) when images are degraded by motion blur or low lighting, exposing vulnerabilities in real-world deployment.

Identified Failure Modes

The paper identifies three recurring failure modes that pose operational risks:

1. Fragile Multimodal Grounding: Models fail to reason beyond macro visual attributes when faced with fine-grained distinctions or degraded conditions. Language priors often override visual evidence, leading to misclassifications (e.g., identifying a T-nut as a hexagonal flange) or hallucinations (e.g., interpreting an oil stain as a marine organism).
2. Insufficiently Comprehensive Responses: Even when technically correct, models often provide vague or underspecified answers. They frequently omit precise defect locations, fail to list all defects in multi-defect scenarios, or lack the explanatory detail required by industrial standards.
3. Weak Logical Grounding on Unanswerable Queries: Models frequently fail to recognize ill-posed queries. Instead of indicating missing evidence, they confidently generate hallucinated answers (e.g., inventing a voltage rating for a capsule that has none), a critical failure in safety-critical contexts.

Significance and Contributions

The paper claims the following contributions and significance:

1. First Comprehensive Robustness Evaluation: It presents the first systematic evaluation of MSLM robustness specifically for multimodal industrial anomaly inspection, a domain where on-device deployment is critical but previously unexplored.
2. RobustMAD Benchmark: It introduces a rigorous benchmark that captures core practical challenges—domain-intensive reasoning, non-standardized queries, and visual quality variations—revealing failure modes that standard benchmarks cannot capture.
3. Novel Evaluation Framework: It designs a multi-dimensional, user-centered scoring scheme for open-ended responses, enabling standardized quantitative assessment of technical accuracy, comprehensiveness, and relevance, supported by human-verified ground truth.
4. Actionable Guidance: By uncovering that top-performing MSLMs can outperform larger proprietary models, the work challenges the "bigger is better" assumption. It provides actionable guidance for next-generation assistant design, emphasizing that architectural innovations (e.g., vision-language token alignment, cross-image reasoning) and targeted post-training (specifically on unanswerable queries and fine-grained distinctions) are more impactful than simple model scaling.

The authors conclude that while MSLMs demonstrate promising competence, they currently fall short of the safety-critical requirements for industrial inspection. RobustMAD serves as a diagnostic tool and a roadmap for developing reliable, deployable multimodal inspection agents.