InsightX Agent: An LMM-based Agentic Framework with Integrated Tools for Reliable X-ray NDT Analysis

This paper introduces InsightX Agent, a novel Large Multimodal Model-based agentic framework that integrates a Sparse Deformable Multi-Scale Detector with an Evidence-Grounded Reflection tool to achieve reliable, interpretable, and interactive X-ray non-destructive testing analysis, demonstrated by a 96.54% F1-score on the GDXray+ dataset.

The Gist

Imagine you are a quality control inspector at a factory. Your job is to look at X-ray images of metal parts (like car wheels or engine blocks) to find tiny cracks, holes, or bubbles hidden inside. This is a tough job. The images are grainy, the defects are tiny, and if you miss one, a car part could fail later.

For years, computers tried to help using "Deep Learning." Think of these old computer programs as very fast, but very literal robots. They could spot a defect and draw a box around it, but they couldn't explain why they thought it was a defect. If you asked, "Are you sure?" the robot would just say, "Confidence: 85%." It was a "black box"—you had to trust it blindly. Sometimes, it would get confused by shadows or scratches and scream "Defect!" when there was nothing wrong (a false alarm), or miss a tiny crack entirely.

Enter "InsightX Agent."

The authors of this paper didn't just build a better robot; they built a team of experts led by a brilliant, reasoning manager. Here is how it works, using simple analogies:

1. The Team Structure

Instead of one brain doing everything, InsightX Agent splits the job into three roles:

• The "Super-Sniffer" (SDMSD): Imagine a highly trained dog that can smell a tiny crumb in a massive stadium. This is the Sparse Deformable Multi-Scale Detector. It scans the X-ray image incredibly fast, looking everywhere at once. It finds everything that might be a defect, even the tiny, hard-to-see ones. It casts a wide net, catching hundreds of potential "suspects."
• The "Skeptical Detective" (The LMM Agent): This is the Large Multimodal Model (a super-smart AI that can see and read). It acts as the team leader. It doesn't just look at the image; it reads the "case file" (the X-ray) and talks to the "Sniffer."
• The "Evidence Reviewer" (EGR): This is the secret sauce. It's like a quality control inspector who double-checks the detective's work. It forces the AI to slow down and think: "Wait, is that really a crack, or just a shadow? Is this box drawn too big? Did we miss anything?"

2. How They Work Together (The "Agentic" Magic)

In the old days, the computer would just say, "Here is a list of defects."

With InsightX Agent, the process is more like a courtroom trial:

1. The Arrest (Detection): The "Super-Sniffer" finds a suspicious spot and says, "I think this is a crack!"
2. The Interrogation (Reflection): The "Detective" (the LLM) takes over. It doesn't just accept the arrest. It uses a tool called Evidence-Grounded Reflection (EGR).
   • Step 1: Context. "Okay, this is a metal casting. What does a real crack look like here?"
   • Step 2: Analysis. "The Sniffer drew a box here. Does the box fit the shape perfectly? No, it's too loose. Let's tighten it."
   • Step 3: Elimination. "Wait, this spot looks like a scratch on the surface, not a hole inside. That's a false alarm. Let's throw it out."
   • Step 4: Confidence Check. "The Sniffer was only 50% sure. But after looking closely, I see the evidence is actually quite strong. Let's bump the confidence up."
3. The Verdict (Output): The final report isn't just a list of coordinates. It's a story. It says: "I found three defects. Defect A is a confirmed crack with high confidence. Defect B is a small imperfection, but I'm not 100% sure, so I flagged it for a human to double-check. Defect C was a false alarm, and here is why I rejected it."

3. Why This is a Big Deal

• No More "Black Boxes": If a human operator asks, "Why did you flag this?", the AI can answer: "Because the shape is irregular and the density is lower than the surrounding metal." It explains its reasoning.
• Trust: Because it admits when it's unsure (flagging "Uncertain" defects) and explains why it rejected false alarms, human workers trust it more.
• Accuracy: By combining the speed of the "Sniffer" with the careful thinking of the "Detective," the system is much better at finding tiny defects and ignoring fake ones. In tests, it got 96.5% of the defects right, which is better than any previous method.

The Trade-off

There is one catch. Because this AI is "thinking" and "reflecting" like a human detective, it takes a little longer to give an answer (about a minute per image) compared to the instant, mindless speed of the old robots.

The Analogy:

• Old AI: A speed-reading machine that glances at a document and highlights words it thinks are mistakes. Fast, but often wrong.
• InsightX Agent: A senior editor who reads the document, checks a dictionary, asks a colleague, and then writes a detailed report explaining exactly what is wrong and why. Slower, but incredibly reliable and trustworthy.

In Summary:
This paper introduces a new way to use AI for industrial safety. Instead of just "seeing" defects, the system reasons about them. It acts like a tireless, hyper-observant expert who never gets tired, never misses a detail, and can explain its logic to the human workers, making industrial safety smarter and more reliable.

Technical Summary

Here is a detailed technical summary of the paper "INSIGHTX AGENT: An LMM-based Agentic Framework with Integrated Tools for Reliable X-ray NDT Analysis."

1. Problem Statement

Non-destructive testing (NDT), specifically X-ray inspection, is critical for industrial quality assurance but faces significant limitations with current AI-driven approaches:

• Lack of Interpretability: Existing deep learning models act as "black boxes," providing bounding boxes and confidence scores without explaining why a defect was detected, hindering operator trust.
• Limited Interactivity: Current systems cannot engage in dialogue, clarify ambiguous detections, or adapt to varying levels of operator expertise.
• Hallucinations and Precision Issues: Directly using Large Multimodal Models (LMMs) for visual grounding often leads to hallucinations (plausible but incorrect defect locations) or requires pixel-perfect annotated datasets that are costly to acquire.
• Inability to Self-Assess: Standard detectors lack the capacity for critical self-reflection to eliminate false positives or recalibrate confidence based on domain knowledge.

2. Methodology: INSIGHTX AGENT Framework

The authors propose INSIGHTX AGENT, a novel agentic framework where a Large Multimodal Model (LMM) acts as a central orchestrator rather than a passive analyzer. The system integrates two specialized tools to achieve reliable, interpretable, and interactive analysis.

A. Core Architecture

The framework operates in a loop where the LMM agent:

1. Recognizes Intent: Determines if the user query requires defect detection or general information.
2. Orchestrates Tools: Invokes the Sparse Deformable Multi-Scale Detector (SDMSD) for initial detection and the Evidence-Grounded Reflection (EGR) tool for validation.
3. Generates Output: Produces structured, interpretable reports with reasoning traces.

B. Key Components

1. Sparse Deformable Multi-Scale Detector (SDMSD):

   • Function: Generates dense defect proposals from multi-scale feature maps and sparsifies them using Non-Maximum Suppression (NMS).
   • Innovation: Utilizes deformable attention mechanisms to focus computational resources on sparse, high-probability candidates. This "dense-to-sparse" strategy is optimized for detecting small, densely clustered defects common in X-ray images while maintaining efficiency.
   • Role: Acts as the "eyes" of the agent, providing initial hypotheses (bounding boxes) rather than final diagnoses.

2. Evidence-Grounded Reflection (EGR):

   • Function: A structured, chain-of-thought-inspired protocol that guides the LMM to critically validate SDMSD proposals.
   • Process: It executes a six-stage sequential validation:
     1. Context Assessment: Evaluates image quality and structural patterns.
     2. Individual Defect Analysis: Performs visual verification, bounding box assessment, and confidence evaluation.
     3. False Positive Elimination: Systematically rejects artifacts, noise, or structural features misidentified as defects.
     4. Confidence Recalibration: Adjusts confidence scores based on evidence strength and consistency.
     5. Quality Assurance: Checks for completeness and overall reasonableness.
     6. Output Generation: Produces structured results (Confirmed, Uncertain, or Rejected) with reasoning logs.
   • Role: Acts as the "brain," applying domain knowledge to refine detections and prevent hallucinations.

3. Domain Adaptation (LoRA):

   • The LMM (based on Qwen2.5-VL) is fine-tuned using Low-Rank Adaptation (LoRA) to avoid catastrophic forgetting.
   • Two-Stage Curriculum:
     • Knowledge Injection: Training on a corpus of 1,000 NDT-specific Q&A pairs to learn terminology and principles.
     • Template Alignment: Imitation learning on structured analytical reports to ensure outputs follow industry-standard formats (Detection, Evidence, Assessment, Recommendation).

3. Key Contributions

1. Paradigm Shift: Moves from passive data processing to active, tool-driven reasoning, establishing a new standard for industrial NDT.
2. Novel Components:
   • SDMSD: A specialized detector optimized for small, dense X-ray defects.
   • EGR Mechanism: A structured reflection protocol that enables autonomous validation, false positive elimination, and confidence recalibration without requiring pixel-perfect training data for the LMM.
3. Interactivity & Trust: The system supports interactive dialogue, allowing operators to query the agent for clarifications, bridging the gap between AI outputs and human decision-making.

4. Experimental Results

The framework was evaluated on the GDXray+ dataset (aluminum casting components).

• Performance Metrics:

  • F1-Score: Achieved 96.54%, outperforming state-of-the-art methods like Faster R-CNN (89.73%), YOLOX-s (95.76%), and DINO (94.41%).
  • Precision: 94.77% (a 2.09% improvement over SDMSD alone), demonstrating the EGR's effectiveness in eliminating false positives.
  • Recall: 98.38% (a slight trade-off of 0.56% compared to SDMSD alone, prioritizing precision).
  • Stability: Exhibited the lowest standard deviation (±0.64% F1-score) across independent runs, indicating superior robustness.

• Ablation Studies:

  • LMM Only: Failed significantly (1.18% precision), proving direct grounding is insufficient for industrial precision.
  • SDMSD Only: High recall but lower precision, confirming the need for the EGR validation layer.
  • Full Framework: The synergy of SDMSD and EGR yielded the best balance of accuracy and reliability.

• Qualitative Analysis:

  • The system successfully handled complex scenarios, such as flagging low-confidence defects as "Uncertain" for expert review rather than discarding them, and rejecting redundant overlapping bounding boxes through logical reasoning.

5. Significance and Impact

• Trustworthy AI: By providing explicit reasoning traces and evidence-based justifications, INSIGHTX AGENT addresses the "black box" problem, fostering greater operator trust in automated systems.
• Industrial Applicability: The framework is designed for high-stakes environments where diagnostic transparency is crucial for safety and regulatory compliance.
• Scalability vs. Latency: While the EGR process introduces latency (approx. 45-60 seconds per image for full reflection), this is acceptable for batch quality control and high-value component analysis where accuracy outweighs raw speed. The authors suggest future optimizations (distillation, quantization) for real-time needs.
• Future Potential: The agentic approach offers a blueprint for integrating LLMs into other critical industrial domains, moving beyond simple classification to complex, interactive diagnostic workflows.

In conclusion, INSIGHTX AGENT demonstrates that combining specialized detection tools with the reasoning capabilities of LMMs creates a robust, interpretable, and highly accurate system for X-ray NDT, overcoming the limitations of both traditional deep learning and standalone LLM approaches.