Vision-Language Models for Infrared Industrial Sensing in Additive Manufacturing Scene Description

1. Problem Statement

Context: Industrial manufacturing, particularly additive manufacturing (3D printing), often operates in low-light or enclosed environments where standard RGB cameras fail. Thermal infrared (IR) cameras provide a robust alternative by capturing temperature contrasts. However, traditional AI solutions for IR imaging rely on supervised deep learning, which requires large, labeled datasets specific to each task, machine, and material. Collecting and labeling this data is time-consuming, labor-intensive, and difficult to scale.

The Gap: While Vision-Language Models (VLMs) like CLIP have revolutionized zero-shot learning (ZSL) for RGB images, they are trained exclusively on natural RGB data. Consequently, they cannot directly interpret single-channel thermal IR data. There is a lack of frameworks that adapt these powerful foundation models to thermal imaging for industrial monitoring without retraining.

Objective: The authors propose VLM-IRIS, a zero-shot framework designed to adapt pre-trained VLMs to thermal infrared data for object presence detection (determining if a part is on the build plate) in 3D printing, eliminating the need for task-specific retraining or labeled IR datasets.

---

2. Methodology: VLM-IRIS Framework

The framework bridges the modality gap between thermal IR and RGB-trained VLMs through three core components:

A. Modality Adaptation (Preprocessing)

Since VLMs (e.g., CLIP) expect 3-channel RGB inputs, raw single-channel thermal data must be converted. The authors tested three preprocessing strategies:

1. Grayscale Mapping: Normalizing thermal intensity and replicating it across three channels. This preserves physical accuracy but lacks color cues.
2. Magma Colormap: A scientific colormap mapping intensity to an orange-red spectrum.
3. Viridis Colormap: A scientific colormap mapping intensity to a green-yellow spectrum.
   Hypothesis: Colormaps (Magma/Viridis) might align better with the color statistics of VLMs pre-trained on natural images, whereas Grayscale tests the model's ability to interpret pure intensity.

B. Prompt Engineering (Centroid Ensembling)

Instead of using a single text prompt per class (e.g., "an object is present"), the authors utilize a Prompt Bank:

• Prompt Bank: Multiple descriptive sentences are generated for each class ("present" and "absent") to capture linguistic variations.
• Centroid Embedding: The text embeddings of all prompts in a bank are averaged to create a single centroid vector for each class.
• Benefit: This reduces sensitivity to specific wording and provides a more stable semantic representation compared to single-prompt inference.

C. Inference via Cosine Similarity

1. Encoding: The preprocessed IR image is fed into the CLIP image encoder (ViT-B/32). The prompt centroids are fed into the CLIP text encoder.
2. Similarity Scoring: The system calculates the cosine similarity between the image embedding and the class centroids.
3. Classification: The class with the highest similarity score is selected as the prediction.
   Note: The framework operates entirely in a zero-shot manner; no fine-tuning or retraining on thermal data occurs.

---

3. Experimental Setup

• Hardware: Prusa MK3S 3D Printer and a FLIR Boson thermal camera mounted overhead.
• Dataset: 200 images (balanced between "object present" and "object absent").
  • Conditions: Captured under two thermal states: Hot Bed (85°C) and Room Temperature (34°C).
  • Variability: Parts varied in geometry, size (50–150mm), and height (4–70mm) to test generalization.
• Model: CLIP ViT-B/32 (pretrained on ~400M image-text pairs).
• Metrics: Accuracy, Precision, Recall, and F1-score.

---

4. Key Results

The study evaluated different combinations of preprocessing (Grayscale, Magma, Viridis) and prompting (Single vs. Centroid) across temperature conditions.

Condition	Best Performing Config	Accuracy	Key Observations
Room Temp	Magma + Centroid	100%	Perfect precision and recall. Magma's color gradients aligned best with CLIP's learned features.
Room Temp	Grayscale + Centroid	99%	Highly effective, proving CLIP can interpret pure thermal contrast when gradients are stable.
Hot Bed	Magma + Centroid	92%	Significantly outperformed Grayscale (83%). The "hot" condition creates soft thermal boundaries; Magma's artificial colors helped CLIP distinguish edges.
General	Centroid vs. Single	N/A	Centroid prompting consistently outperformed single-prompt strategies, reducing variance and improving robustness.

Key Findings:

• Temperature Impact: Performance dropped on the "Hot Bed" dataset for all methods. High temperatures create soft thermal gradients and sensor noise that confuse the model, which expects sharp RGB contrasts.
• Preprocessing Superiority: Magma was the most robust colormap. It introduced structured color gradients that mimicked the patterns VLMs were trained on, helping the model separate objects from the background even when thermal contrast was low.
• Prompting Strategy: Averaging multiple prompts (Centroid) significantly improved stability and accuracy compared to relying on a single sentence.

---

5. Key Contributions

1. VLM-IRIS Framework: The first systematic investigation into adapting Vision-Language Models for zero-shot thermal infrared industrial sensing.
2. Modality Bridging: Demonstrated that simple preprocessing (specifically Magma colormap) can effectively translate thermal data into a domain where RGB-trained VLMs can operate without retraining.
3. Prompt Engineering for IR: Validated that centroid-based prompt ensembling is critical for robust zero-shot classification in thermal imaging, mitigating the sensitivity to specific phrasing.
4. Label-Free Monitoring: Proved that reliable object detection in additive manufacturing can be achieved without collecting large, labeled thermal datasets, addressing a major bottleneck in industrial AI adoption.

---

6. Significance and Future Work

Significance:

• Scalability: This approach allows manufacturers to deploy monitoring systems for new machines or materials instantly by simply updating text prompts, rather than retraining models.
• Cost Reduction: Eliminates the need for expensive, time-consuming data labeling campaigns.
• Reliability: Offers a flexible solution for enclosed or low-light manufacturing environments where RGB cameras fail.

Future Directions:

• Defect Detection: Extending the framework from binary presence detection to multi-class defect identification (e.g., warping, delamination) using descriptive text prompts.
• Fine-Tuning: Investigating lightweight fine-tuning on thermal datasets to further improve accuracy.
• Localization: Integrating mechanisms to not just detect if an object is present, but where it is located (segmentation).
• Explainability: Using XAI techniques to understand which visual cues the VLM relies on when interpreting thermal data.

In conclusion, the paper establishes that VLM-IRIS is a viable, high-accuracy, and adaptable solution for label-free thermal monitoring in additive manufacturing, with the Magma colormap + Centroid prompting configuration emerging as the optimal strategy.