⚡ electrical engineering

Vision-Language Based Expert Reporting for Painting Authentication and Defect Detection

This paper presents a fully automated vision-language model framework that integrates multi-modal pulsed active infrared thermography analysis with structured natural language reporting to enhance the objectivity, reproducibility, and systematic documentation of painting authentication and defect detection.

Original authors: Eman Ouda, Mohammed Salah, Arsenii O. Chulkov, Gianfranco Gargiulo, Gian Luca Tartaglia, Stefano Sfarra, Yusra Abdulrahman

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eman Ouda, Mohammed Salah, Arsenii O. Chulkov, Gianfranco Gargiulo, Gian Luca Tartaglia, Stefano Sfarra, Yusra Abdulrahman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the clues aren't hidden in a safe or a diary; they are hidden inside a painting.

This paper introduces a new "super-detective" system that combines heat-sensing cameras with smart AI to figure out if a painting is real, what's wrong with it, and how to fix it—all without touching the artwork.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Expert Guessing Game"

Traditionally, when conservators (art doctors) want to see what's happening under the paint layers (like cracks, glue failures, or hidden repairs), they use a technique called Active Infrared Thermography.

The Analogy: Imagine you are holding a hot plate of cookies. If there is a chocolate chip hidden inside, the heat moves differently around that chip than it does through the dough. By watching how the heat cools down, you can "see" the chip without breaking the cookie.
The Issue: This heat-camera data is messy. It's full of "static" (noise) and often looks different depending on who is looking at it. Experts have to stare at these heat maps and write long, complicated reports by hand. This is slow, subjective (one expert might see a crack, another might not), and hard to compare across different museums.

2. The Solution: The "Three-Legged Stool" + The "Translator AI"

The authors built a system that does two main things:

Step A: The "Three-Legged Stool" (Multi-Modal Analysis)

Instead of relying on just one way of looking at the heat, the system uses three different mathematical "lenses" to process the data:

Lens 1 (PCT): Looks for patterns in how the heat changes over time.
Lens 2 (TSR): Looks at the speed of the cooling.
Lens 3 (PPT): Looks at the "phase" or rhythm of the heat waves.

The Magic: Sometimes, Lens 1 sees a defect, but Lens 2 thinks it's just a shadow. Sometimes Lens 3 sees a defect that the others miss. The system acts like a jury. It only counts a "defect" as real if at least two or three of the lenses agree. This filters out the "fake" clues (artifacts) and keeps only the real problems.

Step B: The "Translator AI" (Vision-Language Model)

Once the system has found the real defects, it doesn't just give a scientist a bunch of numbers. It feeds the heat maps and the original photo into a Vision-Language Model (VLM).

The Analogy: Think of the VLM as a super-smart art historian intern. You show it the heat maps and say, "Tell me what you see, but be honest about what you don't know."
The Output: Instead of a confusing graph, the AI writes a clear, structured report in plain English. It says things like: "I see a suspicious area near the boy's shoulder. It looks like the glue might be failing, but I'm not 100% sure. It could also be an old repair."

3. The Test Case: The "Boy" and the "Girl"

The team tested this on two beautiful 19th-century Italian wooden art panels (marquetry) featuring a boy and a girl.

What they found: The system successfully spotted where the thin wood veneers were peeling away from the backing, where old restorations had been added, and where dirt had built up.
The Result: The AI produced consistent reports for both paintings. It didn't get confused by the edges of the painting (a common problem for heat cameras) and it clearly distinguished between "artistic features" and "damage."

4. Why This Matters

No More Guesswork: It creates a standard way to report damage, so Museum A and Museum B can compare notes easily.
Honest AI: The system is programmed to admit uncertainty. It won't say "This is a fake!" if it's not sure. It says, "Here is the evidence, and here is what it might mean."
Preservation: It helps art doctors fix paintings faster and more accurately, ensuring these treasures last for future generations.

In a nutshell: This paper describes a new tool that turns confusing heat data into a clear, honest, and standardized story about the health of a painting, acting as a bridge between complex physics and human decision-making.

1. Problem Statement

The assessment of cultural heritage objects, particularly layered paintings and marquetry, relies heavily on Non-Destructive Testing (NDT) techniques like Active Infrared Thermography (AIRT). While AIRT is effective at detecting subsurface defects (e.g., delamination, voids, moisture), its routine application in conservation faces three critical challenges:

Artifact Susceptibility: Thermal measurements are prone to noise from boundary heat loss, non-uniform excitation, and environmental reflections, leading to false positives.
Lack of Standardization: Interpretation is currently bespoke and expert-dependent. Different laboratories use varying post-processing parameters (e.g., for Principal Component Thermography or Pulsed Phase Thermography), resulting in inconsistent results and a lack of reproducibility.
Reporting Gap: There is no standardized framework to translate complex, multi-modal thermal data into structured, human-readable natural language reports. Existing methods produce numerical or image-based outputs that require manual expert interpretation, hindering systematic integration into conservation documentation.

2. Methodology

The authors propose a fully automated Thermography–Vision–Language Model (VLM) framework that bridges raw thermal data and structured expert reporting without human intervention during inference. The workflow consists of four main stages:

A. Data Acquisition & Preprocessing

Specimens: Two 19th-century Italian marquetry panels ("Boy" and "Girl") were tested.
Setup: Pulsed AIRT was performed using a Xenon flash (3.2 kJ) and a mid-wave infrared (MWIR) camera.
Preprocessing: Thermal sequences ( $\Delta T(x, y, t)$ ) underwent baseline correction, temporal smoothing, and Region of Interest (ROI) selection to ensure consistency.

B. Multi-Modal Feature Extraction

Three complementary thermographic processing techniques were applied to the thermal sequences to extract distinct physical features:

Principal Component Thermography (PCT): Uses Singular Value Decomposition (SVD) to separate dominant temporal modes, highlighting localized thermal heterogeneities while suppressing global diffusion trends.
Thermographic Signal Reconstruction (TSR): Models the logarithmic cooling curve using polynomial fitting. It extracts slope maps, which are highly sensitive to changes in heat diffusion caused by subsurface discontinuities.
Pulsed Phase Thermography (PPT): Applies Fourier Transform to the temporal signal to generate phase maps. Phase data is robust against non-uniform heating and surface emissivity variations, making it ideal for detecting interface separations.

C. Consensus Anomaly Detection

To mitigate the limitations of single-modality analysis, the framework fuses the outputs:

Mask Generation: Binary anomaly masks are created for PCT, TSR, and PPT outputs using standardized thresholds (z-score and percentile-based).
Fusion Strategy: A consensus mask is generated by retaining only regions detected by multiple modalities (specifically fusing TSR slope and PPT phase-edge masks).
Goal: This suppresses boundary artifacts and isolated noise while preserving spatially coherent anomalies supported by multiple physical indicators.

D. Vision-Language Model (VLM) Integration

The fused thermal data and the original optical image are fed into a VLM (specifically Qwen-VL-Chat).

Prompt Engineering: The model is guided by a structured, domain-constrained prompt that forces it to generate a report in three specific sections:
1. Thermal Output Analysis: Interpretation of PCT, TSR, and PPT maps with physical explanations.
2. Authenticity Assessment: A cautious discussion distinguishing observations from hypotheses, explicitly stating that thermography alone cannot confirm authenticity.
3. Defect Locations & Causes: A bulleted summary of anomaly locations, supporting modalities, and likely physical causes (e.g., delamination, retouching).
Constraints: The system is instructed to acknowledge uncertainty and avoid hallucinations by grounding all claims in the provided thermal evidence.

3. Key Contributions

First Integrated VLM Framework for Heritage Thermography: The paper introduces the first system that couples multi-modal AIRT analysis with structured, modality-aware natural language reporting for cultural heritage.
Consensus Anomaly Detection Strategy: A novel fusion method that prioritizes cross-modality agreement to suppress boundary artifacts and heat-loss errors, significantly improving the robustness of defect localization.
Standardized, Explainable Reporting: The framework generates reproducible, structured reports that explicitly separate factual thermal observations from interpretive hypotheses, addressing the "black box" nature of AI in scientific diagnostics.
Uncertainty Quantification: The system is designed to explicitly acknowledge diagnostic limitations and uncertainty, a critical requirement for scientific and conservation applications.

4. Results

The framework was validated on the "Boy" and "Girl" marquetry samples:

Thermal Analysis:
- PCT effectively captured global gradients but required intermediate components (PC2-PC4) to isolate defects.
- TSR Slope maps enhanced defect visibility by highlighting deviations from ideal cooling.
- PPT Phase provided the clearest spatial coherence for structural anomalies, outperforming amplitude maps in handling surface heterogeneity.
Consensus Performance:
- The consensus strategy reduced the anomaly area to 8.84% (Boy) and 7.02% (Girl) of the ROI, filtering out noise present in individual modalities.
- The "Boy" sample showed concentrated detachments, while the "Girl" sample exhibited fragmented adhesion weaknesses.
VLM Output:
- The VLM successfully generated structured reports that correctly identified anomaly locations (e.g., edges, background, specific figures).
- The model provided plausible physical interpretations (e.g., "material heterogeneity," "past interventions") while strictly avoiding definitive claims on authenticity, adhering to the "cautious" prompt constraints.
- The reports were consistent across both samples, demonstrating the framework's generalizability.

5. Significance and Impact

This research represents a paradigm shift in conservation science by moving from expert-dependent, manual interpretation to automated, standardized, and explainable reporting.

Reproducibility: By standardizing the processing pipeline and reporting format, the framework allows for consistent comparison across different collections and laboratories.
Accessibility: It lowers the barrier for non-experts to understand complex thermographic data through natural language summaries.
Scalability: The modular design allows for the integration of other NDT techniques (e.g., X-ray, hyperspectral) in the future, with the VLM serving as a unified interface for multi-source data synthesis.
Trustworthiness: By enforcing physical grounding and explicit uncertainty statements, the framework mitigates the risks of AI hallucination, making it a viable tool for supporting, rather than replacing, expert conservation judgment.