How Do Inpainting Artifacts Propagate to Language?

This paper investigates how visual artifacts from diffusion-based inpainting systematically degrade language generation in vision-language models, establishing a diagnostic framework that links pixel-level reconstruction fidelity to semantic caption quality through the analysis of intermediate model representations.

The Gist

Imagine you have a very smart, artistic assistant who is great at looking at pictures and describing them in words. Let's call this assistant the "Picture-to-Word Translator."

Now, imagine someone hands the assistant a photo that has a big, ugly hole in the middle. The assistant can't see the hole, so they ask a "Magic Paintbrush" (a computer program called an AI inpainting model) to fill in the missing part.

The Magic Paintbrush does a great job. To your eyes, the hole is gone, and the picture looks perfect. But here's the catch: The Magic Paintbrush is a bit of a liar. It fills the hole with something that looks right but isn't actually what was there. Maybe it paints a cat where a dog used to be, or changes a red shirt to a blue one, just because it "thinks" that's what usually goes there.

The Big Question:
When the Picture-to-Word Translator sees this "fixed" photo, do they notice the lie? Or do they confidently describe the fake cat as a real one?

This paper is a detective story about exactly that. The researchers wanted to see how these tiny, invisible lies created by the Magic Paintbrush mess up the words the Translator writes.

The Experiment: A Two-Stage Test

The researchers set up a simple game to test this:

1. The Setup: They took thousands of photos and digitally "cut out" the center of them.
2. The Fix: They used different versions of the Magic Paintbrush (based on a technology called Diffusion) to fill in the holes. Some versions were very careful; others were a bit sloppy.
3. The Translation: They showed both the original photo and the fixed photo to the Picture-to-Word Translator.
4. The Comparison: They compared the descriptions. Did the translator say "a man in a blue shirt" for the original, but "a woman in a blue shirt" for the fixed one?

What They Found (The "Aha!" Moments)

1. The "Look Good" Trap
The researchers found that just because a photo looks perfect to our eyes doesn't mean the words will be correct.

• Analogy: Imagine a magician who swaps a real apple for a plastic one. It looks exactly like an apple. If you ask a child to describe the fruit, they might say "It's a red, crunchy apple." But if you bite into it, it's plastic. The Magic Paintbrush creates "plastic apples" that fool the eye but confuse the brain.
• The Result: When the paintbrush made small, subtle changes (like blurring the edges of the hole), the Translator was usually fine. But when the paintbrush just "cut and pasted" a new object into the hole, the Translator got confused and started making up facts (hallucinating).

2. The "Deep Brain" Confusion
The researchers looked inside the Translator's "brain" (its computer layers) to see what was happening.

• Analogy: Think of the Translator's brain like a factory assembly line. The first workers just look at colors and shapes. The workers at the end of the line decide what the object is (e.g., "That's a dog, not a cat").
• The Result: They found that the fake paintings didn't confuse the first workers. But by the time the image reached the end of the assembly line, the workers were totally confused. The "lie" from the paintbrush grew bigger as it traveled through the brain, causing the final description to be wrong.

3. The "Smooth vs. Sharp" Rule
They discovered that how you hide the hole matters.

• Analogy: If you tear a piece of paper and try to tape it back on with jagged, sharp edges, it's obvious. But if you gently fade the edges so it blends in, it's harder to spot.
• The Result: When the Magic Paintbrush used "smooth" fading to fill the hole, the Translator kept its cool and wrote good descriptions. When it used "sharp" cuts, the Translator panicked and wrote nonsense.

Why This Matters

This isn't just about fixing photos. It's about trust.

We are starting to use AI to fix old photos, remove unwanted people from vacation pics, or even generate medical images for doctors. If we use these tools, we need to know: If the AI fixes the picture, can we trust the description that comes with it?

The paper gives us a warning: Don't trust the picture just because it looks good. If an AI has "painted over" a part of an image, the words it generates about that image might be lying to you, even if the picture looks perfect.

The Bottom Line

The researchers built a "lie detector" for AI. They proved that when AI fixes a broken picture, it often introduces tiny lies that trick the AI into writing false stories. To get the truth, we need to check not just how the picture looks, but how the AI thinks about it.

Technical Summary

1. Problem Statement

Vision-Language Models (VLMs) are increasingly deployed in multi-stage pipelines where visual inputs are processed or reconstructed (e.g., via image inpainting) before being fed into language models. While modern diffusion-based inpainting models produce visually plausible reconstructions optimized for pixel-level realism, they often introduce subtle, semantically meaningful artifacts.

The core problem addressed is that downstream captioning models lack awareness of which image regions have been synthesized. Consequently, they treat these artifacts as genuine visual evidence, leading to:

• Object substitutions (e.g., a man becomes a woman).
• Attribute changes (e.g., shirt color changes).
• Category-level errors.

The paper investigates the extent to which reconstruction fidelity influences downstream caption correctness and semantic grounding, a relationship that remains under-explored despite the widespread use of inpainting in VLM pipelines.

2. Methodology

The authors propose a two-stage diagnostic framework designed to isolate the impact of visual reconstruction artifacts on language generation without retraining the models.

• Framework:

  1. Degradation: Input images are synthetically degraded using three masking strategies:
     • Hard Center Mask: Complete removal of a central region.
     • Gaussian Blur: Smooth attenuation of high-frequency details.
     • Low-Dimensional Degradation: Aggressive compression preserving layout but removing fine semantics.
  2. Reconstruction: Degraded images are reconstructed using diffusion-based inpainting models (Stable Diffusion 1.5, 2.0, and 3.0).
  3. Captioning: Both original and reconstructed images are passed to frozen VLMs (LLaVA, BLIP, Qwen2.5-VL) to generate captions.
  4. Analysis: The resulting captions are compared against ground truths using linguistic and semantic metrics.

• Evaluation Axes:

  1. Correlation Analysis: Examining the relationship between reconstruction fidelity metrics (MSE, PSNR, SSIM, LPIPS) and caption quality metrics (BLEU, METEOR, ROUGE-L, SimCSE, SBERT).
  2. Internal Representation Analysis: Analyzing a frozen ViT-Base encoder to measure:
     • Embedding Similarity: Cosine similarity between CLS tokens of original vs. reconstructed images.
     • Attention Drift: Layer-wise Total Variation Distance (TVD) of attention maps.
     • Spatial Localization: Whether attention drift is confined to the reconstructed regions.

• Datasets: The study spans diverse domains:

  • Natural images (Flickr, RefCOCOg).
  • Medical imaging (ROCOv2, Indiana X-Ray).
  • Audio spectrograms (GTZAN).
  • Time-series plots (TRUCE).

3. Key Contributions

1. Model-Agnostic Diagnostic Framework: A novel setup to analyze reconstruction-induced effects in vision-language pipelines by freezing all models and varying only the input reconstruction process.
2. Empirical Evidence of Fidelity-Dependence: Multi-dataset evidence demonstrating a consistent, positive correlation between visual reconstruction quality and downstream caption quality.
3. Internal Mechanism Analysis: Discovery that inpainting artifacts cause layer-dependent attention drift and spatially localized changes in vision encoders, specifically affecting deeper layers where semantic features are formed.

4. Key Results

A. Reconstruction Fidelity Correlates with Caption Quality

• Strong Predictors: Perceptual distance (LPIPS) and pixel-level error (MSE) show the strongest and most consistent correlations with caption quality. Lower error/higher perceptual realism leads to better semantic grounding.
• Weak Predictors: Structural Similarity Index (SSIM) shows weak or inconsistent correlations, particularly for natural images, suggesting global structural similarity is insufficient to predict semantic correctness.
• Domain Variance:
  • Natural Images (Flickr, RefCOCOg): Strong, monotonic correlations.
  • Medical (ROCOv2): Strong correlations despite higher absolute reconstruction errors, indicating semantic grounding is preserved if structural cues remain.
  • Failure Cases (GTZAN, X-Ray): No meaningful correlation found. This is attributed to "caption impoverishment"—the ground truth captions are too generic or repetitive (e.g., genre labels) to reflect visual changes, rendering reconstruction fidelity irrelevant to the output.

B. Masking Strategy Impacts Semantic Stability

• Hard Center Masking: Induces the largest drops in caption quality. The abrupt spatial discontinuity disrupts object identity and relational cues.
• Smooth Degradation (Gaussian/Low-Dim): Preserves caption quality more effectively, as the spatial continuity helps the model maintain object identity despite semantic loss.

C. Internal Representation Shifts

• Layer-Dependent Drift: Attention drift (measured by TVD) increases with network depth. Early layers remain stable, while deeper layers (responsible for high-level semantics) show significant divergence between original and reconstructed inputs.
• Spatial Localization: Attention drift is spatially localized to the reconstructed regions. The unmasked outer regions remain stable, confirming that the model's semantic instability is directly tied to the inpainted content.
• Embedding Separation: UMAP visualizations show that hard center-masked reconstructions create a clear separation from the original embedding distribution, whereas smoother masking strategies maintain closer alignment.

5. Significance and Implications

• Diagnostic Utility: The paper provides a practical framework for evaluating the robustness of VLMs against upstream visual preprocessing. It highlights that standard pixel-level metrics (like PSNR) are not enough; perceptual metrics (LPIPS) are better predictors of semantic failure.
• Pipeline Awareness: It underscores the necessity of "reconstruction-aware" diagnostics. In multi-stage pipelines, optimizing for pixel realism does not guarantee semantic fidelity, and subtle artifacts can cascade into significant hallucinations in language generation.
• Limitations & Future Work: The study focuses on diffusion-based inpainting and frozen models. It notes that end-to-end trained systems might adapt differently, and the findings may not generalize to tasks like Visual Question Answering (VQA) where reasoning is more complex.

In conclusion, the paper establishes that visual reconstruction quality is a critical determinant of language generation quality in VLMs. It demonstrates that artifacts introduced by inpainting are not merely visual noise but systematically alter the internal semantic representations of the model, leading to predictable degradation in downstream language tasks.