Synthetic Craquelure Generation for Unsupervised Painting Restoration

1. Problem Statement

The preservation of cultural heritage requires non-invasive digital methods to restore cracked paintings (craquelure). However, existing approaches face significant challenges:

• Data Scarcity: There is a severe lack of pixel-level annotated datasets for training deep learning models on painting cracks.
• Complexity of Patterns: Craquelure consists of extremely thin, irregular, low-contrast fissures that are difficult to distinguish from natural brushstrokes, textured paint, and aged varnish.
• Limitations of Current Methods:
  • Classical methods (e.g., morphological top-hat transforms) often require extensive parameter tuning and fail on textured surfaces or low-contrast cracks.
  • Deep learning methods struggle with extreme class imbalance (cracks are sparse) and tend to oversmooth fine structures, failing to localize thin patterns.
  • General photo restoration models (trained on synthetic photographic degradation) fail to capture the unique morphology of painting cracks.

2. Methodology

The authors propose a fully unsupervised, annotation-free framework that combines classical image processing with a learning-based refinement module. The pipeline consists of four main stages:

A. Synthetic Data Generation

To bypass the need for manual annotations, the authors created a domain-specific synthetic dataset:

• Generator: Uses Bézier trajectories to simulate crack paths.
• Geometry: Models realistic crack features, including tapered geometry (thinner at ends, thicker in the middle) and branching patterns.
• Realism: Applies morphological erosion and Gaussian blurring to mimic pigment bleeding and soft edges.
• Dataset: Generates triplets $(I, M, \tilde{I})$ consisting of a clean painting, a binary crack mask, and a damaged version where cracks are filled with gray values.

B. Initial Crack Detection (Classical)

A coarse, over-inclusive mask is generated using:

1. Morphological Top-Hat Transform: Specifically the black top-hat to highlight dark cracks against lighter backgrounds.
2. Size-Based Filtering: Removes connected components smaller than 5 pixels to eliminate noise.
   Result: A binary mask containing candidate cracks but also many false positives (brushstrokes).

C. Learning-Based Refinement (Deep Learning)

A SegFormer backbone (MiT-B0) is used to refine the coarse mask and suppress false positives.

• Parameter-Efficient Fine-Tuning (LoRA): Instead of full fine-tuning, Low-Rank Adaptation (LoRA) is applied to attention and feed-forward layers. This allows stable training on the limited synthetic dataset.
• Detector-Guided Input: The model takes a 4-channel input: the RGB image + the initial binary detection mask. This guides the network to focus on refining candidate regions.
• Loss Function: A Masked Hybrid Loss is used:
  • Weighted Cross-Entropy: Focuses training on detector-activated areas.
  • Dice Loss: Ensures global structural consistency for thin cracks.
• Logit Adjustment: The initial detector mask is added to the model's output logits as a soft prior, biasing the model toward predicting cracks where the detector found them, while allowing correction via ground truth supervision.

D. Crack Inpainting (Restoration)

Once the final crack mask is refined, the damaged pixels are restored using two complementary methods applied only to crack regions:

1. Modified Trimmed Mean (MTM): A non-recursive, outer-to-inner filling scheme using local neighborhood statistics. Effective for thin, line-like cracks.
2. Anisotropic Diffusion (AD): A Partial Differential Equation (PDE) based method that diffuses intensity along homogeneous directions (isophotes) while preserving edges. This is superior for wider cracks and textured areas.

3. Key Contributions

1. First Fully Unsupervised Framework: The system performs crack detection and restoration entirely without manual pixel-level annotations, relying solely on synthetic data.
2. Domain-Specific Synthetic Dataset: A novel generator creating realistic, tapered, and branched craquelure patterns that generalize well to real artworks.
3. Hybrid Classical-Learning Architecture: A unique "detector-guided" strategy where a classical morphological detector provides a spatial prior to a transformer-based segmentation network. This mitigates the class imbalance problem and preserves high-frequency details that pure transformers often lose.
4. Integrated Workflow: Combines interpretable detection, refinement, and crack-aware inpainting into a single, computationally practical pipeline for conservation.

4. Experimental Results

The method was evaluated on four real-world paintings with manually annotated ground truth (used only for testing, not training).

• Performance vs. Baselines:
  • Outperformed the traditional Grassfire algorithm (a state-of-the-art interactive method) in all metrics (Accuracy, F1, SSIM).
  • Significantly outperformed SOTA photographic restoration models (Wan et al., Pik-Fix) in a zero-shot setting.
  • Key Metric: Achieved an SSIM of 64.87, compared to 55.71 for the next best model (Pik-Fix).
• Ablation Study:
  • The LoRA adaptation provided the largest performance gain, enabling stable training on synthetic data.
  • The Masked Hybrid Loss was critical for suppressing false positives (F1 score improved from ~28% to ~61%).
  • Anisotropic Diffusion (AD) consistently yielded higher restoration quality (SSIM) than MTM, particularly for preserving structural details in wider cracks.
• Qualitative Results: The pipeline successfully isolated fine craquelure without blurring or removing original brushwork, a common failure point in transformer-based models.

5. Significance

This work addresses a critical bottleneck in digital heritage conservation: the lack of annotated data for training AI models on delicate artworks. By proving that synthetic data driven by domain-specific physics (crack geometry) can train a model to generalize to real-world scenarios, the authors provide a robust, non-invasive tool for conservators. The hybrid approach successfully bridges the gap between the interpretability of classical image processing and the semantic power of deep learning, offering a new paradigm for restoring fragile cultural artifacts without manual supervision.