Visual Context-Aware and Spatial Receptive Field-Based algorithm for mural inpainting

Technical Summary: Visual Context-Aware and Spatial Receptive Field-Based Algorithm for Mural Inpainting (VCSR)

Problem Statement
The restoration of Dunhuang murals faces significant challenges due to the unique degradation patterns of these cultural relics, including cracking, peeling, and pigment loss. While digital image restoration offers a non-invasive alternative to physical repair, existing deep learning methods struggle with specific limitations in this domain. Generative Adversarial Networks (GANs) often fail to accurately reproduce historical elements and may introduce noise or structural distortions. Convolutional Neural Networks (CNNs) are limited by receptive field constraints, making it difficult to model long-range dependencies required for complex mural patterns. Conversely, Transformer-based models, while effective at global correlation, often suffer from "blocky artifacts" due to insufficient information exchange between windows and high computational costs. Furthermore, State-Space Models (SSMs) like Mamba, though efficient for long sequences, typically rely on causal scanning strategies that flatten 2D images into 1D sequences. This process causes "local pixel forgetting," where adjacent spatial pixels are separated in the sequence, leading to the loss of neighborhood information and insufficient spatial perception.

Methodology
The authors propose the Visual Context-Aware and Spatial Receptive Field-Based (VCSR) algorithm, built upon a U-Net architecture with an encoder-decoder structure. The method addresses the aforementioned limitations through three primary innovations:

1. Residual State-Space Block (RSSB):

   • Attentive State-Space Module (ASSM): To overcome the causal limitations of standard SSMs, the ASSM integrates Prompt Vectors into the state-space equations. These vectors act as semantic cues, allowing the model to "query" globally similar pixels rather than relying solely on the sequential order of the flattened image. This enables the model to break free from traditional scanning orders and achieve global awareness.
   • Local Compensation: Recognizing that flattening 2D features into 1D sequences disrupts local spatial relationships, the RSSB incorporates local convolutions after the ASSM to restore neighborhood similarity and refine local details.
   • Channel Attention: A channel attention mechanism is embedded to identify and enhance key channel representations, mitigating channel redundancy often found in SSMs.

2. Multi-Scale Deformable Sliding Window Attention (MSDA):

   • Located in the decoder, this module extends traditional sliding window attention by introducing deformable offsets. These offsets dynamically adjust the position and shape of the attention window, allowing the model to focus on key regions beyond fixed local windows.
   • The module operates across multiple scales with different kernel sizes for different attention heads, facilitating the capture of hierarchical features from coarse to fine. This enhances the representation of irregular spatial patterns common in murals.

3. Spatially Perceptive Feedforward Network (SPFN):

   • To further strengthen spatial awareness, the SPFN extracts spatial information from both the input and output of the MSDA module.
   • It utilizes a gating mechanism to integrate this spatial information into the main feature stream. By generating a spatially perceptive indicator, the SPFN guides the network to better preserve positional information and long-range dependencies, ensuring the reconstruction of local details aligns with global structural consistency.

Key Contributions
The paper outlines three main contributions:

1. The design of the RSSB, which combines an Attentive State-Space Module with prompt vectors to achieve global semantic awareness and local convolutions to compensate for spatial information loss, effectively addressing local pixel forgetting.
2. The proposal of the MSDA module, which utilizes deformable offsets and multi-scale processing to expand the receptive field adaptively, improving feature extraction across different spatial ranges and reducing blocky artifacts.
3. The introduction of the SPFN, which enhances spatial dependencies and positional awareness through a gating mechanism, improving the reconstruction quality of local details.

Experimental Results
The VCSR algorithm was evaluated on the Dunhuang Mural Challenge dataset (2,850 images), covering various damage severities (1% to 40% masking) and random powder masks. The method was compared against state-of-the-art models including MISF, PUT, HINT, ECDB, SEM, and MXT.

• Quantitative Performance: VCSR achieved superior results across all mask ratios. Improvements included:
  • PSNR: Increased by 0.18% to 0.94% compared to the best baseline.
  • SSIM: Increased by 1.17% to 2.98%.
  • L1 Error: Reduced by 16.5% to 35.92%.
  • LPIPS: Decreased by 6.53% to 14.31%, indicating significant gains in perceptual quality.
• Qualitative Analysis: Visual comparisons demonstrated that VCSR produces smoother transitions, better preserves edge details (e.g., facial features, jewelry, and flame textures), and avoids the structural misalignments and blurring observed in competing methods.
• Ablation Studies: Experiments confirmed that each module (RSSB, MSDA, SPFN) contributes positively to performance, with the full integration yielding the best results in both metrics and visual fidelity.

Significance and Claims
The paper claims that the VCSR algorithm provides a robust solution for the digital restoration of cultural heritage, specifically addressing the dual challenges of local pixel forgetting and insufficient spatial perception. By effectively modeling global dependencies while preserving local spatial coherence, the method offers a significant improvement in the realism and structural integrity of restored murals. The authors note that while the algorithm excels in technical restoration metrics, future work should focus on integrating specific artistic styles and historical-cultural contexts to further enhance the cultural fidelity and artistic authenticity of the restoration process. The work aims to support the non-contact, repeatable, and cost-effective preservation of the Mogao Grottoes' invaluable historical records.