SWST-Net: Semantic-aware Wavelet-drivenStructure and Texture Interaction Network forImage Inpainting

Technical Summary: SWST-Net for Image Inpainting

Problem Statement
Image inpainting aims to reconstruct missing regions in an image while preserving structural, textural, and semantic consistency with the surrounding content. While traditional exemplar-based methods struggle with large missing areas and complex backgrounds due to a lack of semantic understanding, deep learning-based approaches (e.g., GANs) have improved performance but face specific limitations. Vanilla convolutions often result in over-smoothed boundaries or textural artifacts. Furthermore, existing methods that attempt to jointly model structure and texture often suffer from entangled representations where the two information types interfere with each other. Conversely, two-stage strategies that generate structure first and then texture often fail because predicting accurate structural layouts from heavily corrupted images is inherently difficult, leading to erroneous priors that degrade the final texture synthesis. The core challenge remains effectively decoupling structure and texture representations while promoting their interaction under semantic guidance to achieve high-fidelity reconstruction.

Methodology: SWST-Net
The authors propose the Semantic-aware Wavelet-driven Structure and Texture Interaction Network (SWST-Net). The architecture is designed to explicitly separate and then collaboratively reconstruct structure and texture information.

1. Frequency-Domain Decomposition (DWT):
   The input image is decomposed using the Discrete Wavelet Transform (DWT) with Haar wavelets. This separates the image into a low-frequency sub-band (LL), representing global structure, and high-frequency sub-bands (LH, HL, HH), representing fine-grained texture details. These components are fed into separate encoders.

2. Dual-Encoder and Shared-Decoder Framework:

   • Structure Branch: Processes the low-frequency sub-band. It utilizes a Multi-Scale Enhancement Residual Block (MSEB) containing Multi-Scale Attention Enhancement (MSAE) modules to capture global structural features.
   • Texture Branch: Processes the concatenated high-frequency sub-bands. It employs Fast Fourier Convolution Residual Blocks (FFCB) to capture both local details and long-range contextual information.
   • Shared Decoder: Reconstructs the final image from the fused features using an inverse wavelet transform (IDWT) and upsampling.

3. Semantic-Guided Alignment Module (SGAM):
   To ensure semantic consistency between the two branches, the network integrates high-level semantic priors extracted from a pre-trained Masked Autoencoder (MAE). Through a cross-attention mechanism, these semantic features are aligned with the structure and texture features in the encoders, guiding the inpainting process to maintain semantic coherence.

4. Bi-directional Cross-Domain Feature Fusion Module (BCFFM):
   Located at the bottleneck of both branches, this module facilitates interaction between the structure and texture streams. Inspired by spatially-adaptive de-normalization, it learns dynamic affine parameters to modulate instance normalization layers. This allows texture features to refine structure features and vice versa, enabling dynamic integration without losing domain-specific information.

5. Training Objective:
   The model is trained using a multi-task loss function comprising L1 loss (on unmasked regions), adversarial loss (using WGAN-GP), feature matching loss, and a High Receptive Field (HRF) perceptual loss based on a pre-trained ResNet-50 with dilated convolutions.

Key Contributions

1. Decoupled Architecture: The paper introduces a dual-branch framework that uses wavelet transforms to explicitly decouple structure and texture, addressing the interference issues found in shared-generator models.
2. Semantic Integration: The SGAM effectively injects high-level semantic priors from a pre-trained MAE into the dual-branch process, ensuring semantic consistency without relying on potentially erroneous structural predictions from corrupted inputs.
3. Cross-Domain Interaction: The BCFFM is designed to enable effective, bidirectional guidance between structure and texture branches, enhancing collaborative optimization.
4. Performance: The method achieves state-of-the-art results on standard benchmarks, balancing computational efficiency with reconstruction quality.

Experimental Results
The authors evaluated SWST-Net on three datasets: Paris StreetView, CelebA-HQ, and Places.

• Quantitative Performance: The method consistently outperforms recent state-of-the-art approaches (including CTSDG, WSA-GAN, W-Net, CoordFill, HINT, and SEM-Net) across various masking ratios (10%–50%). It achieves superior scores in pixel-level metrics (PSNR, SSIM) and perceptual metrics (FID, LPIPS). Notably, on the Places dataset, it achieved the lowest FID (10.891) and competitive no-reference quality scores (NIQE and BRISQUE).
• Qualitative Performance: Visual comparisons demonstrate that SWST-Net produces sharper structural lines (e.g., brick walls, window frames) and more realistic textures (e.g., facial details, plant filaments) compared to competitors, which often exhibit blurring, distortion, or color transitions.
• Ablation Studies: Experiments confirmed the necessity of each component. Removing the DWT (w/o DWT) or the MAE guidance (w/o MAE) led to significant drops in performance. Similarly, replacing the BCFFM with simple concatenation or removing the MSEB resulted in degraded structural coherence and texture fidelity.
• Efficiency: With 113M parameters and an inference time of 70ms per image, SWST-Net offers a favorable trade-off between computational cost and performance, being significantly faster than HINT and SEM-Net while providing higher quality than CTSDG and WSA-GAN.

Significance and Limitations
The paper claims that SWST-Net offers a promising solution for high-quality image inpainting by successfully addressing the "chicken-and-egg" dilemma of structure-texture dependency through wavelet-driven decoupling and semantic guidance. The method effectively preserves structural integrity and texture details simultaneously.

However, the authors modestly acknowledge limitations. The model still struggles with very large missing regions where contextual information is insufficient to constrain the generation, leading to smooth but potentially inaccurate content. Additionally, the model fails when reconstructing specific semantic elements like text or logos that require knowledge not present in the surrounding context or the pre-trained semantic priors. These areas are identified as directions for future work.