💻 computer science

Thinking inside the Convolution for Image Inpainting: Reconstructing Texture via Structure under Global and Local Side

This paper proposes a novel image inpainting method that mitigates information loss during convolutional downsampling by leveraging a mutual reconstruction guidance strategy between structure and texture feature maps via statistical normalization and denormalization, achieving state-of-the-art performance across various resolutions.

Original authors: Haipeng Liu, Yang Wang, Biao Qian, Yong Rui, Meng Wang

Published 2026-02-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haipeng Liu, Yang Wang, Biao Qian, Yong Rui, Meng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, detailed painting, but someone has scratched out a large chunk of it with a marker. Your goal is to repaint that missing area so perfectly that no one can tell it was ever damaged. This is called image inpainting.

For a long time, computers tried to fix this by looking at the pixels around the hole and guessing what color should go there. But this often resulted in blurry smudges or weird patterns because the computer lost the "big picture" (the structure) and the "fine details" (the texture) while trying to shrink the image down to process it.

This paper introduces a new way to fix these holes, which the authors call "Thinking inside the Convolution." Here is how they do it, explained simply:

1. The Problem: Losing the Blueprint and the Wallpaper

Think of an image as a house.

Structure is the blueprint: the walls, the door frames, and the rooflines. It's the skeleton.
Texture is the wallpaper and paint: the grain of the wood, the pattern on the fabric, the color of the bricks.

When current computers try to fix a hole, they often use a process called "downsampling." Imagine taking a high-resolution photo and shrinking it to a tiny thumbnail to save space, then trying to blow it back up to full size.

The Issue: When you shrink the photo, you lose the fine details (texture). When you blow it back up, the computer tries to guess the details, but it often gets them wrong.
The Old Mistake: Previous methods tried to mix the blueprint and the wallpaper together and hope they helped each other. The authors found that this actually made things worse. The "blueprint" (structure) is sparse and delicate; if you try to rebuild it using the messy "wallpaper" (texture), the blueprint gets destroyed.

2. The Solution: The Blueprint Builds the Wallpaper

The authors discovered a one-way street that works much better: The Blueprint should guide the rebuilding of the Wallpaper.

They realized that even if the texture gets blurry during the shrinking process, the "skeleton" (structure) remains strong enough to tell the computer exactly where the edges and shapes should be.

The Analogy: Imagine you are rebuilding a broken mosaic. Instead of trying to guess the pattern of the tiles (texture) first, you first lay down the strong, clear outline of the picture (structure). Once the outline is set, it becomes very easy to fill in the specific colors and patterns (texture) because you know exactly where they belong.

3. Two Types of Guides: Global and Local

To make this even better, the paper uses two different types of "guides" to help rebuild the texture:

The Global Guide (The Big Picture): This looks at the whole image at once. It asks, "Where is the horizon? Where is the main wall?" It helps fix the big, sweeping textures.
The Local Guide (The Close-Up): This looks at tiny, specific spots. It asks, "What is the grain of the wood right here?" It helps fix the small, detailed textures.

The authors found that the Global Guide is best at fixing the Local details, and the Local Guide is best at fixing the Global picture. It's a bit like a team where the architect (Global) helps the bricklayer (Local) place individual bricks, and the bricklayer helps the architect understand the overall shape of the wall.

4. The "Cross-Layer Balance"

As the computer processes the image, it goes through different stages, like zooming in and out.

In the early stages (when the image is still large), the "Big Picture" guide is the most important.
In the later stages (when the image is very small and detailed), the "Close-Up" guide becomes more important.

The authors built a special module called a "Cross-Layer Balance Module." Think of this as a smart traffic controller. It watches the process and says, "Right now, we need more help from the Architect," or "Now, we need more help from the Bricklayer." It balances the two guides so that the texture is never lost, no matter how much the image is shrunk or expanded.

The Result

By using the structure to rebuild the texture (instead of mixing them up), and by balancing the "Big Picture" and "Close-Up" guides, their method creates much sharper, more realistic images.

Before: The fixed area looked blurry, with mismatched patterns or broken lines.
After: The fixed area looks like it was never damaged. The lines are straight, and the patterns match perfectly.

The paper proves this works on faces, street views, and nature scenes, showing that when you let the "skeleton" lead the way, the "skin" (texture) heals perfectly.

Technical Summary: Thinking inside the Convolution for Image Inpainting: Reconstructing Texture via Structure under Global and Local Side

Problem Statement
Image inpainting has largely advanced through encoder-decoder pipelines utilizing Convolutional Neural Networks (CNNs). While recent studies identify high-frequency structure and low-frequency texture as crucial information to be encoded, existing methods inevitably suffer from information loss during the convolutional downsampling process. Specifically, current approaches often overlook how structure and texture feature maps interact to mitigate this loss. Prior works, such as CTSDG, attempt mutual guidance between structure and texture but fail because directly fusing sparse structure maps with dense texture maps can break down the sparse structure. Furthermore, methods like ZITS attempt to preserve texture by fusing high-frequency structures but fail to address the loss of the structure feature map itself or the specific mechanism of how one modality can best guide the reconstruction of the other during downsampling. Consequently, these methods struggle to capture semantic information in deeper layers, particularly for high-resolution images (e.g., 512×512), often resulting in non-ideal upsampling outputs.

Methodology
The proposed framework, ConvInpaint-TSGL, addresses these limitations by systematically reconstructing texture feature maps via structure feature maps during the convolutional downsampling process, rather than simple fusion. The methodology is built on three core pillars:

Directional Reconstruction (Structure $\to$ Texture):
The authors establish that structure feature maps are inherently sparse, while texture maps are dense. Experimental validation (via FID and KL divergence) demonstrates that sparse structure maps can effectively reconstruct dense texture maps, whereas the inverse (reconstructing sparse structure from dense texture) leads to the breakdown of structural information. Therefore, the pipeline adopts a strategy where the structure feature map guides the reconstruction of the texture feature map.
Global and Local Normalization Strategies:
To maximize the utility of the structure map, the authors employ statistical normalization and denormalization:
- Global Normalization: Computes statistical mean and variance over the entire feature map channel, capturing global statistical properties.
- Local Normalization: Computes statistics across channels at specific spatial positions, capturing local statistical properties.
  The paper finds that global structure feature maps are best suited to reconstruct locally normalized texture maps (Global $\to$ Local), while local residual structure feature maps (derived from the difference between upsampled texture and previous layers) are best suited to reconstruct globally normalized texture maps (Local $\to$ Global).
Cross-Layer Balance Module:
Recognizing that global structure information decreases slower than local residual structure information in early stages, but surpasses it in later stages, the authors propose a Cross-Layer Balance Module. This module:
- Augments the reconstruction from global structure feature maps twice to maintain balance with local residual structure maps.
- Groups feature maps into streams based on global and local residual structures.
- Utilizes feature equalization (spatial and channel-wise) to balance these streams before feeding them into the decoder for upsampling.

Key Contributions

Reconstruction Paradigm: The paper systematically answers that structure feature maps should guide texture reconstruction, not vice versa, to alleviate information loss during downsampling.
Dual-Stream Normalization: It introduces a novel strategy of reconstructing texture under global normalization via global structure, and under local normalization via local residual structure, fusing these via element-wise addition.
Cross-Layer Balance: A new module is proposed to dynamically balance the contributions of global and local residual structures across different stages of the encoder, ensuring optimal feature preservation for the decoder.
Architecture Agnosticism: The proposed encoder strategy is designed to be a drop-in replacement for existing state-of-the-art (SOTA) encoders, demonstrating versatility.

Experimental Results
Extensive experiments were conducted on the Paris StreetView (PSV), CelebA, and Places2 datasets with irregular masks at resolutions of 256×256 and 512×512.

Quantitative Performance: The method outperforms SOTA approaches (including PENNet, HiFill, CTSDG, ZITS, and LaMa) across PSNR, SSIM, and FID metrics. Notably, substituting the encoders of other SOTA methods with the proposed module (e.g., "Ours (PENNet)") consistently yields superior results, validating the efficacy of the downsampling strategy.
Qualitative Performance: Visual results show that the method preserves semantic consistency and fine details (e.g., facial features, architectural patterns) better than competitors, which often suffer from artifacts or blurred textures, especially in high-masking ratios.
Ablation Studies: Experiments confirm that the specific combination of "Global $\to$ Local" and "Local $\to$ Global" reconstruction, followed by the cross-layer balance, achieves the best performance compared to using single-stream strategies or inverse reconstruction directions.

Significance and Claims
The paper claims that its primary significance lies in addressing the fundamental issue of information loss during convolutional downsampling in image inpainting. By shifting the paradigm from "mutual fusion" to "structure-guided reconstruction" under global and local statistical constraints, the method effectively preserves both structural integrity and textural richness. The authors assert that this approach is particularly critical for high-resolution inpainting, where conventional methods fail to capture semantic information in deeper layers due to texture feature map loss. The work provides a robust, modular solution that enhances existing encoder-decoder pipelines without requiring a complete architectural overhaul.

1. The Problem: Losing the Blueprint and the Wallpaper

2. The Solution: The Blueprint Builds the Wallpaper

3. Two Types of Guides: Global and Local

4. The "Cross-Layer Balance"

The Result

More like this