💻 computer science

Freqformer: Image-Demoiréing Transformer via Effective Frequency Decomposition

Freqformer is a compact, Transformer-based framework for image demoiréing that achieves state-of-the-art performance by employing effective frequency decomposition to separate patterns into distinct high- and low-frequency components, which are then processed via a dual-branch architecture and adaptively fused using a learnable Frequency Composition Transform.

Original authors: Xiaoyang Liu, Bolin Qiu, Zheng Chen, Libo Zhu, Zihan Zhou, Kai Liu, Jiezhang Cao, Yulun Zhang

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaoyang Liu, Bolin Qiu, Zheng Chen, Libo Zhu, Zihan Zhou, Kai Liu, Jiezhang Cao, Yulun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a photo of a computer screen with your phone. Instead of a clear picture, you get a weird, wavy, rainbow-colored mess. This is called a moiré pattern. It happens because the tiny dots on the screen and the tiny dots on your camera's sensor don't line up perfectly, creating a confusing interference pattern.

For a long time, computers have tried to fix this by just "guessing" what the clean picture should look like, but they often get confused because the weird waves look a lot like real details (like hair or fabric texture).

This paper introduces a new AI model called Freqformer that fixes this problem much better by using a clever trick: separating the mess into two different buckets.

The Core Idea: The "Two-Bucket" Strategy

Instead of trying to fix the whole messy image at once, Freqformer splits the problem into two distinct parts, just like separating the "shape" of an object from its "color."

The High-Frequency Bucket (The Texture): This bucket holds the sharp details and the annoying, wavy moiré patterns. Think of this as the "crunchy" part of the image. The paper says these patterns are very local—they happen in small, specific spots.
- The Strategy: The AI looks at this bucket by taking small, random "snippets" (crops) of the image. It doesn't need to see the whole picture to fix a tiny wavy line; it just needs to zoom in on that specific spot.
The Low-Frequency Bucket (The Color): This bucket holds the smooth colors and the overall lighting. The moiré patterns here look less like waves and more like a weird color tint (e.g., the whole image looks too blue or too red).
- The Strategy: Because these color issues are smooth and spread out, the AI can shrink this bucket down to a tiny size (like resizing a photo to be very small) without losing important information. It fixes the color on this tiny version, then blows it back up. This is much faster and prevents the AI from accidentally blurring out real details.

The Special Tools

To make this work, the authors built three special tools:

The Magic Separator (Frequency Decomposition): Old methods used rigid mathematical tools (like a fixed ruler) to split the image, which often left "jagged edges" or made the picture look blocky. Freqformer uses a learnable separator. Imagine a smart filter that learns exactly how to split the image perfectly without leaving any messy leftovers, ensuring the "texture" and "color" buckets are perfectly clean.
The Smart Mixer (Learnable FCT): Once the AI fixes the texture in the first bucket and the color in the second, it has to put them back together. Old methods just added them back up, which sometimes caused errors to pile up. Freqformer uses a Learnable Frequency Composition Transform. Think of this as a smart chef who tastes the two ingredients before mixing them, ensuring they blend perfectly without ruining the flavor.
The Focused Eye (SA-CA Module): Moiré patterns often look different in red, green, and blue channels. The model uses a Spatial-Aware Channel Attention module. Imagine a security guard who knows exactly which parts of the image are "suspicious" (where the moiré is) and which colors are most affected. This guard focuses the AI's attention only on the problem areas, ignoring the rest to save energy and speed.

Why It's Better

The paper claims that Freqformer is a lightweight champion.

Smaller Size: It has fewer "parameters" (the brain cells of the AI) than many other top models. It's like having a compact car that gets better gas mileage than a massive truck.
Higher Quality: It removes the rainbow waves and fixes the color tints better than previous methods, resulting in a cleaner, more realistic photo.
Efficiency: Because it shrinks the color bucket and only zooms in on texture spots, it doesn't need to do as much heavy lifting, making it suitable for high-resolution 4K images.

In short, Freqformer doesn't try to fight the moiré pattern with brute force. Instead, it intelligently sorts the image into "texture" and "color," fixes each one with the perfect tool for the job, and then blends them back together seamlessly.

Technical Summary: Freqformer

Problem Statement

Image demoiréing—the removal of moiré patterns caused by aliasing between screen subpixel layouts and camera color filter arrays—remains a challenging low-level vision task. Existing methods often fail to effectively disentangle the complex interplay between texture corruption and color distortions inherent in moiré patterns. While frequency-aware approaches (e.g., using Haar wavelets or block-based DCT) offer a promising direction, they suffer from significant limitations:

Spatial Discontinuity & Error Accumulation: Discrete transforms like DWT and DCT lack spatial smoothness, leading to pixel-level distortions and error accumulation during fixed inverse processes.
Resolution Degradation: Recursive decomposition aggressively downsamples spatial resolution, risking secondary aliasing and shift-variance issues.
Incomplete Decoupling: Traditional methods often fail to fully separate high-frequency moiré textures from low-frequency components or introduce block artifacts due to assumptions of local signal stationarity.
Computational Inefficiency: Many state-of-the-art models rely on heavy U-Net backbones (10M–50M parameters), making them computationally inefficient for ultra-high-resolution (e.g., 4K) inference.

Methodology

The authors propose Freqformer, a Transformer-based framework designed to address these limitations through targeted frequency separation and adaptive fusion.

1. Novel Frequency Decomposition

Unlike traditional DWT or DCT, Freqformer introduces a learnable, anti-aliasing frequency decomposition strategy.

Mechanism: It utilizes a recursive convolution process with a specific kernel and dilation rates to split an input moiré image ( $I_m$ ) into a high-frequency component ( $I_h$ ) and a low-frequency component ( $I_l$ ).
Separation Characteristics:
- High-Frequency ( $I_h$ ): Contains the moiré textures and fine details. It exhibits strong spatial locality.
- Low-Frequency ( $I_l$ ): Contains smooth color information but suffers from color distortions (both moiré-induced and dataset-related shifts). It is robust to scale variations.
Advantage: This method preserves spatial smoothness and avoids the resolution degradation and shift-variance issues associated with standard discrete transforms.

2. Dual-Branch Asymmetric Architecture

The framework employs a dual-branch encoder-decoder structure, each tailored to the specific characteristics of its frequency component:

High-Frequency Branch: Uses a crop-based training strategy. Since high-frequency moiré textures are locally dependent, this branch processes full-resolution inputs (or crops thereof) to preserve fine structural details without needing global context.
Low-Frequency Branch: Uses a resize-based training strategy. The low-frequency component is downsampled (e.g., to 0.1× resolution) for training and inference. This approach concentrates on correcting global color distortions while avoiding the detail loss risks associated with downsampling natural textures.
Shared Backbone: Both branches utilize a hierarchical Transformer architecture featuring Spatial-Aware Channel Attention (SA-CA) modules.

3. Key Modules

Spatial-Aware Channel Attention (SA-CA): Designed to aggregate spatial dependencies and inter-channel complementary information efficiently. Inspired by the observation that moiré artifacts vary across color channels (with the green channel often being less affected), SA-CA uses Residual Dilated Dense Blocks (RDDB) for large receptive fields and a gated Feed-Forward Network (FFN) to filter inappropriate information. It selectively emphasizes moiré-prone regions without excessive computational cost.
Learnable Frequency Composition Transform (FCT): Instead of a fixed inverse transform, Freqformer uses a learnable FCT module to adaptively fuse the outputs of the high and low-frequency branches. This module operates in the feature space (before projection to image dimensions) using a Transformer block (PostFusion) to ensure consistent and high-fidelity reconstruction, mitigating reconstruction errors that might arise from simple addition.

4. Training Strategy

The model is trained in two phases:

Independent Training: The high-frequency and low-frequency branches are trained separately using their respective crop and resize strategies.
Joint Fine-tuning: Both branches and the learnable FCT module are jointly fine-tuned to optimize feature fusion and final reconstruction.

Loss Functions: The training utilizes a combination of $L_1$ loss and perceptual loss (based on VGG16) to balance pixel accuracy and perceptual quality.

Key Contributions

Frequency Decomposition Strategy: A novel method to effectively separate moiré patterns into high-frequency (texture) and low-frequency (color) components, enabling targeted learning for each distinct artifact type.
Dual-Branch Asymmetric Design: An architecture that processes high and low frequencies independently using crop-based and resize-based strategies, respectively, coupled with a learnable FCT module for adaptive fusion.
Efficient SA-CA Module: A lightweight attention mechanism that enhances spatial and channel-wise feature interactions, allowing the model to achieve state-of-the-art performance with a compact parameter count (approx. 6M).

Experimental Results

Extensive experiments were conducted on four benchmarks: TIP2018, LCDMoire, FHDMi, and UHDM (4K resolution).

Quantitative Performance: Freqformer achieves state-of-the-art (SOTA) results across multiple metrics (PSNR, SSIM, LPIPS).
- On UHDM, it ranks first in SSIM and LPIPS and second in PSNR, closely trailing ESDNet-L.
- On FHDMi, it achieves the best scores in all metrics (PSNR: 25.26, SSIM: 0.8518, LPIPS: 0.1253).
Model Efficiency: Despite its high performance, Freqformer maintains a compact model size of 6.065M parameters, significantly smaller than competitors like MopNet (58.5M) and MBCNN (14.2M). It requires only 2.46 TFLOPS for 4K image processing, comparable to ESDNet and lower than ESDNet-L and other restoration models.
Qualitative Performance: Visual comparisons show Freqformer effectively removes moiré artifacts and color distortions while preserving fine details and original tones, outperforming generative models (e.g., OSEDiff) and other restoration baselines which often leave residual traces or suffer from color shifts.

Significance and Claims

The paper claims that Freqformer advances the field of image demoiréing by demonstrating that integrating frequency decomposition, learnable fusion, and spatial-channel attention can effectively solve the complex interplay of texture and color artifacts.

Modest Claims: The authors acknowledge that while Freqformer achieves SOTA performance, its inference speed (0.58s/image for 4K) is currently slower than purely convolution-based models like ESDNet-L (0.25s/image). They attribute this to standard attention operations being less optimized for hardware acceleration and operator fusion, identifying this as a direction for future optimization.
Core Insight: The work highlights that treating moiré removal as a holistic restoration task is insufficient; instead, decoupling the problem into frequency-specific sub-tasks with tailored training strategies yields superior results with fewer parameters.