🤖 machine learning

FREPix: Frequency-Heterogeneous Flow Matching for Pixel-Space Image Generation

FREPix introduces a frequency-heterogeneous flow matching framework for pixel-space image generation that explicitly decomposes and trains low- and high-frequency components separately, achieving competitive FID scores on ImageNet while enabling efficient coarse-to-fine generation.

Original authors: Mingfeng Lin, Jiakun Chen, Liang Han, Liqiang Nie

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingfeng Lin, Jiakun Chen, Liang Han, Liqiang Nie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to paint a realistic portrait of a cat.

The Old Way (Latent Space)
Most modern AI painters don't work directly on the canvas. Instead, they first shrink the cat down into a tiny, compressed "blueprint" (a latent space), do all their thinking and sketching on that small blueprint, and then try to blow it back up to full size at the end.

The Problem: If the blueprint is too small or blurry, the final painting will look fuzzy or weird, no matter how good the AI is. It's like trying to reconstruct a high-definition movie from a low-resolution thumbnail.

The New Problem (Pixel-Space)
Some researchers decided to skip the blueprint and paint directly on the full-size canvas (pixel-space). This avoids the "fuzzy blueprint" problem. However, painting a whole canvas at once is incredibly hard. It's like trying to paint the cat's fur, the shape of its ears, the background, and the lighting all in one giant, chaotic splash of paint. The AI gets confused about what to do first: the big shape or the tiny details?

The FREPix Solution: The "Frequency" Chef
The authors of this paper, FREPix, realized that natural images (like photos of cats) aren't just one big mess. They are made of two very different things that behave differently:

Low Frequencies (The Big Picture): These are the smooth, slow-changing parts. The overall shape of the cat, its color, and where it is sitting. This is the "skeleton" or the "coarse sketch."
High Frequencies (The Tiny Details): These are the fast-changing parts. The individual whiskers, the texture of the fur, and the sharp edges. This is the "fine detail."

The paper argues that current AI tries to learn the skeleton and the whiskers at the exact same time, using the same rules. FREPix says, "No, let's treat them differently."

The FREPix Recipe (How it Works)

Think of FREPix as a two-person kitchen team with a very specific workflow:

1. The "Structure" Chef (Low-Frequency)
First, the AI focuses only on the big picture. It ignores the whiskers and fur texture completely. It draws a smooth, blurry outline of the cat.

Analogy: Imagine a sculptor first carving the rough shape of a statue out of a block of clay. They don't worry about the eyes or the hair yet; they just get the pose right.

2. The "Detail" Chef (High-Frequency)
Once the "Structure Chef" has finished the rough outline, the "Detail Chef" takes over. They look at that rough outline and only add the fine details on top of it.

Analogy: Now that the statue has the right shape, a second artist comes in to carve the eyes, the wrinkles, and the hair texture. They don't try to change the pose; they just refine what's already there.

3. The "Different Speeds" Rule
The paper also noticed that these two chefs work at different speeds.

The Structure (the big shape) is easy to figure out early on. The AI should learn this fast.
The Details (the whiskers) are hard to figure out until the shape is already there. The AI should learn this slower and later in the process.
Analogy: It's like building a house. You pour the foundation and frame the walls (fast, early). You don't start painting the wallpaper or installing the doorknobs until the house is actually standing. FREPix forces the AI to follow this order explicitly, rather than hoping it figures it out by accident.

Why is this a big deal?

The paper claims that by separating the "big picture" from the "tiny details" and giving them their own special paths and rules, the AI becomes much better at painting directly on the full canvas without needing a compressed blueprint.

The Results:

Better Quality: On a standard test (ImageNet), their method produced images that were very sharp and realistic, competing with the best methods that use the "blueprint" approach.
Faster Learning: The AI learned to paint well much faster than other pixel-based methods. It didn't need to train for as long to get good results.
Efficiency: It created high-quality images using less computer power than other direct-painting methods.

In a Nutshell:
FREPix is a new way for AI to generate images. Instead of trying to learn the whole picture at once, or shrinking the picture down to a tiny blueprint, it breaks the image into "Big Shapes" and "Tiny Details." It teaches the AI to draw the shapes first, and then add the details second, using different rules for each. This makes the AI smarter, faster, and able to create high-quality images directly from scratch.

Technical Summary: FREPix

Problem Statement

While latent-space diffusion models have become the dominant paradigm for image generation due to their efficiency, they introduce a structural bottleneck: generation quality is strictly bounded by the reconstruction fidelity of the Variational Autoencoders (VAEs) used to compress the data. This has renewed interest in pixel-space generation, where models operate directly on raw images to avoid representation artifacts. However, existing pixel-space methods largely treat image generation as a frequency-homogeneous process. They model the entire image with a single state and a shared interpolation path, implicitly relying on the network to discover the separation between global structures (low-frequency) and fine details (high-frequency). This approach overlooks the distinct statistical properties, energy distributions, and learning dynamics of these frequency components, imposing an unnecessarily restrictive inductive bias on the generation process.

Methodology: FREPix

The authors propose FREPix (FREquency-heterogeneous flow matching), a framework that explicitly decomposes pixel-space image generation into low- and high-frequency components throughout the entire pipeline. The methodology consists of four key innovations:

1. Frequency-Decomposed State Space and Heterogeneous Interpolation

Instead of treating the image $x_t$ as a single homogeneous state, FREPix reparameterizes the state using an orthonormal Discrete Wavelet Transform (DWT), $W$ . The image is decomposed into a low-frequency sub-state $l_t$ (structure) and a high-frequency sub-state $h_t$ (detail).
Crucially, the framework assigns heterogeneous interpolation paths to these sub-states. Rather than applying a single shared schedule to all frequencies, FREPix defines separate strictly increasing schedules, $g_l(t)$ and $g_h(t)$ , for the low- and high-frequency components:
$l_t = g_l(t) l + (1 - g_l(t)) \epsilon_l$
$h_t = g_h(t) h + (1 - g_h(t)) \epsilon_h$
This allows the transport dynamics to reflect the natural asymmetry of image formation, where global structure is established before fine details are refined. The authors prove that this heterogeneous interpolation is a principled extension of standard flow matching, preserving the validity of the continuity equation and learnability.

2. Factorized Generative Modeling via Explicit Architectural Decoupling

To prevent the decomposed transport from collapsing back into a unified prediction problem, FREPix employs an explicitly decoupled architecture. The generation target is factorized into two specialized modules:

Structure Predictor ( $f_\phi$ ): A Diffusion Transformer (DiT) that takes the noisy low-frequency sub-state $l_t$ and predicts the clean low-frequency component $\hat{l}$ . This module captures long-range dependencies and global semantics.
Detail Refiner ( $g_\phi$ ): A lightweight, attention-free decoder (adapted from DeCo) that takes the noisy high-frequency sub-state $h_t$ and the predicted low-frequency structure $\hat{l}$ as a condition to predict the clean high-frequency component $\hat{h}$ .

This design turns "coarse-to-fine" generation from an implicit architectural tendency into an explicit design principle, enforcing a conditional dependency where high-frequency details are generated based on the established low-frequency structure.

3. Frequency-Aligned Flow Matching Objective

The training objective is aligned with the decomposed structure. The framework converts the network's clean-image prediction ( $\hat{x}$ ) into an induced velocity field based on the heterogeneous interpolation path. A frequency-aware loss is then defined, applying time-dependent weights $\lambda_l(t)$ and $\lambda_h(t)$ to the low- and high-frequency velocity errors respectively:
$\mathcal{L}_{FA} = \mathbb{E}_{t,x,\epsilon} \left[ \lambda_l(t) \| W_l(v_\theta - v_t) \|^2 + \lambda_h(t) \| W_h(v_\theta - v_t) \|^2 \right]$
This allows the model to rebalance optimization difficulty over time (e.g., prioritizing low-frequency structure early in the trajectory and high-frequency refinement later) without altering the target marginal velocity field.

4. Auxiliary Losses

The final training objective combines the frequency-aligned flow matching loss with Representation Alignment (REPA) and LPIPS perceptual losses to ensure feature consistency and local pattern recovery.

Key Results

FREPix was evaluated on ImageNet class-to-image generation at 256×256 and 512×512 resolutions.

Performance: At 256×256, FREPix-XL (674M parameters) achieves a 1.91 FID and 295.6 IS after 320 epochs. At 512×512, it reaches 2.38 FID and 334.7 IS. These results are competitive with state-of-the-art pixel-space models (e.g., DeCo, PixelDiT) and approach the performance of latent-space models, despite operating directly on pixels.
Training Efficiency: The model demonstrates strong performance early in training. After only 80 epochs, it achieves a 2.29 FID, outperforming other pixel-space baselines at similar stages.
Sampling Efficiency: FREPix excels in the low-NFE (Number of Function Evaluations) regime. Under 25-step Euler sampling, it achieves a 2.59 FID, significantly outperforming DeCo (3.30) and PixNerd (3.28).
Computational Cost: FREPix-XL operates at 230 GFLOPs per forward pass, which is lower than most recent pixel-space methods (e.g., PixelFlow at 5818 GFLOPs) and competitive with latent-space models (e.g., DiT at 238 GFLOPs), demonstrating an improved quality-computation trade-off.

Significance and Claims

The paper claims that explicitly modeling frequency heterogeneity provides a stronger inductive bias for end-to-end pixel-space generation. By making the distinct roles of low- and high-frequency components explicit in the state space, transport path, architecture, and objective, FREPix transforms the "coarse-to-fine" generation process from an emergent behavior into a hard design principle.

The authors assert that this approach:

Mitigates Frequency Entanglement: By decoupling prediction targets, the model reduces the effective dimensionality and statistical complexity each branch must learn, as supported by their generalization bounds.
Improves Sampling Efficiency: The heterogeneous transport design is particularly beneficial when sampling with few steps, suggesting that aligning the interpolation path with the natural recovery order of image frequencies accelerates convergence.
Offers a Viable Alternative to Latent Space: FREPix demonstrates that pixel-space generation can achieve competitive quality without the representational bottleneck of VAEs, while maintaining computational efficiency comparable to latent-space methods.

The paper concludes that frequency heterogeneity is a useful perspective for designing future pixel-space generative models, though it acknowledges limitations regarding the specific family of interpolation schedules explored and the fixed nature of the wavelet decomposition used.