💻 computer science

Pixel-Space Diffusion Transformers

This paper reviews Pixel-Space Diffusion Transformers (pDiTs), which bypass the limitations of latent compression by modeling raw pixels directly to enable high-fidelity, end-to-end optimization and unified multimodal systems that integrate visual generation and understanding within a single Transformer architecture.

Original authors: Renye Yan, Jikang Cheng, You Wu, Ling Liang, Wei Peng, Athanasios V. Vasilakos, Qingyu Zhao, Yu Zhang, Ehsan Adeli, Kilian M. Pohl, Guoying Zhao

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC0 1.0

Original authors: Renye Yan, Jikang Cheng, You Wu, Ling Liang, Wei Peng, Athanasios V. Vasilakos, Qingyu Zhao, Yu Zhang, Ehsan Adeli, Kilian M. Pohl, Guoying Zhao

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to paint a masterpiece. For a long time, the smartest way to do this was to give the robot a "cheat sheet." First, you'd take a real photo and squish it down into a tiny, blurry sketch (a "latent" space) to save memory. The robot would learn to paint based on that sketch, and then you'd try to un-squish it back into a real picture. It was fast and efficient, but like trying to recreate a high-definition movie from a low-resolution thumbnail, you often lost the tiny details: the sharp edges of a building, the specific font on a sign, or the texture of a cat's fur. The robot was stuck guessing what the missing pieces looked like.

Now, a new wave of researchers is asking: "What if we skip the sketch entirely?" Instead of squishing the image down, they are teaching the robot to paint directly on the full-resolution canvas, pixel by pixel. This is the world of Pixel-Space Diffusion. Think of it as the difference between trying to describe a complex recipe by only listing the main ingredients versus actually tasting every single spice as you cook. It's much harder and requires a lot more energy (computing power), but the result is a dish that tastes exactly right, with no missing flavors. This paper explores how we are finally getting good at this "direct painting" method using a powerful new type of brain architecture called a Transformer, which is great at connecting distant dots in a picture.

The Big Switch: From Sketches to Raw Pixels

This paper is a massive tour guide for a rapidly changing field called Pixel-Space Diffusion Transformers (pDiTs). The authors are essentially saying that the old way of doing things—squishing images down into a compressed "latent" space before generating them—is hitting a wall. That old method, while efficient, acts like a filter that throws away the fine-grained details we care about, like crisp text, intricate patterns, and perfect anatomical structures. Once that information is lost in the "squish," the robot can never truly get it back.

The paper argues that we are now entering a new era where we can model images directly in their raw, high-definition form. Instead of a two-step process (squish, then generate), pDiTs do it in one smooth step: they take the noisy, messy pixels and learn to turn them into a clean, perfect image. It's like going from trying to rebuild a house by looking at a blurry blueprint to actually walking through the construction site and fixing every brick in place.

The Problem with the "Old Way"

The authors explain that the previous champions of image generation, known as Latent Diffusion Models (LDMs), relied on a "compress-then-diffuse" strategy. Imagine trying to send a giant, detailed painting through a tiny mail slot. You have to fold it up tight (compression) to get it through. The problem is, when you unfold it on the other side, some of the creases are permanent, and the fine details are gone.

In technical terms, these models use a pre-trained "tokenizer" (like a VAE) to compress the image. The paper points out that this creates a "bottleneck." If the tokenizer isn't perfect at keeping the tiny details, the diffusion model can't magically invent them later. It's a fundamental limit: you can't generate what you didn't save. Furthermore, because the compression and the generation are trained separately, they often disagree on what is important, leading to a mismatch that limits how good the final image can be.

The New Hero: Pixel-Space Diffusion Transformers

So, what is the solution? The paper introduces pDiTs. These are models that skip the compression step entirely. They look at the raw pixels of an image and learn to generate them directly.

However, the authors are careful to note that this isn't just "going back to the old days." Early attempts at pixel-based generation were too slow and messy because the computers couldn't handle the sheer amount of data. The new pDiTs are different because they use Transformers—a type of AI architecture that is incredibly good at understanding relationships between different parts of an image, no matter how far apart they are.

The paper breaks down how these new models solve the massive challenges of working with raw pixels:

The "Too Much Data" Problem: Looking at every single pixel is computationally expensive. The paper describes clever tricks like using "large patches" (grouping pixels together) to speed things up, or using "hierarchical" structures that look at the big picture first and then zoom in on the details.
The "Noise" Problem: In the early stages of generating an image, the picture is just static noise. The paper explains that these new models use better math (like "Flow Matching") to navigate this noise more efficiently, predicting the final clean image directly rather than guessing the noise step-by-step.
The "One Brain for Everything" Problem: Perhaps the most exciting part of the paper is the idea of Unified Multimodal Modeling. Traditionally, AI models for understanding images (like recognizing a cat) and models for generating images (like drawing a cat) were separate. pDiTs suggest we can put text, images, and instructions all into the same "token space." Imagine a single brain that can read a prompt, understand the image, and draw the result all at once, without needing to translate between different languages.

What the Paper Actually Finds (and What It Doesn't)

The authors review a wide range of recent methods and suggest that while pixel-space diffusion is computationally heavier, it offers a higher ceiling for quality. They argue that for tasks requiring extreme fidelity—like rendering readable text, precise medical scans, or complex 3D scenes—the pixel-space approach is superior because it doesn't lose information to a compression filter.

However, the paper is very clear about what this does not mean:

It does not mean Latent Diffusion is dead. The authors explicitly state that latent models are still better for many general tasks because they are faster and cheaper. The pixel-space approach is a trade-off: you pay more in computing power to get better details.
It is not a magic bullet. The paper suggests that simply switching to pixels isn't enough; you need specific architectural designs (like frequency decoupling, where the model handles smooth colors and sharp edges separately) to make it work well.
It is not yet "solved." The paper highlights that challenges remain, particularly in balancing the cost of computation with the quality of the image, and in preventing the model from "forgetting" how to generate images when it is also trying to learn to understand them.

The Future: A Unified Vision

The paper concludes by painting a picture of the future where these models become the foundation for Unified Vision Foundation Models. Instead of having one AI for understanding and another for creating, we might have a single system that can do both seamlessly. It could look at a photo, understand the story, edit the lighting, and generate a new version with perfect text and textures, all in one go.

The authors suggest that while we are still figuring out the best ways to manage the heavy computing costs, the direction is clear: moving away from fixed, lossy compression and toward direct, end-to-end modeling of the raw visual world. It's a shift from "guessing the details" to "seeing the details," promising a future where AI-generated images are not just impressive, but indistinguishable from reality in their finest grain.

Technical Summary: Pixel-Space Diffusion Transformers (pDiT)

1. Problem Statement

The paper identifies a fundamental limitation in the current dominant paradigm of Latent Diffusion Models (LDMs). While LDMs achieve efficiency by compressing images into a low-dimensional latent space via a pre-trained Variational Autoencoder (VAE) before performing denoising, this two-stage approach introduces an irreversible information bottleneck.

Lossy Compression: Fixed visual tokenizers discard high-frequency textures, sharp boundaries, and minute local structures during the compression phase. This information cannot be recovered by the downstream diffusion model, capping the maximum visual fidelity.
Objective Mismatch: The reconstruction objective of the VAE (minimizing pixel reconstruction error) often conflicts with the generative objective of the diffusion model. This separation prevents end-to-end optimization and creates a mismatch between representation learning and generation.
Granularity Schism in Unified Models: In unified multimodal models, visual understanding requires abstract, invariant representations, while visual generation requires absolute sensitivity to spatial layouts and fine-grained details. Traditional pipelines force a trade-off by using separate modules (e.g., VAEs for generation, CLIP/DINO for understanding), preventing a truly shared representation space.

Consequently, the paper argues for a return to Pixel-Space Diffusion, which models raw pixels directly to eliminate the VAE bottleneck, enabling end-to-end optimization of both representation and generation. However, this approach historically faced prohibitive computational costs and optimization difficulties in high-dimensional spaces.

2. Methodology and Architectural Evolution

The paper reviews the resurgence of pixel-space diffusion, driven by Pixel-Space Diffusion Transformers (pDiTs). Unlike early pixel models that used convolutional U-Nets, modern pDiTs leverage scalable Transformer architectures and advanced training strategies to overcome high-dimensional challenges.

A. Theoretical Foundations

The paper contrasts two primary mathematical frameworks used in pixel-space modeling:

Stochastic Denoising (DDPM): Formulates generation as a discrete-time Markov reverse process, typically predicting noise ( $\epsilon$ ) or the clean image ( $x_0$ ).
Continuous Flow (Flow Matching & Rectified Flow): Reformulates generation as a deterministic continuous transport problem, learning a velocity field that moves samples from a noise distribution to the data distribution. This approach, particularly Rectified Flow, straightens trajectories, reducing integration error and improving sampling stability.

B. Architectural Taxonomy

To address the quadratic complexity of global self-attention on high-resolution images and the conflict between global semantics and local details, the paper categorizes modern pDiT architectures into seven types:

Single-Stream Large-Patch Transformers: (e.g., JiT) Use large patch sizes to reduce token counts and directly predict clean images ( $x_0$ -prediction), simplifying the training objective.
Hierarchical and Hourglass pDiTs: (e.g., HDiT) Organize tokens across multiple resolutions, aggregating global interactions at low resolution and restoring details via upsampling paths.
Global–Local Decoupled Architectures: (e.g., DiP, PixelDiT) Explicitly separate global semantic modeling (large patches) from local detail recovery (small patches or lightweight modules).
Frequency-Decoupled Architectures: (e.g., DeCo) Separate low-frequency semantic modeling from high-frequency texture recovery, often using spectral filtering or distinct loss weights.
Implicit Neural Field Decoding: (e.g., PixNerd) Interpret patch tokens as conditional representations for local implicit neural fields, allowing continuous coordinate-based decoding of pixels.
Cross-Scale Semantic Anchoring: (e.g., HyperDiT) Connect coarse semantic tokens with fine-grained detail tokens to provide global guidance for local reconstruction.
Unified Multimodal Shared-Token Architectures: (e.g., HiDream-O1-Image) Map text tokens, task conditions, and raw pixel patches into a single shared token space, allowing a unified Transformer to handle generation, editing, and understanding simultaneously.

C. Training Strategies

The paper highlights key algorithmic shifts enabling pixel-space viability:

Direct Clean-Image Prediction ( $x_0$ -prediction): Anchors optimization directly on the data manifold, accelerating convergence compared to noise prediction.
Representation Alignment: Techniques like REPA (Representation Alignment) and SGA (Spatial Gram Alignment) use frozen Vision Foundation Models (VFMs) to provide semantic priors, though the paper cautions against rigid token-wise alignment that may suppress high-frequency details.
Structured Loss Weighting: Dynamically adjusting loss weights for different frequency bands or spatial regions to balance global structure and local fidelity.

3. Key Contributions

This paper serves as the first systematic survey dedicated to Pixel-Space Diffusion Transformers. Its primary contributions include:

Comprehensive Taxonomy: A structured classification of modern pDiT architectures based on how they handle token granularity, global-local coordination, and frequency decoupling.
Theoretical Deconstruction: A detailed analysis of the mathematical foundations (DDPM vs. Flow Matching) and the specific challenges of high-dimensional pixel modeling (quadratic complexity, optimization difficulty, frequency interference).
Unified Modeling Framework: An exploration of how pixel-space diffusion enables a "native" shared token space for multimodal models, bridging the gap between visual understanding and generation without fixed VAE bottlenecks.
Application Landscape: A review of pDiT applications beyond standard image generation, including video, 3D reconstruction, and medical imaging, where pixel-level fidelity is critical.
Critical Discussion: A balanced discussion on the Compute-Fidelity Trade-off, arguing that pDiT is not a universal replacement for LDMs but a specialized solution for tasks requiring extreme fidelity and end-to-end optimization.

4. Results and Applications

The paper synthesizes findings from existing literature (e.g., Simple Diffusion, JiT, PixelDiT, HiDream-O1) to demonstrate that:

High-Fidelity Generation: pDiTs can generate images with superior text rendering, sharp edges, and complex textures compared to LDMs, as they are not constrained by VAE compression artifacts.
Editing and Restoration: Direct pixel modeling allows for more precise local editing and inpainting, preserving unedited regions without propagation errors from latent decoding.
3D and Volumetric Data: The paradigm extends to 3D Gaussian splats (PixGS) and medical volumetric data (PRDiT, PPDM), where preserving anatomical plausibility and subtle structural details is paramount.
Unified Capabilities: Models like HiDream-O1-Image demonstrate that text-to-image generation, instruction-based editing, and subject personalization can be unified within a single architecture, sharing visual representations across tasks.

5. Significance and Future Outlook

The paper claims that the significance of pDiT lies not merely in improving image quality, but in redefining the foundation of visual generation. By removing the fixed VAE interface, pDiT allows for:

End-to-End Optimization: Jointly optimizing representation learning and generation objectives.
Unified Multimodal Learning: Creating a shared token space where text, conditions, and pixels interact directly, facilitating tasks that require both semantic reasoning and pixel-level precision (e.g., OCR, complex editing).

However, the authors remain modest about the immediate dominance of pDiT. They emphasize that the computational cost of high-resolution pixel modeling remains a significant barrier. The future of the field is not a simple replacement of LDMs, but the development of scalable structured computation mechanisms (dynamic token granularity, adaptive attention, non-invasive guidance) that balance computational efficiency with the high-fidelity benefits of pixel-space modeling. The ultimate goal is a new generation of vision foundation models that can simultaneously support high-fidelity generation, fine-grained control, and cross-modal understanding.