Technical Summary: RaPD (Resolution-Agnostic Pixel Diffusion)

1. Problem Statement

Natural images are inherently continuous, yet modern generative models predominantly operate on discrete spatial grids (either in VAE latent spaces or pixel spaces). This discretization imposes a fundamental limitation: generating images at arbitrary, unseen resolutions requires increasing the number of tokens and computational cost, making native high-resolution generation challenging.

Existing approaches to arbitrary-resolution generation typically introduce continuity only at the decoding stage.

• GANs use coordinate-queried MLPs but suffer from limited generation capacity.
• Latent Diffusion Models (LDMs) employ flexible decoders, but the diffusion process itself remains confined to a discrete, compressed latent space, inheriting the VAE bottleneck and tying semantics to discrete content.
• Pixel Diffusion Models operate directly on pixels but still generate on fixed discrete grids.
• Neural Image Fields (NIFs) like LIIF and CLIF enable resolution-free rendering but are trained primarily for interpolation and reconstruction. Their latents are optimized for local appearance cues and lack the semantic structure required for high-quality generative modeling.

The core problem addressed is the reconstruction–generation gap: existing continuous representations are excellent for super-resolution reconstruction but fail when used directly for text-to-image generation. The paper asks whether image generation can be formulated over "continuous pixels" rather than discrete grids or compressed proxies.

2. Methodology

The authors propose RaPD (Resolution-agnostic Pixel Diffusion), a framework that performs diffusion directly within a continuous Neural Image Field (NIF) latent space. The framework consists of two primary stages and three key technical components.

2.1 Basic Framework

RaPD adopts a two-stage design:

1. Stage 1 (NIF Learning): An encoder $E_\phi$ maps an input image to a dense, high-resolution NIF latent $z$ without spatial downsampling. A Coordinate-Queried Attention Renderer (CQAR) renders this latent at arbitrary target coordinates.
2. Stage 2 (Diffusion Training): The encoder and renderer are frozen. A DiT-style denoiser is trained to predict the flow-matching velocity on normalized NIF latents, conditioned on text embeddings.

At inference, a single denoised latent is rendered at any target resolution by changing only the query coordinate grid, keeping the diffusion cost fixed regardless of output size.

2.2 Semantic Representation Guidance (SRG)

To bridge the gap between reconstruction-oriented NIFs and generative modeling, RaPD introduces SRG, which operates at both the representation and diffusion-training levels:

• Semantic Distillation: Since standard NIF latents lack semantic structure, RaPD injects semantics from a frozen vision foundation model (DINOv2 ViT-B/14) into the dense NIF latent during Stage 1.
  • The NIF latent $z$ is projected to match the DINOv2 patch grid.
  • An alignment loss comprising a cosine loss (enforcing per-position semantic agreement) and a distance-matrix similarity loss (preserving pairwise relational geometry) is optimized.
  • This acts as formative supervision, shaping the latent into a generation-aware representation without sacrificing coordinate-rendering capabilities.
• Latent-Density-Aware Timestep Shift: Standard flow matching schedules are designed for compressed VAE latents. RaPD's NIF latents are dense and high-resolution. To compensate for this increased information density, the authors introduce a larger timestep shift factor ($s=32$) in the flow matching schedule. This prioritizes heavy noise stages (small $t$), which are critical for generating coherent content in dense latent spaces.

2.3 Coordinate-Queried Attention Renderer (CQAR)

Existing implicit decoders (e.g., LIIF, CLIF) use pointwise MLPs or shallow CNNs that process local features independently. These fail to utilize the global semantic structure injected by SRG.

RaPD replaces these with CQAR, which performs attention over the queried output coordinates rather than the input latent grid.

• Token Construction: For each target coordinate, a token is constructed from the nearest-neighbor latent feature, absolute coordinate, relative offset, and cell size.
• Windowed Attention: These tokens are processed by Swin-style windowed self-attention blocks. This enables inter-pixel semantic reasoning, allowing the renderer to reason globally across queried pixels while maintaining scalable computational cost.

3. Key Contributions

1. RaPD Framework: A resolution-agnostic pixel diffusion framework that generates continuous image signals directly in an NIF latent space, decoupling the diffusion cost from the output resolution.
2. Semantic Representation Guidance (SRG): A strategy to transform reconstruction-oriented continuous latents into generation-aware representations via semantic distillation and adapted flow matching schedules.
3. Coordinate-Queried Attention Renderer (CQAR): A novel decoder that replaces local implicit operators with coordinate-conditioned, scale-aware attention, enabling spatially coherent arbitrary-resolution rendering.

4. Experimental Results

The paper evaluates RaPD against state-of-the-art latent diffusion models (e.g., SD3, FLUX.1-dev) and pixel diffusion models (e.g., PixelGen, DeCo, PixNerd).

• Generation Quality: On GenEval, RaPD achieves an overall score of 0.85, outperforming PixelGen (0.79) and DeCo (0.83), with significant improvements in compositional categories (two-object, counting, color). On DPG-Bench, RaPD scores 81.6, surpassing pixel-diffusion baselines.
• Arbitrary-Resolution Generation: RaPD demonstrates strong generalization to out-of-distribution resolutions (up to $4096^2$) despite being trained with supervision up to $512^2$. The generated images maintain coherent global structure and fine details at high resolutions.
• Efficiency: A critical advantage is computational scalability. While fixed-grid pixel diffusion models (PixelGen, DeCo) exhibit quadratic growth in inference time with resolution, RaPD keeps the denoising cost nearly fixed.
  • At $4096^2$, PixelGen and DeCo take >61 seconds.
  • RaPD takes only 2.13 seconds for the same resolution, as the diffusion cost is independent of the output size.

5. Significance and Limitations

Significance:
The paper claims that RaPD fundamentally shifts the paradigm of image generation by formulating the process over "continuous pixels." It successfully demonstrates that high-quality, resolution-agnostic generation is possible without the information bottleneck of VAEs or the quadratic cost scaling of discrete pixel diffusion. By treating the latent space as a continuous signal rather than a discrete grid, RaPD achieves a unique combination of generation quality and inference efficiency at high resolutions.

Limitations:
The authors acknowledge several constraints:

1. Data Scale: The model is trained on ~36 million images (BLIP3o), which is substantially fewer than commercial-scale models (e.g., SD3, FLUX), potentially limiting absolute generation quality.
2. Text Encoder: The use of a frozen, relatively compact Qwen3-1.7B text encoder may constrain the understanding of complex prompts compared to systems using larger language models.
3. Inference Optimization: The current implementation does not yet utilize advanced optimizations like FlashAttention, fused kernels, or FP8 quantization, meaning reported latencies are conservative upper bounds.

The paper concludes that future work could focus on scaling datasets/models, extending the framework to controllable generation/editing, and implementing hardware-aware inference acceleration.