Technical Summary: Efficient Image Synthesis with Sphere Latent Encoder

Problem Statement

Few-step image generation has advanced significantly through consistency and meanflow-based methods, yet these approaches often suffer from training instability, sensitivity to hyperparameters, and inherent conflicts in optimization objectives. An alternative, the Sphere Encoder [40], achieves high-quality generation in few steps by projecting latent representations onto a hypersphere. However, the Sphere Encoder framework faces two critical limitations:

1. Computational Inefficiency: The generation process requires repeated transitions between pixel space and latent space (decoding to an image, then re-encoding) for every sampling step. This introduces substantial computational overhead during both training and inference.
2. Objective Conflict: The framework jointly optimizes reconstruction and generation within a single encoder-decoder architecture. Prior research suggests these tasks favor different representations; for instance, lower noise levels improve reconstruction but degrade generative performance by reducing hyperspherical coverage. This tension forces the use of larger architectures to maintain sample quality.

Methodology

The authors propose Sphere Latent Encoder, a framework that decouples reconstruction from generation to eliminate iterative pixel-space operations. The core innovation is performing the entire generative process within a fixed spherical latent space.

Architecture

The method consists of three primary components:

1. Fixed Image Tokenizer: A pretrained representation autoencoder (RAE) [43] is used as a fixed encoder ($E$) and decoder ($D$). The encoder, based on DINOv2, maps images to latent representations, while the decoder reconstructs images. This component is frozen during training.
2. Latent Denoising Model: A separate transformer-based denoising network ($G$, utilizing the SiT architecture [25]) operates exclusively in the latent space. It takes a noisy spherical latent as input and predicts the clean latent.
3. Spherical Projection: Latent representations are projected onto a hypersphere using an RMSNorm-based spherification function ($F$), ensuring a uniform latent structure.

Training Objective

Unlike Sphere Encoder, which computes losses in pixel space, Sphere Latent Encoder shifts all supervision to the latent space to improve efficiency. The training involves two noisy spherical latents, $v_{NOISY}$ and $v_{noisy}$, generated by adding Gaussian noise at different scales.

• Reconstruction Loss ($L_{recon}$): Combines an $\ell_1$ loss and cosine similarity loss between the predicted clean latent $G(v_{noisy})$ and the ground truth clean latent $z$. This aligns the denoised latent with the clean representation.
• Consistency Loss ($L_{cons}$): Enforces consistency between predictions from different noise levels ($v_{NOISY}$ and $v_{noisy}$). The prediction from the lower-noise latent is treated as a fixed target (via stop-gradient) to encourage local smoothness on the hypersphere.
• Removal of Latent Consistency Loss: The authors explicitly remove the latent consistency loss used in Sphere Encoder (which re-encodes decoded images), finding that it adds computational overhead without improving performance in their decoupled setup.

Sampling Procedure

Sampling occurs entirely in latent space (Algorithm 1):

1. A latent is sampled from a Gaussian distribution.
2. It is projected onto the hypersphere.
3. The denoising model $G$ refines the latent using classifier-free guidance.
4. The refined latent is re-projected, perturbed with noise, and used as input for the next step.
5. This iterative refinement continues for $T$ steps.
6. The decoder is invoked only once at the end to map the final latent to pixels.

The authors also propose a modified noise sampling strategy using a logit-normal distribution to better suit few-step sampling, differing from the uniform sampling in Sphere Encoder.

Key Contributions

1. Sphere Latent Encoder Framework: A generative framework operating entirely in spherical latent space that eliminates repeated pixel-latent transitions. This reduces inference cost by approximately 85% (roughly 6.5× fewer FLOPs) compared to Sphere Encoder at the same sampling steps.
2. Decoupled Design: By leveraging a fixed pretrained autoencoder and a separate denoising model, the framework allows reconstruction and generation to specialize independently, resolving the objective conflict inherent in joint optimization.
3. Efficient Optimization: The method avoids first-order approximation objectives (e.g., Jacobian-vector products) required by some few-step flow methods, instead learning a direct latent denoising process on a structured spherical manifold.

Experimental Results

The method was evaluated on Animal-Faces, Oxford-Flowers, and ImageNet-1K (256×256).

• Performance vs. Sphere Encoder: On Animal-Faces and Oxford-Flowers, the proposed method significantly outperformed Sphere Encoder in both generation quality (FID) and computational cost (GFLOPs) across 2, 4, and 6 sampling steps. For example, on Animal-Faces with 4 steps, the FID improved from 18.23 (Sphere Encoder) to 6.89 (Ours), while GFLOPs dropped from 4554 to 390.
• ImageNet-1K Results: With 4 sampling steps (8 NFE with guidance), the method achieved an FID of 2.25, a substantial improvement over Sphere Encoder's 4.02 under the same budget. It also outperformed several recent few-step baselines (e.g., MeanFlow, $\alpha$-Flow) and approached the performance of multi-step diffusion models (e.g., DiT-XL/2) that require 250+ steps, but with significantly fewer function evaluations.
• Ablation Studies:
  • Noise Distribution: Log-normal sampling outperformed uniform sampling.
  • Losses: Combining reconstruction and consistency losses yielded the best results; adding the latent consistency loss degraded performance.
  • Autoencoder: The choice of tokenizer was critical. RAE [43] (with DINOv2 features and $d=768$) significantly outperformed FLUX-VAE and GAE, highlighting the need for strong semantic representations and high-dimensional latent spaces.

Significance and Claims

The paper claims that Sphere Latent Encoder offers a simpler and more efficient pipeline for few-step image generation. By removing the dependence on iterative pixel-space refinement, the method achieves:

• Higher Efficiency: Drastic reduction in FLOPs and inference time compared to Sphere Encoder.
• Better Quality: Improved sample quality (lower FID) due to the decoupling of reconstruction and generation, allowing the denoising model to focus solely on generation without compromising reconstruction fidelity.
• Stability: A training process that avoids the instability associated with Jacobian-vector products and the conflicting objectives of joint optimization.

The authors position this work as a step toward scalable, efficient generative modeling that leverages the benefits of spherical latent spaces without the computational penalties of previous implementations. They acknowledge limitations, noting the current focus on class-conditional generation and the dependency on the quality of the pretrained autoencoder, which may constrain one-step sampling capabilities.