Technical Summary: Cross-scale Aligned Supervision for Training GANs

1. Problem Statement

The paper addresses a fundamental flaw in the interpretation of standard scale-wise adversarial supervision in modern multi-stage Generative Adversarial Networks (GANs). While such architectures are often described as performing "coarse-to-fine" hierarchical generation, the authors argue that they fail to construct a proper hierarchy.

The core issue is identified as cross-scale trajectory misalignment:

• Independent Optimization: In standard scale-wise supervision, each intermediate generator output ($x_k$) is evaluated independently by a discriminator at its specific resolution. The adversarial gradient pushes each stage toward a realistic mode at that resolution, but it does not constrain the output to represent the same sample trajectory as the final output.
• Lack of Refinement: Because the intermediate image $x_k$ is optimized for realism but not enforced as the explicit refinement target for the next stage (which proceeds via generator features $f_k$), subsequent stages may deviate from the previous output. Instead of incrementally refining the image, later stages may "rewrite" the sample entirely, converging toward a different realistic mode.
• Consequence: This results in intermediate images that are individually realistic but incoherent with one another, breaking the intended coarse-to-fine generation process. The authors demonstrate that under standard supervision, the discrepancy between intermediate and final outputs remains large, and inter-stage rewriting is substantial rather than incremental.

2. Methodology: CAT (Cross-scale Aligned Transformer)

To resolve the trajectory misalignment, the authors propose CAT, a framework that retains the benefits of scale-wise adversarial feedback while enforcing cross-scale consistency.

Core Components

1. Scale-wise Discriminator:

   • The discriminator remains scale-wise, evaluating each intermediate output ($x_k$) at its own resolution.
   • To ensure strict scale-wise discrimination without cross-scale information leakage, the authors employ a block-diagonal attention mask within a shared transformer discriminator. This ensures that tokens from scale $k$ can only attend to tokens from the same scale, preventing the discriminator from relying on cross-scale compatibility to judge realism.

2. Generator-side Consistency Regularization:

   • The primary innovation is a simple consistency loss added to the generator objective.
   • Instead of adding another realism objective, this loss aligns intermediate stage outputs ($h_k$) with the final stage output ($h_K$).
   • The loss is defined as:
     $$L_{cons} = \frac{1}{K} \sum_{k=0}^{K-1} w_k \|h_k - h_K\|_2^2$$
   • Weighting Strategy: The weights $w_k$ are scaled such that lower-resolution (coarser) stages have weaker constraints. This acknowledges that coarse outputs are inherently ambiguous (many high-resolution samples can share the same low-resolution structure), preventing the loss from imposing overly rigid point-to-point constraints on early stages while still ensuring they remain on the same sample trajectory.

3. Training Objective:
   The generator is trained to minimize:
   $$L_G = L_{adv} + \lambda_{cons} L_{cons}$$
   where $L_{adv}$ is the standard scale-wise adversarial loss and $\lambda_{cons}$ is a hyperparameter (set to 0.1 in experiments).

3. Key Contributions

• Identification of Misalignment: The paper provides a rigorous analysis demonstrating that standard scale-wise adversarial supervision does not inherently create a coarse-to-fine hierarchy. It introduces metrics (discrepancy $\delta_k$, rewrite magnitude $R_k$, and direction alignment $A_k$) to quantify the failure of standard methods to maintain a coherent sample trajectory.
• CAT Architecture: The proposal of a method that decouples the discriminator's scale-wise feedback from the generator's trajectory alignment. By keeping the discriminator scale-wise and adding generator-side consistency, the method ensures intermediate outputs support the final sample rather than optimizing disconnected side predictions.
• Efficiency and Performance: The method achieves state-of-the-art results with significantly fewer training epochs and computational resources compared to existing one-step baselines.

4. Experimental Results

Experiments were conducted on ImageNet-256 (256×256 resolution) in the latent space of SD-VAE.

• Performance: The CAT-H/2 model achieves an FID-50K of 1.56 after only 60 training epochs.
  • This outperforms strong one-step GAN baselines (e.g., GAT-XL/2 at 2.18 FID) and competitive one-step diffusion/flow models (e.g., iMF-XL/2 at 1.72 FID).
  • Notably, CAT-H/2 achieves this with ~13 times fewer training epochs than the iMF-XL/2 baseline (60 vs. 800 epochs).
• Computational Efficiency:
  • CAT-H/2 requires 1,040.2 GFLOPs for training per sample per iteration, compared to 2,297.2 GFLOPs for GAT-XL/2 and 1,306.3 GFLOPs for iMF-XL/2.
  • It achieves better FID with lower total training compute (approx. 16.7× fewer training GFLOPs than iMF-XL/2).
• Ablation Studies:
  • Removing the consistency loss ($L_{cons}$) results in a significant performance drop (e.g., FID increases from 1.93 to 3.00 for the Medium model at 40 epochs).
  • The analysis confirms that adding $L_{cons}$ reduces the discrepancy ($\delta_k$) and rewrite magnitude ($R_k$) while increasing direction alignment ($A_k$), validating that the method successfully enforces cross-scale trajectory alignment.
• Qualitative Results: Uncurated samples show diverse, high-fidelity images across various categories, with latent interpolation demonstrating smooth transitions.

5. Significance and Claims

The paper claims that generator-side cross-scale alignment is a critical, previously overlooked principle for scaling transformer-based GANs.

• Re-evaluating Hierarchical Supervision: The work challenges the assumption that scale-wise adversarial supervision alone is sufficient for hierarchical generation. It posits that without explicit alignment, multi-stage GANs suffer from trajectory misalignment, leading to incoherent refinement.
• Competitive One-Step Generation: The results suggest that when hierarchical adversarial supervision is properly organized (via consistency regularization), transformer-based GANs can serve as highly competitive one-step generative models, rivaling or surpassing diffusion and flow-based models in both quality and training efficiency.
• Scalability: The method demonstrates that high-quality generation can be achieved without scaling the discriminator capacity proportionally to the generator, provided that cross-scale consistency is enforced on the generator side.

The authors conclude that their approach offers a new direction for efficient, high-quality image synthesis, though they acknowledge limitations such as the reliance on manually specified scale hierarchies and the need for further systematic analysis of generator-discriminator capacity balance.