🤖 AI

Scalable GANs with Transformers

This paper introduces GAT, a scalable Generative Adversarial Network that combines transformer-based architectures with latent-space training and targeted stabilization techniques to achieve state-of-the-art, efficient image generation across a wide range of model capacities.

Original authors: Sangeek Hyun, MinKyu Lee, Jae-Pil Heo

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sangeek Hyun, MinKyu Lee, Jae-Pil Heo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot artist to paint perfect pictures of cats, dogs, and cars. For a long time, the best way to do this was using a method called a GAN (Generative Adversarial Network). Think of a GAN as a high-stakes art forgery game:

The Forger (Generator): Tries to paint a fake picture so good that no one can tell it's fake.
The Detective (Discriminator): Tries to spot the fake and catch the forger.

They play this game over and over. The Forger gets better at faking, and the Detective gets better at spotting. Eventually, the Forger creates such perfect fakes that the Detective can't tell the difference.

The Problem:
While this game works well for small projects, it's been very hard to scale up. If you try to make the Forger and Detective "smarter" by giving them more brain power (making the models bigger), the game usually breaks. The Forger gets lazy, the Detective gets confused, and the whole system crashes. Meanwhile, other types of AI (like Diffusion models) have been scaling up beautifully, creating massive, high-quality images, but they are slow because they have to paint the picture one tiny pixel at a time, step-by-step.

The Solution: GAT (Generative Adversarial Transformers)
The authors of this paper decided to try a new approach. They combined the fast "one-shot" nature of GANs with the powerful "brain architecture" of Transformers (the same tech behind modern chatbots and image generators). They call their new system GAT.

Here is how they fixed the scaling problems, explained simply:

1. The "Sketchbook" Trick (Latent Space)

Instead of asking the Forger to paint every single pixel of a 256x256 image directly (which is like painting a masterpiece with a tiny brush), they asked the Forger to paint on a compressed sketchbook first.

The Metaphor: Imagine the Forger doesn't paint the final photo. Instead, they paint a simplified, abstract map (a "latent space") that a separate, pre-trained machine (a VAE) instantly turns into a high-quality photo.
Why it helps: It's much faster and easier to learn the "map" than the whole picture, allowing the AI to scale up without getting overwhelmed.

2. The "Lazy Student" Problem & The "Noise Hierarchy"

When they made the Forger (Generator) bigger, they noticed a weird problem: The early layers of the brain were going to sleep.

The Metaphor: Imagine a factory assembly line. The first workers (early layers) are supposed to build the rough shape of the car, and the last workers add the paint and wheels. But in big GANs, the first workers just stood there doing nothing, leaving all the heavy lifting to the last few workers.
The Fix (MNG): The authors introduced a "Noise Hierarchy." They forced the early workers to practice on very blurry, noisy, distorted versions of the target image. They forced the middle workers to practice on slightly less blurry versions, and the last workers on the perfect image.
The Result: Now, every worker on the assembly line has a job. The early ones learn the big shapes, and the late ones learn the fine details. This keeps the whole "brain" active and useful.

3. The "Speed Limit" Rule (Adaptive Learning Rate)

When you make a model bigger, it naturally learns faster. If you give a giant brain the same "learning speed" (learning rate) as a small brain, the giant brain will overshoot and crash.

The Metaphor: Imagine driving a small go-kart and a massive semi-truck. If you tell both drivers to "step on the gas," the truck will fly off the road. You need to tell the truck to drive slower to stay safe.
The Fix: They created a simple rule: The bigger the model, the slower the learning speed. This keeps the training stable, no matter how huge the model gets.

The Results

By combining these tricks, the authors created GAT-XL, a massive model that:

Paints in one step: Unlike other slow methods that take 250+ steps to finish an image, GAT does it in one single step. It's like snapping a photo instead of painting a masterpiece.
Is incredibly fast: It reached the best-ever quality (FID score of 2.18) on the ImageNet dataset in just 60 training sessions (epochs). Other top methods took 4 times longer to reach similar quality.
Scales perfectly: They tested models from "Small" to "Extra-Large," and the bigger models consistently got better, proving the system works at any size.

In Summary:
The paper shows that you can have the best of both worlds: the speed of a GAN (one-step generation) and the power of modern Transformers (scaling up to huge sizes). They did this by teaching the AI to work on a "sketchbook," forcing every part of its brain to stay active using "noisy practice," and adjusting the speed limit based on the model's size. The result is a super-fast, high-quality image generator that learns efficiently.

Technical Summary: Scalable GANs with Transformers (GAT)

1. Problem Statement

While scalability has driven recent breakthroughs in generative modeling (particularly in autoregressive and diffusion families), Generative Adversarial Networks (GANs) remain underexplored in this regard. Existing attempts to scale GANs often rely on single, high-capacity models with extensive task-specific tuning rather than demonstrating genuine, systematic scalability. Furthermore, standard GAN training faces specific hurdles when scaling up:

Early Layer Underutilization: In large transformer-based generators, early layers often become inactive, contributing marginally to image synthesis.
Optimization Instability: Naively increasing model depth and width while keeping hyperparameters (specifically learning rates) identical leads to convergence failures. Larger models exhibit larger per-step output changes under the same learning rate, destabilizing the sensitive adversarial dynamics.

2. Methodology

The authors propose Generative Adversarial Transformers (GAT), a framework that combines two key ingredients for scalability: training in a compact Variational Autoencoder (VAE) latent space and utilizing purely transformer-based architectures for both the generator and discriminator.

2.1 Architecture

Latent Space: The model operates in the latent space of a pre-trained, frozen VAE (specifically SD-VAE), reducing computational burden while preserving perceptual fidelity.
Generator: A pure Vision Transformer (ViT) architecture. It takes a latent code $z$ and condition $c$ , processes them through a stack of transformer blocks (GAT blocks), and outputs images via an unpatchify layer (linear decoder). It employs adaptive normalization and LayerScale for stability.
Discriminator: A ViT backbone with a dedicated [cls] token for classification, also utilizing LayerScale.

2.2 Key Innovations for Scalability

To address the specific failure modes of scaling GANs, the authors introduce two primary mechanisms:

A. Multi-level Noise-perturbed Image Guidance (MNG)
To prevent early layers from becoming inactive, the generator is divided into $K$ stages, each producing an intermediate output $\hat{x}_k$ .

Noise Hierarchy: Each intermediate output is perturbed by Gaussian noise with a strength that decreases monotonically with depth ( $\alpha_1 < \alpha_2 < \dots < \alpha_K = 1$ ).
Supervision: All perturbed intermediate outputs are forwarded to the discriminator. This forces early layers to learn coarse structures under strong noise and later layers to refine fine details under weak noise.
Effect: This encourages a coarse-to-fine refinement trajectory, ensuring all layers contribute actively to the synthesis process without requiring an explicit image hierarchy (unlike MSG-GAN).

B. Width-Aware Learning Rate Scaling
To maintain stable training dynamics across different model sizes, the authors propose a scaling rule for the learning rate ( $\eta$ ).

Principle: As model width (channel dimension $C$ ) increases, the expected squared norm of inputs grows linearly, causing larger per-step output updates for a fixed learning rate.
Rule: The learning rate should decrease inversely with the channel dimension to keep the magnitude of functional updates constant:
$\eta_{adapt} = \eta_{base} \cdot \frac{C_{base}}{C_{model}}$
Effect: This prevents divergence in larger models and eliminates the need for manual hyperparameter tuning for each scale.

C. Additional Objectives

Representation Alignment (REPA): The discriminator is aligned with a frozen Vision Foundation Model (VFM, e.g., DINOv2) using a similarity loss on real data. This encourages the discriminator to learn semantically rich features, providing better adversarial feedback.
Adversarial Loss: The framework uses a relativistic pairing loss with two-sided gradient penalties (approximated R3GAN).

3. Key Results

The authors validate GAT across model sizes from Small (S) to Extra-Large (XL) on the ImageNet-256 dataset.

State-of-the-Art Performance: The GAT-XL/2 model achieves a FID of 2.18 on ImageNet-256 in just 60 epochs. This is a significant improvement over strong 1-NFE (one-step) baselines like MeanFlow-XL/2 (FID 3.43) and requires 4× fewer epochs than other strong baselines (e.g., GigaGAN requires 480 epochs).
Scalability: Performance improves monotonically as model size increases (S $\to$ XL). There is a strong negative correlation (-0.95) between inference cost (GFLOPs) and FID, and a similar correlation for total training compute, following a power law: $FID(C) \approx 3.52 \times 10^5 \cdot C^{-0.456}$ .
Efficiency: GAT retains the single-step inference advantage of GANs (1 NFE), offering roughly a 200× speedup in inference time compared to multi-step diffusion models (e.g., DiT) at comparable quality.
Ablation Studies:
- Removing MNG leads to inactive early layers and degraded performance.
- Using a fixed learning rate across scales causes divergence in larger models or slow convergence in smaller ones.
- The REPA objective significantly boosts performance, demonstrating that techniques from diffusion models transfer effectively to GANs.
Generalization: The method scales to higher resolutions (ImageNet-512) and different tokenizers (FLUX-e2e) and supports text-to-image generation (MS-COCO) and unconditional generation (FFHQ).

4. Significance and Claims

The paper claims that GAT successfully establishes a scalable GAN framework that rivals the performance of modern diffusion and autoregressive models while retaining the unique advantages of GANs:

Single-Step Inference: High-efficiency generation without iterative denoising.
Latent Manipulation: The ability to perform smooth latent interpolation and semantic editing, which remains coherent even at large scales.
Data Efficiency: Achieving state-of-the-art results with significantly fewer training epochs compared to diffusion baselines.

The authors position this work as a step toward "scaling GANs," demonstrating that with the right architectural choices (pure transformers) and optimization strategies (MNG and width-aware learning rates), adversarial learning can be made robust and scalable, challenging the notion that GANs are inherently limited in their scaling potential compared to diffusion models.