💻 computer science

Adapting Self-Supervised Representations as a Latent Space for Efficient Generation

The paper introduces RepTok, an efficient generative framework that represents images as single continuous latent tokens derived from fine-tuned self-supervised vision transformers, enabling competitive class-conditional and text-to-image synthesis with significantly reduced training costs and spatial redundancy.

Original authors: Ming Gui, Johannes Schusterbauer, Timy Phan, Felix Krause, Josh Susskind, Miguel Angel Bautista, Björn Ommer

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ming Gui, Johannes Schusterbauer, Timy Phan, Felix Krause, Josh Susskind, Miguel Angel Bautista, Björn Ommer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to paint.

In the past, the standard way to do this was to give the robot a massive, high-resolution canvas and ask it to figure out every single pixel (every tiny dot of color) one by one. This is like trying to build a house by laying every single brick individually while standing on a ladder. It works, but it takes forever, requires a huge team, and uses a massive amount of energy.

RepTok is a new, clever shortcut invented by researchers that changes the game. Instead of making the robot paint every brick, they teach it to understand the essence of the house first, then just fill in the details.

Here is how RepTok works, broken down into simple analogies:

1. The "Smart Brain" (The Pre-trained Encoder)

Imagine you have a genius art critic who has spent their whole life looking at millions of paintings. They know what a "cat," a "sunset," or a "chair" looks like. They can describe the idea of a cat perfectly, but if you ask them to draw one from memory, the details might be a bit fuzzy. They know the concept, but not the specific whiskers or fur texture.

In the paper, this "genius critic" is a Self-Supervised Learning (SSL) model. It's a pre-trained AI that already understands the world very well.

2. The "One-Word Summary" (The Single Token)

Usually, to describe a complex image to a robot, you need a long list of instructions (thousands of words or tokens). This is like giving the robot a 100-page manual to build a chair.

RepTok asks a bold question: Can we describe an entire image with just ONE word?

The researchers found that if they take that "genius critic" and give them a tiny bit of extra training, the critic can compress the entire image into a single, continuous "thought" or "token."

The Magic Trick: They don't retrain the whole critic. They just tweak the "summary button" (called the [cls] token) so it remembers a few extra details about the specific picture.
The Result: Instead of a 100-page manual, the robot now has a single, perfect sentence that captures the whole image.

3. The "Architect" (The Generative Decoder)

Now that the robot has this single "summary sentence," it needs to turn it back into a picture. This is the job of the Decoder.

Think of the Decoder as a master builder.

Old Way: The builder had to guess the shape of the house based on a messy, scattered pile of bricks (a 2D grid of pixels).
RepTok Way: The builder is handed that single, perfect "summary sentence." Because the sentence is so clear and structured, the builder can instantly know exactly how to assemble the house. They don't need to guess; they just follow the blueprint.

4. The "Safety Net" (Cosine Similarity Loss)

There was a risk: If they tweaked the "genius critic" too much to remember every tiny detail, the critic might forget how to think clearly. The "summary sentence" might become a jumbled mess that the builder can't understand.

To fix this, they added a Safety Net.

Imagine the critic is trying to write a summary. The Safety Net gently taps them on the shoulder and says, "Hey, keep it sounding like you. Don't drift too far from your original style."
This ensures the summary remains smooth and logical (mathematically, this keeps the "latent space" smooth), so the builder can still generate beautiful, coherent images.

Why is this a Big Deal?

1. It's Incredibly Fast (The Efficiency)
Because the robot only has to process one token instead of thousands, it doesn't need a super-complex brain to do the work.

Old Way: Requires a massive, expensive engine (like a Transformer with "Attention" mechanisms) to juggle thousands of pieces at once.
RepTok Way: Because there's only one piece to juggle, they can use a tiny, simple engine (an MLP-Mixer). It's like switching from a Formula 1 car to a sleek, efficient electric scooter. It gets the job done with 90% less energy.

2. It Still Looks Great (The Quality)
Even though it's simple and fast, the images it creates are just as good as the slow, expensive ones. It can generate high-quality pictures of cats, cars, and landscapes that look realistic.

3. It's Flexible
The researchers showed that this works not just for simple "draw a cat" tasks, but also for complex "draw a cat wearing sunglasses on a beach" tasks (Text-to-Image). Because the "summary" is so efficient, adding text instructions doesn't slow the system down much.

The Bottom Line

RepTok is like realizing you don't need to memorize every single grain of sand on a beach to describe it. You just need one perfect, well-structured sentence that captures the beach's soul. By using a pre-trained "expert" to write that sentence and a simple "builder" to turn it back into a picture, they created a way to generate images that is fast, cheap, and surprisingly smart.

1. Problem Statement

Current state-of-the-art generative models (Diffusion and Flow-based models) face significant computational bottlenecks:

High-Dimensional Pixel Space: Early models operate directly on pixels, requiring massive compute.
Latent Space Redundancy: Latent Diffusion Models (LDMs) compress images into lower-dimensional spaces but typically retain a 2D grid structure. This grid fails to exploit the high spatial redundancies inherent in natural images, leading to unnecessary token counts and computational overhead during training and inference.
Inefficiency of Existing Tokenizers: While recent works like TiTok reduce token counts to 1D sequences, they often still rely on multiple discrete tokens. Other approaches (e.g., REPA) align features but do not fundamentally change the latent space structure to be a single, compact vector.

The core challenge is to find a latent representation that is extremely compact (minimizing spatial redundancy), continuous (enabling differentiable training), and semantically structured (suitable for generation) while allowing for faithful image reconstruction.

2. Methodology: RepTok

The authors propose RepTok, a framework that adapts pre-trained Self-Supervised Learning (SSL) vision transformers to serve as a single-token latent space encoder.

A. Core Architecture

Encoder (SSL Adaptation): Instead of training a VAE from scratch, RepTok utilizes a pre-trained SSL encoder (e.g., DINOv2, MAE, CLIP).
- Minimal Fine-tuning: Only the [cls] token embedding is fine-tuned; the rest of the encoder remains frozen.
- Single Token: The image is represented by a single continuous vector ( $z \in \mathbb{R}^{1 \times 768}$ ) derived from the pooled [cls] token, eliminating the 2D grid entirely.
Decoder: A generative decoder (typically an MLP-Mixer or Transformer) maps the single latent token back to the pixel space (or a learned latent space via a pre-trained VAE).
Training Objective:
1. Flow Matching: The primary objective is a standard flow matching loss, training the decoder to map Gaussian noise to the target image conditioned on the latent token $z$ .
2. Cosine-Similarity Regularization: To prevent the fine-tuned [cls] token from drifting too far from the original SSL manifold (which ensures smooth geometry for generation), a cosine similarity loss is added:
  $\mathcal{L}_{cos}(x) = \lambda (1 - \cos(z, z_{frozen}))$
  where $z$ is the fine-tuned token and $z_{frozen}$ is the output of the frozen SSL encoder. This balances the need for reconstruction details with the preservation of the smooth latent geometry.

B. Generation Pipeline

Latent Generation: A separate generative model (G) is trained to synthesize the single latent token $z$ from noise using flow matching.
Inference: The process involves generating $z$ from noise, then decoding $z$ into the final image.
Attention-Free Architecture: Because the latent space consists of only one token, token-to-token interactions are unnecessary. This allows the use of MLP-Mixers (attention-free architectures) for the generator, drastically reducing computational complexity compared to standard Transformers.

3. Key Contributions

Single-Token Continuous Latent Space: The paper demonstrates that a single continuous token, derived from a lightly fine-tuned SSL [cls] embedding, is sufficient to capture both high-level semantics and low-level spatial details required for high-fidelity reconstruction.
Efficient Training via Minimal Adaptation: By freezing the bulk of the SSL encoder and only adapting the [cls] token, the method injects necessary reconstruction details without destroying the pre-trained semantic geometry.
Attention-Free Generation: The extreme compression to a single token enables the use of simple, attention-free MLP-Mixer architectures for the generative model, removing the quadratic scaling cost of self-attention.
Scalability to Text-to-Image (T2I): The framework naturally extends to T2I by integrating cross-attention for text conditioning, achieving competitive zero-shot performance with minimal training budgets.

4. Experimental Results

The authors evaluated RepTok on ImageNet-1K (class-conditional) and MS-COCO (text-to-image).

ImageNet Generation (Class-Conditional):
- Performance: RepTok achieves competitive gFID scores (e.g., 1.88 for RepTok-L with CFG) compared to heavy baselines like DiT-XL/2 and SiT-XL/2.
- Efficiency: It reduces training compute by over 90% compared to transformer-based diffusion baselines. For instance, it requires only 1.7% of the FLOPs consumed by SiT.
- Reconstruction: Achieves an rFID of 1.85 and PSNR of 14.94, outperforming methods relying purely on semantic codes (like RCG) and matching spatial tokenizers.
Text-to-Image Synthesis:
- Achieves competitive zero-shot FID on MS-COCO after training for <20 hours on 4x A100 GPUs.
- Performance scales effectively with larger frozen language backbones (CLIP, InternVL, Gemma).
Qualitative Analysis:
- Interpolation: The latent space exhibits smooth transitions in both semantics and spatial configuration (shape, size, rotation), confirming the preservation of the SSL manifold's geometry.
- Reconstruction: High-fidelity reconstructions are possible from a single token, even for out-of-distribution images (MS-COCO) despite training only on ImageNet.

5. Significance and Impact

Paradigm Shift in Latent Modeling: RepTok challenges the necessity of 2D grid latents or multi-token sequences, proving that a single continuous vector can serve as a complete latent representation for generative modeling.
Computational Efficiency: By leveraging pre-trained SSL representations and removing attention mechanisms from the generator, RepTok offers a path to training high-quality generative models with significantly lower energy and compute costs.
Reusability of SSL Models: It highlights the untapped potential of SSL encoders (like DINOv2) not just as feature extractors for downstream tasks, but as the foundational "encoder" for generative pipelines themselves.
Practicality: The ability to achieve competitive results with limited training budgets (hours rather than weeks) makes high-quality generative modeling more accessible to researchers with limited resources.

In summary, RepTok bridges the gap between efficient representation learning and generative modeling, offering a lightweight, high-performance alternative to current diffusion and flow-based paradigms.