💻 computer science

REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion

REGLUE is a unified latent diffusion framework that enhances image synthesis by entangling VAE latents with both global and locally compressed Vision Foundation Model semantics within a single SiT backbone, achieving superior sample quality and faster convergence compared to existing baselines.

Original authors: Giorgos Petsangourakis, Christos Sgouropoulos, Bill Psomas, Theodoros Giannakopoulos, Giorgos Sfikas, Ioannis Kakogeorgiou

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Giorgos Petsangourakis, Christos Sgouropoulos, Bill Psomas, Theodoros Giannakopoulos, Giorgos Sfikas, Ioannis Kakogeorgiou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to paint a masterpiece. You don't just want it to copy a photo pixel-by-pixel; you want it to understand what it is painting. This is the world of Latent Diffusion Models, the current superstars of AI image generation. Think of these models as artists who start with a canvas covered in static noise (like TV snow) and slowly, step-by-step, clean it up until a clear picture emerges. Usually, they learn by trying to reconstruct images they've seen before. But there's a catch: this "reconstruction" method is like teaching a chef by only showing them the final dish. The chef learns the taste eventually, but it takes a long time to figure out the recipe, and the early attempts might look a bit muddy.

To speed things up, scientists have started bringing in "expert consultants"—pre-trained Vision Foundation Models (VFMs). These are like super-smart art critics who have studied millions of paintings and can instantly tell you if a picture has a dog, a tree, or a specific mood. The big question in this corner of computer science is: How do we get our painting robot to listen to these experts without getting confused? Some researchers tried to just show the robot the expert's final verdict, while others tried to feed it the expert's notes on every tiny patch of the canvas. The paper you are about to read tackles the messy middle ground, asking how to best combine the robot's own sketching skills with the expert's detailed, patch-by-patch advice.

The Problem: The Robot is Slow and the Expert is Too Chatty

Meet REGLUE (Representation Entanglement with Global-Local Unified Encoding). Before REGLUE showed up, AI artists had two main ways to use those expert consultants. One way was to just listen to the expert's "big picture" summary (like saying "it's a cat"). The other was to try to listen to the expert's notes on every single tiny square of the image (like "this pixel is fur, that one is an eye").

The problem? The "big picture" summary is too vague to help with fine details, and the "tiny square" notes are often too messy and huge to fit into the robot's brain. Previous attempts to simplify these notes were like trying to squish a high-definition movie into a text message using a basic translator (linear compression); you lose the nuance, and the robot gets confused.

The REGLUE Solution: A Smart Translator and a Team Huddle

The authors of this paper propose a new way to run the art class. They introduce a lightweight semantic compressor. Imagine this as a super-smart, tiny translator that sits between the expert and the robot. Instead of just summarizing the whole image or dumping the raw notes, this translator takes the expert's complex, multi-layered observations and condenses them into a compact, organized "cheat sheet" that the robot can actually use.

Here is how REGLUE works in practice:

The Team Huddle: The robot doesn't just look at its own sketch (the image latent) or the expert's notes separately. REGLUE forces them to hold hands. It takes the robot's sketch, the expert's "big picture" summary (the global token), and the expert's detailed "patch-level" notes (the local semantics) and mixes them all together into one single stream of information.
The Non-Linear Magic: The key innovation is that the translator (the compressor) doesn't just use simple math to shrink the data. It uses a non-linear approach, which is like a chef who knows how to blend ingredients perfectly rather than just chopping them up. This preserves the rich, complex details of the expert's knowledge while making them small enough to fit.
The Double Check: To make sure the robot is really listening, the system also adds a "homework check" (an external alignment loss). At certain points during the training, the robot has to prove it understands the expert's notes by matching its internal thoughts to the expert's original thoughts.

What They Found: Speed and Clarity

The researchers tested this new method on ImageNet, a massive collection of 256×256 images, using a model called SiT-B/2. The results were quite impressive.

Faster Learning: The REGLUE-trained robot learned much faster than the others. In just 300,000 steps of training, REGLUE reached a quality score (FID) of 14.5. To beat that score, the previous best method (REG) needed 400,000 steps. REGLUE reached an even better score of 12.9 at 400,000 steps, while the standard robot (SiT-B/2) was stuck at a much worse 33.0.
The Secret Sauce: The paper explicitly rules out the idea that just adding more data helps. They showed that if you use a simple, linear translator (like the one used in a previous method called ReDi), the results are mediocre (FID 21.4). It is the non-linear compression that unlocks the power. Without it, the robot gets overwhelmed.
Local vs. Global: They found that the "patch-level" (local) details are the most important. A robot that only listened to the "big picture" (global) summary performed poorly (FID 25.7). But when the robot got the detailed patch notes, performance jumped.
The Perfect Combo: The best results came from combining everything: the detailed local notes, the big picture summary, and the homework check. Using a powerful expert model called DINOv3-B, REGLUE achieved a stunning FID of 12.3, and with the standard DINOv2-B, it hit 12.9.

Why It Matters

The paper suggests that the future of AI art isn't just about making bigger robots or feeding them more data. It's about how we connect the robot to the knowledge it already has. By using a smart, non-linear translator to organize the expert's advice and forcing the robot to learn from both the big picture and the tiny details at the same time, REGLUE creates images that are sharper, more coherent, and learned in less time.

The authors are careful to note that this isn't magic; it's a specific engineering choice. They proved that simply stacking more features without the right compression actually makes things worse. But with their "entanglement" strategy, they managed to squeeze out significant improvements in image quality and training speed, suggesting that the way we structure information inside these AI brains is just as important as the brains themselves.

Technical Summary: REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion

Problem Statement

Latent Diffusion Models (LDMs) have established state-of-the-art performance in image synthesis by shifting the generative paradigm to a compact VAE latent space. However, training these models presents a fundamental challenge: the standard reconstruction-style denoising objective provides only indirect semantic supervision. Consequently, high-level semantics (objects, layout, relations) emerge slowly during training, limiting both sample quality and convergence speed.

Recent attempts to address this by injecting semantics from Vision Foundation Models (VFMs) have been suboptimal. Existing approaches either rely solely on external representation alignment (distilling features as an external teacher) or jointly model semantic signals within the diffusion process but restrict them to narrow slices of VFM features. Specifically:

REG models only a single image-level global token ([CLS]), lacking spatial granularity.
ReDi models patch-level (local) features but relies on linear PCA projection, which constrains representations to a low-dimensional linear subspace, discarding rich nonlinear and multi-layer spatial information.

The authors argue that effectively leveraging VFM features requires jointly modeling spatial semantics within the diffusion model while preserving their nonlinear, multi-layer information through compact, learned representations.

Methodology: REGLUE

The paper introduces REGLUE (Representation Entanglement with Global–Local Unified Encoding), a unified latent diffusion framework that jointly models three modalities within a single SiT (Scalable Interpolant Transformer) backbone:

VAE Image Latents: The standard reconstruction-optimized latent space.
Local (Patch-level) Semantics: Compressed, spatially structured VFM features.
Global (Image-level) Semantics: The VFM [CLS] token.

Key Components

1. Lightweight Semantic Compressor
To address the dimensionality mismatch between high-dimensional VFM features ( $D_f \gg D_z$ ) and VAE latents, REGLUE employs a lightweight, nonlinear convolutional semantic compressor ( $E_\psi$ ).

Architecture: A shallow convolutional autoencoder pre-trained offline to reconstruct multi-layer VFM features.
Function: It aggregates multi-layer patch-level features (via channel-wise concatenation) and maps them to a compact, spatially structured representation ( $s^*$ ) with significantly reduced dimensionality ( $D_s \ll \sum D_\ell$ ).
Benefit: Unlike linear PCA, this nonlinear compression preserves essential semantic information while re-balancing the capacity of the diffusion model, preventing the semantic modality from dominating the VAE latents.

2. Unified Diffusion Modeling
The SiT backbone jointly models the VAE latents ( $z_t$ ), compressed local semantics ( $s_t$ ), and the global token ( $cls_t$ ).

Forward Process: Independent noise is injected into all three modalities using a shared schedule.
Tokenization: The VAE latents and compressed semantics are patchified and projected to a common token dimension. They are merged channel-wise to maintain a single grid of $N$ tokens, avoiding the quadratic overhead of concatenating sequences. The global token remains a separate token.
Objective: The model is trained with a multimodal velocity loss, predicting the velocity for all three modalities simultaneously.

3. External Representation Alignment
As an auxiliary signal, REGLUE incorporates a feature-alignment loss (inspired by REPA). At a selected Transformer block, hidden tokens are projected to match frozen VFM targets (both global and local) via a cosine similarity loss. This regularizes internal representations toward the teacher VFM without requiring the diffusion model to generate the raw VFM features directly.

Key Contributions

Unified Framework: REGLUE is the first framework to jointly model image-level (global) and patch-level (local) VFM semantics alongside VAE latents within a single diffusion backbone.
Nonlinear Semantic Compression: The proposal of a lightweight, nonlinear semantic compressor that aggregates multi-layer VFM features into a compact, semantics-preserving space. This enables significant gains via patch-level joint modeling, outperforming linear projection methods.
Synergistic Design: The paper demonstrates that patch-level semantics, global tokens, and external alignment act synergistically. The primary performance gains stem from the spatial joint modeling of compressed patch features, while global tokens and alignment serve as complementary enhancements.
Efficiency: The method improves generative performance and accelerates convergence without increasing the diffusion model's parameter count or inference-time compute (throughput remains comparable to baselines).

Experimental Results

Experiments were conducted on the ImageNet 256×256 generation benchmark using SiT-B/2 and SiT-XL/2 backbones.

Performance Improvements:

FID Reduction: REGLUE consistently outperforms baselines. On SiT-B/2 at 400K steps, REGLUE achieves an FID of 12.9, compared to 33.0 for vanilla SiT, 24.4 for REPA, 21.4 for ReDi, and 15.2 for REG.
Convergence Speed: REGLUE reaches the 1M-step performance of ReDi and REG using less than 30% and 80% of their respective training iterations. For instance, SiT-XL/2+REGLUE reaches an FID of 2.5 at 1M steps, outperforming REG (2.7) and ReDi (5.1).
Unconditional Generation: In unconditional settings, REGLUE achieves an FID of 28.7 on SiT-B/2, surpassing the conditional baseline (33.0) and other SOTA methods.

Ablation Insights:

Local vs. Global: Patch-level (local) semantics are critical; modeling them alone yields larger gains than modeling only global tokens.
Compression: Nonlinear compression is essential. Linear PCA (as used in ReDi) yields suboptimal results (FID 21.4 vs. 14.3 for nonlinear compression without alignment).
Layer Aggregation: Aggregating the last four layers of the VFM (9–12) before compression provides the best signal, whereas including shallow layers degrades performance.
Diffusability: Frequency analysis (DCT) reveals that the nonlinear compressor suppresses high-frequency components that diffusion models struggle to model, resulting in a "diffusion-friendly" representation while preserving semantic accuracy.

Significance and Claims

The paper claims that REGLUE represents a significant advancement in latent diffusion by effectively "entangling" reconstruction-optimized and semantics-optimized representations. The authors assert that the critical design choice for leveraging VFMs is not merely aligning features externally or modeling a single token, but rather jointly modeling spatial semantics within the diffusion process using nonlinear, compact representations.

The work demonstrates that:

Spatial VFM semantics are crucial for capturing fine-grained structure.
Nonlinear compression is the key mechanism to unlock the full benefit of these rich, multi-layer features.
Global tokens and external alignment act as complementary, lightweight enhancements that further refine the generation process.

By achieving state-of-the-art results with fewer training iterations and no increase in inference cost, REGLUE offers a scalable and efficient path toward higher-quality image synthesis. The code is publicly available to facilitate further research in joint image-feature synthesis.