🤖 AI

Unsupervised Decomposition and Recombination with Discriminator-Driven Diffusion Models

This paper proposes a discriminator-driven diffusion model that enables unsupervised factorized representation learning and high-quality compositional generation by using an adversarial signal to ensure physical and semantic consistency in recombined samples, demonstrating superior performance on image benchmarks and enhanced robotic exploration capabilities.

Original authors: Archer Wang, Emile Anand, Yilun Du, Marin Soljačić

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Archer Wang, Emile Anand, Yilun Du, Marin Soljačić

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, magical LEGO set. You've built thousands of different castles, spaceships, and animals. But here's the catch: you don't know which specific bricks make up the wheels, the windows, or the wings. You just see the final, finished toy.

Now, imagine you want to build a new toy by taking the wheels from your spaceship, the windows from your castle, and the wings from your dragon. If you just randomly snap pieces together, you might end up with a weird, broken mess where the wheels don't fit the body or the wings are too heavy.

This paper is about teaching a computer how to do exactly that: take complex images or videos, figure out which "LEGO bricks" (factors) make them up, and then snap them together to create something new that actually looks real.

Here is the simple breakdown of how they did it:

1. The Problem: The "Frankenstein" Effect

Current AI models (specifically "Diffusion Models," which are like artists that slowly turn static noise into a clear picture) are great at making pictures. But they are terrible at understanding the parts of the picture.

If you ask a standard AI to mix a picture of a "red car" with a "blue sky," it might just blur them together or create a car that looks like it's melting into the clouds. It hasn't learned that "car" and "sky" are separate things that can be swapped around. It's like trying to mix two smoothies and expecting to get the fruit back out separately.

2. The Solution: The "Bouncer" (The Discriminator)

The authors introduced a clever trick: a Bouncer.

The Artist (Generator): This is the AI trying to build the new, mixed-up image. It takes the "wheels" from Image A and the "sky" from Image B and tries to glue them together.
The Bouncer (Discriminator): This is a second AI whose only job is to stand at the door and say, "Is this a real, natural-looking image, or is it a weird, glitchy Frankenstein monster?"

How they train together:

The Artist tries to mix parts from two different photos.
The Bouncer looks at the result. If it looks weird (like a car floating in the sky or a face with two noses), the Bouncer says, "Nope, that's fake!"
The Artist gets a "scolding" (a mathematical penalty) and tries again, adjusting its internal "LEGO bricks" so the mix looks more natural.
Over time, the Artist gets so good at mixing that the Bouncer can't tell the difference between a "real" photo and a "mixed" photo.

3. The Magic Result: Learning the "Bricks"

Because the Artist is constantly being punished for making weird mixes, it is forced to learn the true structure of the world.

It learns that "lighting" is one brick.
It learns that "object shape" is another brick.
It learns that "background" is a third brick.

Once it learns these separate bricks, it can swap them around freely. It can take a "sunny day" brick and put it on a "nighttime city" brick, and the result will look like a perfectly logical, sunny city.

4. Why This Matters: From Pictures to Robots

The paper didn't just stop at pretty pictures. They tested this on robot videos.

Imagine a robot learning to pick up a cup. It has watched a video of a human doing it.

Without this method: The robot only knows exactly how that one human moved. If the cup is in a slightly different spot, the robot gets confused.
With this method: The robot learns the "bricks" of the action: "reach," "grasp," "lift," and "place."
The Superpower: The researchers mixed the "reach" from one video with the "grasp" from another. The robot then generated a new plan to pick up a cup in a way it had never seen before.

This allowed the robot to explore its world much more effectively, trying out thousands of new combinations of movements to see what works, rather than just copying what it saw.

The Big Takeaway

Think of this paper as teaching an AI to be a master chef instead of a photocopier.

A photocopier just copies the whole dish.
A master chef understands the ingredients (salt, pepper, heat, time). They can take the "spicy sauce" from a Mexican dish and put it on a Japanese noodle dish, and it will still taste delicious.

By using a "Bouncer" to constantly check if the new combinations make sense, the AI learned to separate the ingredients of reality, allowing it to cook up entirely new, realistic scenarios that it has never seen before.

1. Problem Statement

The paper addresses the challenge of unsupervised disentanglement and compositional generation in complex data (images and videos).

The Goal: To decompose data into factorized latent representations (e.g., separating background, lighting, object attributes, or motion primitives) without explicit supervision, such that these factors can be recombined to synthesize new, physically plausible, and semantically coherent samples.
The Challenge: Fully unsupervised disentanglement is theoretically impossible without additional assumptions. Existing methods (like $\beta$ -VAE or standard diffusion models) often fail to learn representations that support compositional generalization. When latent factors from different sources are naively recombined, the resulting samples often lie "off-manifold," producing visual artifacts or physically impossible scenarios (e.g., inconsistent lighting or broken object geometry).
The Gap: Current approaches lack a mechanism to verify whether a recombination is plausible without relying on expensive external feedback (like human preference or physical simulation) during the training of the generative model itself.

2. Methodology

The authors propose a framework that integrates Diffusion Models with an Adversarial Discriminator to enforce physical and semantic consistency during recombination.

A. Factorized Generative Model

The core architecture builds upon Decomp Diffusion, which models the denoising prediction $\epsilon_\theta$ as an average of contributions from $K$ distinct latent factors:
$\epsilon_\theta(x_t, t) = \frac{1}{K} \sum_{k=1}^K \epsilon_\theta(x_t, t, z_k)$
where $z = (z_1, \dots, z_K)$ are the latent factors. This formulation allows for the decomposition of the image/video into components.

B. Discriminator-Driven Adversarial Training

The key innovation is the introduction of a recombination discriminator ( $D_\psi$ ) trained to distinguish between:

Single-source samples: Generated from latents inferred from a single input image/video.
Recombined samples: Generated by mixing latent factors from two different source inputs ( $z^{(A)}$ and $z^{(B)}$ ) via a binary mask $S$ .

Training Objective:
The system uses an alternating minimization strategy:

Discriminator Update: Trained to minimize classification loss, distinguishing real (single-source) predictions from fake (recombined) predictions.
$L_{clf} = -\mathbb{E}[\log D_\psi(\hat{x}_{single}) + \log(1 - D_\psi(\hat{x}_{recomb}))]$
Generator/Encoder Update: Trained to minimize reconstruction loss (standard diffusion loss) plus an adversarial loss that encourages recombined samples to fool the discriminator.
$L_{adv} = -\mathbb{E}[\log D_\psi(\hat{x}_{recomb})]$
$L_{total} = L_{MSE} + \lambda L_{adv}$

Key Insight: By optimizing the generator to "fool" the discriminator, the model is implicitly forced to learn a latent space where coordinate-wise recombination results in samples that are indistinguishable from valid data. This acts as a regularizer that contracts statistical dependence between latent components, pushing the representation toward a Cartesian product structure (factorization).

C. Application to Robotics (Video)

For robotic videos, the method conditions the diffusion model on the first frame to stabilize the scene layout. The latent factors capture action primitives (e.g., reaching, grasping, moving). Recombining these factors allows the generation of novel task trajectories that were not present in the training demonstrations but remain physically executable.

3. Key Contributions

Adversarial Signal for Factorization: Introduced a novel training signal using a discriminator to distinguish single-source vs. recombined samples, effectively guiding unsupervised models toward better disentanglement without ground-truth labels.
Improved Compositional Generation: Demonstrated that this adversarial feedback significantly reduces "off-manifold" artifacts in recombined images and videos, ensuring physical and semantic consistency.
Robotic Exploration: Applied the method to the LIBERO benchmark, showing that recombining learned action components generates diverse video plans that significantly expand state-space coverage for robot exploration.
Theoretical Analysis: Provided theoretical justification showing that enforcing recombination closure implies a Cartesian-product structure in the latent space and that random recombination contracts mutual information between latent components.

4. Experimental Results

A. Image Datasets (CelebA-HQ, Virtual KITTI, CLEVR, Falcor3D)

Perceptual Quality: The method achieved significantly lower Fréchet Inception Distance (FID) scores compared to the baseline (Decomp Diffusion) and other models (InfoGAN, $\beta$ $β$ -VAE).
- Example: On CelebA-HQ, FID dropped from 82.70 (baseline) to 43.98.
Disentanglement: Measured by MIG (Mutual Information Gap) and MCC (Mean Correlation Coefficient). The proposed method consistently outperformed baselines, indicating better separation of independent factors.
- Example: On Falcor3D, MCC improved from 0.657 (Decomp Diffusion) to 0.707.

B. Robotics (LIBERO Benchmark)

State-Space Coverage: The method was used to generate novel robotic trajectories by recombining action primitives.
Performance: The "Recombine+Generate" approach increased the number of visited state-space bins by 35.8% in Scene 5 and 73.8% in Scene 6 compared to the baseline without discriminator feedback.
Physical Consistency: The generated plans, when executed, showed lower physical inconsistency scores ( $L_{phys}$ ), meaning the robot could successfully execute the recombined actions (e.g., placing a pudding on a plate edge, a combination not seen in training).

5. Significance and Impact

Bridging Representation and Generation: The paper successfully bridges the gap between representation learning (disentanglement) and generative modeling (composition). It proves that improving the quality of recombination is a viable proxy for learning better, more factorized representations.
Scalable Feedback: It offers a scalable alternative to expensive external feedback (like human labeling or real-world robot trials) by using an internal discriminator to enforce plausibility.
Robotics Advancement: The application to robotics demonstrates a practical pathway for directed exploration. Instead of random exploration, agents can systematically explore the combinatorial space of learned skills, leading to more robust policies in unseen environments.
Generalizability: The approach is domain-agnostic, working effectively on static images (faces, scenes) and dynamic sequences (robotic videos), suggesting broad applicability in generative AI.

In summary, this work establishes that adversarial feedback on recombination is a powerful mechanism for learning disentangled, compositional representations in diffusion models, leading to higher quality generation and more effective exploration in robotic control.