🤖 machine learning

Marginal Matching Does Not License Factorized Sampling: Auditing Conditional Style Leakage in Factorized Generative Models

This paper demonstrates that matching the marginal distribution of a latent style variable to a Gaussian prior is insufficient to guarantee independence from class labels, as factorized generative models can still exhibit significant conditional style leakage that allows class prediction despite appearing Gaussian in aggregate.

Original authors: Duong Bach, Hai Nguyen Hong, Cuong Do

Published 2026-08-07

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Duong Bach, Hai Nguyen Hong, Cuong Do

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a super-smart robot artist. You want this robot to be able to paint a picture of anything—a cat, a car, or a cloud—just by giving it a simple instruction like "draw a cat." To do this, you teach the robot to split its brain into two separate parts. One part, let's call it the "Content Brain," learns what makes a cat a cat (pointy ears, whiskers). The other part, the "Style Brain," learns the random, messy details that make this specific cat different from that cat (is it fluffy? is it black? is it sleeping?).

The goal is for the robot to be a master mixer. You want to be able to take the "Content Brain" of a cat and the "Style Brain" of a dog, and have the robot paint a fluffy, black, sleeping dog that looks exactly like the dog it was trained on, just with the cat's instructions. If the robot works perfectly, the "Style Brain" should be completely blind to what kind of animal it is looking at; it should only see "randomness."

For years, scientists have used a clever trick to check if the robot's "Style Brain" is doing its job. They check if the average of all the styles the robot has ever seen looks like a perfect, random cloud of points (a Gaussian distribution). If the average looks random, they assume the robot has successfully forgotten which animal it was looking at. It's like checking if a bag of mixed-up socks looks like a random pile from the outside. If it does, you assume you can't tell which sock belongs to which foot just by looking at the pile.

But here is the twist: what if the bag of socks is actually perfectly organized on the inside, even though the pile looks random from the outside? What if all the left-foot socks are secretly hiding in one corner of the bag and all the right-foot socks are in another, but they are mixed up so well that the total pile still looks like a random mess? If you reach in and grab a sock, you might accidentally grab a left-foot sock, and suddenly your robot thinks, "Oh, I must be drawing a left-footed creature!"

This is exactly the problem a new paper from researchers at VinUniversity in Vietnam is exposing. They found that the "random pile" test is a lie. Just because the average style looks random doesn't mean the robot has forgotten the labels. In fact, the robot might be hiding the secret identity of every single image inside its "Style Brain," waiting to be used.

The Great Style Leak

The researchers, Duong Bach, Hai Nguyen Hong, and Cuong Do, decided to audit a bunch of these robot artists to see if they were actually playing fair. They didn't just look at the "average pile" (the marginal statistics); they looked at the specific piles for each animal. They asked: "If I only look at the styles of cats, do they look different from the styles of dogs?"

The answer was a resounding yes. In fact, the leak was massive.

They tested a model they built (called F-CS-WAE) and four other popular robot artists. Even though all of them passed the "random pile" test with flying colors (their global statistics were nearly perfect), the researchers found that a simple computer program could look at the "Style Brain" and guess what animal was being drawn with terrifying accuracy. On one dataset, the robot's style brain was 100% accurate at guessing the animal label. On another, it was 99.15% accurate.

This is like a spy who has memorized the entire secret code of a country. Even if the spy's notes, when shuffled and averaged, look like random scribbles, the spy can still decode the message perfectly if they know which page they are looking at. The researchers showed that the "Style Brain" wasn't just carrying random noise; it was carrying the exact identity of the animal, hidden in plain sight.

Why the Robot Fails the Magic Trick

So, why does the robot fail when you try to mix and match?

Imagine you are the robot. You have been trained to paint cats. When you see a cat, your "Content Brain" says "Cat!" and your "Style Brain" says "Fluffy, black, sleeping." You learn that these two things always go together. You never see a "Fluffy, black, sleeping" style attached to a "Dog" instruction.

Now, you are asked to paint a "Dog." You get the "Dog" instruction for your Content Brain. But for the Style Brain, you are told to grab a random style from the big bag of all styles. Because the bag is secretly organized, you might accidentally grab the "Fluffy, black, sleeping" style that belongs to a cat.

Here is the disaster: Your robot brain has learned that "Fluffy, black, sleeping" only goes with "Cat." It has never seen that style with "Dog." So, when you try to paint a dog with a cat's style, your robot gets confused. It trusts the style more than the instruction. It ignores the "Dog" command and paints a cat instead.

The researchers found that this happens all the time. When they tried to generate a new image of a specific class (like a "3" in the MNIST number dataset) using a random style, the robot only got it right 16% of the time. It kept painting the wrong numbers, usually the ones that were most popular in its "Style Brain." The robot was so addicted to the style clues that it forgot the instructions entirely.

The Four-Part Mystery

The paper doesn't just say "it's broken"; it breaks down exactly why it's broken into four distinct pieces, like a detective solving a crime scene:

The Style Prior Mismatch: The "Style Brain" doesn't look exactly like the random cloud the scientists wanted it to be.
The Style Leakage (The Big Culprit): The "Style Brain" knows the label. It knows if it's a cat or a dog. This is the main problem.
The Semantic Prior Mismatch: The "Content Brain" isn't perfectly aligned with the instructions either.
The Within-Class Dependence: Even inside the group of "Cats," the "Content Brain" and "Style Brain" are holding hands and whispering secrets to each other.

The researchers proved mathematically that to make the robot work perfectly, all four of these problems need to be fixed. You can't just fix the leakage (Problem 2) and call it a day. Even if you force the "Style Brain" to forget the label, if the other three problems are still there, the robot will still fail.

Trying to Fix the Leak

The team tried several ways to patch the hole. They tried making the robot pay attention to the labels while it was learning, forcing the "Style Brain" to look the same for every animal. They tried different mathematical tricks, like "gradient reversal" (which is like telling the robot, "If you guess the animal correctly, you get a penalty!").

Some of these tricks worked better than others. One method, called "Gradient Reversal," was very good at making the "Style Brain" forget the label, dropping the guessing accuracy down to 21% (which is much better than 100%, but still not perfect). Another method, using a "Per-Class Style MMD," dropped it to 42.6%.

But here is the catch: even when they fixed the leakage, the robot still didn't work perfectly. Why? Because the other problems (like the "Content" and "Style" brains holding hands) were still there. The researchers showed that fixing the leakage is necessary, but it is not enough. It's like fixing a leak in a boat but forgetting that the boat also has a hole in the bottom and a broken rudder. You still sink.

The Dataset Surprise

The researchers also discovered something interesting about how hard the problem is. They tested on two different sets of pictures: MNIST (simple black-and-white numbers) and CIFAR-10 (colorful, complex photos of animals and objects).

On the simple numbers (MNIST), the robot was a master thief; it could guess the label from the style almost perfectly. On the complex photos (CIFAR-10), the robot was a bit more clumsy, but it still leaked a lot of information. The researchers found that the more complex and diverse the pictures were, the less the robot relied on the style to guess the label. This makes sense: if every cat looks totally different, it's harder to hide a secret code in the style. But even on the complex dataset, the robot still leaked enough to be dangerous.

The Final Lesson

The most important takeaway from this paper is a warning to anyone building these robot artists. You cannot just check if the "Style Brain" looks random on average. That test is a trap. You have to check if the style is truly blind to the label for every single class.

The researchers showed that even if a model passes all the standard tests and looks great on paper, it might be completely useless at the one thing it was built to do: mixing and matching styles. They found that a model could be 99% accurate at recognizing a cat, but when you asked it to paint a "Cat with a Dog's style," it would fail 84% of the time.

In the end, the paper suggests that we need new ways to test our robots. We need to look under the hood, not just at the hood. We need to make sure that the "Style Brain" is truly independent, not just pretending to be. Until we do that, our robot artists will keep painting the wrong pictures, confused by the secret codes they learned to hide.

Technical Summary: Marginal Matching Does Not License Factorized Sampling

Problem Statement

Factorized generative models aim to separate content (semantic) and style information into distinct latent variables, $z_c$ and $z_s$ . A standard practice in these models is to regularize the style latent $z_s$ toward a fixed prior (typically a standard Gaussian, $N(0, I)$ ) using a marginal statistic, such as the Maximum Mean Discrepancy (MMD) between the aggregated posterior $q(z_s)$ and the prior. Practitioners often treat a low marginal divergence as a certificate that $z_s$ carries no class information (i.e., $z_s \perp y$ ).

The paper argues that this certification is fundamentally flawed. Matching the marginal distribution $q(z_s)$ to a prior constrains nothing about the class-conditional distributions $q(z_s | y)$ . It is mathematically possible for $q(z_s)$ to be exactly Gaussian while every conditional $q(z_s | y=k)$ occupies a distinct, non-overlapping region of the latent space. Consequently, $z_s$ can remain maximally informative about the label $y$ even when the global marginal statistics appear perfect.

This gap leads to conditional style leakage: during training, the decoder learns to associate specific class information with $z_s$ because it only sees $(z_c, z_s)$ pairs derived from the same image. At generation time, when $z_c$ is sampled from a class-conditional prior and $z_s$ is sampled from the marginal prior (averaging over all classes), the decoder receives a pair it never saw during training. If $z_s$ encodes class information, the decoder may fail to generate the intended class, relying instead on the "leaked" style code to determine identity.

Methodology

Theoretical Decomposition

The authors provide an exact decomposition of the mismatch incurred by a factorized sampler, denoted as $M_{fact}$ . This mismatch is the expected KL divergence between the true joint posterior and the factorized sampler's distribution:
$M_{fact} = I_q(z_c; z_s | y) + E_y[KL(q(z_c | y) \parallel p(z_c | y))] + I_q(z_s; y) + KL(q(z_s) \parallel p(z_s))$
The four non-negative terms are:

Style-prior mismatch: $KL(q(z_s) \parallel p(z_s))$ (Global marginal fit).
Style leakage: $I_q(z_s; y)$ (Mutual information between style and label).
Semantic-prior mismatch: $E_y[KL(q(z_c | y) \parallel p(z_c | y))]$ .
Within-class dependence: $I_q(z_c; z_s | y)$ .

The authors prove that valid factorized sampling requires all four terms to vanish. Therefore, eliminating style leakage (Term 2) is necessary but not sufficient for successful independent sampling; the other terms must also be addressed.

Auditing Framework

To detect leakage, the paper proposes a diagnostic framework using finite-sample proxies for the theoretical terms:

Global MMD: Proxies Term 1.
Inter-class separation ( $\Delta_{inter}$ ) and Linear Probe (LP): Proxies Term 2. $\Delta_{inter}$ measures the mean separation between class-conditional style distributions, while LP measures the accuracy of a linear classifier predicting $y$ from $z_s$ .
JointMMD: Proxies Term 4 by comparing real within-class $(z_c, z_s)$ pairs against recombined pairs where $z_s$ is permuted within the same class.

Case Study and Remedies

The authors instantiate these concepts using F-CS-WAE (Factorized Class-Structured Spherical Cauchy WAE), which uses a hyper-spherical semantic prior and a Euclidean style prior. They evaluate this model and four reference baselines (VAE, WAE-MMD, $\beta$ -TCVAE, FactorVAE) on MNIST and CIFAR-10.

To address the leakage, they test several remedies:

Per-class Style MMD: Instead of matching the global marginal, this loss matches the style distribution of each class $k$ to the prior $N(0, I)$ individually.
Gradient Reversal: A standard adversarial technique to remove label information from $z_s$ .
HSIC Penalty: Penalizing dependence between $z_s$ and $y$ .
Joint Term-4 Intervention: Directly targeting within-class dependence.

They also evaluate post-hoc sampling strategies, such as using a class-conditional diagonal Gaussian prior or an empirical style bank, without retraining the model.

Key Results

Marginal Metrics Hide Leakage

Empirical results show that global MMD can be near zero while leakage is severe.

On MNIST, baselines like $\beta$ -TCVAE and FactorVAE achieved near-zero global MMD but still allowed linear probes to recover labels from $z_s$ with 74–76% accuracy (chance is 10%).
The F-CS-WAE case study showed a global MMD of 0.0013, yet the linear probe achieved 100% accuracy.
Crucially, the F-CS-WAE model achieved 99.15% clustering accuracy and high reconstruction quality (SSIM 0.983), yet its class-conditional generation accuracy (Gen-ACC) under standard sampling was only 16%. This demonstrates that standard diagnostics (clustering, reconstruction, global MMD) fail to detect the failure mode relevant to factorized generation.

Decoder Sensitivity

Experiments involving latent swapping (taking $z_c$ from class $A$ and $z_s$ from class $B$ ) revealed that the decoder relies heavily on $z_s$ for identity. On MNIST, the external classifier assigned the swapped output to the style-donor class 96.5% of the time, confirming that the decoder learned to read identity from the "leaked" style code.

Effectiveness of Remedies

Per-class MMD: Reduced the linear probe accuracy from 100% to 42.6% and $\Delta_{inter}$ by 81%. However, it did not eliminate leakage, and the JointMMD (Term 4 proxy) remained statistically unchanged.
Gradient Reversal: Achieved the strongest reduction in leakage among Term-2 remedies, lowering probe accuracy to 21%.
Joint Intervention: Only the intervention targeting Term 4 (within-class dependence) significantly reduced the JointMMD proxy (by 2.7 $\times$ ), confirming that reducing mutual information $I(z_s; y)$ does not automatically resolve within-class dependencies.
Robustness: Six perturbations (capacity, curriculum, prior geometry, supervision) failed to eliminate leakage entirely. While capacity strongly modulated leakage magnitude (especially on CIFAR-10), no single design choice was the sole cause.

Transferability of Fixes

The paper highlights a significant gap between datasets:

MNIST: A post-hoc class-conditional prior raised Gen-ACC from 0.16 to 0.97.
CIFAR-10: The same parametric fix only reached 0.41. An empirical style bank reached 0.88, suggesting that a simple diagonal Gaussian assumption is insufficient for complex datasets.

Significance and Claims

The paper claims that no divergence computed solely on the marginal $q(z_s)$ can certify that $z_s \perp y$ . The reliance on marginal statistics (like global MMD) provides a false sense of security regarding class invariance in factorized models.

The authors emphasize that:

Structural Necessity: Eliminating style leakage is necessary but not sufficient for valid factorized sampling; the other three terms in the decomposition must also be addressed.
Diagnostic Necessity: Practitioners must use conditional diagnostics (e.g., linear probes, inter-class separation) rather than marginal statistics to verify class invariance.
Decoder-Level Reality: Even if a latent space appears disentangled, the decoder may still exploit leaked information, leading to generation failures that standard metrics miss.
Dataset Dependence: Remedies that work on simple datasets (MNIST) may fail on more complex ones (CIFAR-10), and the severity of leakage is strongly dependent on dataset complexity and model capacity.

The paper concludes that reporting marginal statistics is insufficient to verify the properties practitioners claim for factorized generative models, and that a more rigorous, conditional auditing framework is required to ensure the validity of independent sampling.