💻 computer science

The Learnability Gap in Medical Latent Diffusion

This paper identifies and formalizes a "learnability gap" in medical latent diffusion models, revealing that despite high reconstruction fidelity, the latent representations of large-scale pretrained autoencoders are inherently difficult for classifiers to learn from, a structural issue that persists across architectures and cannot be resolved by domain-specific fine-tuning.

Original authors: Mischa Dombrowski, Felix Nützel, Bernhard Kainz

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mischa Dombrowski, Felix Nützel, Bernhard Kainz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Translator" Problem

Imagine you are trying to teach a student (a computer program) to recognize rare diseases in medical scans, like spotting a tiny, specific type of tumor in a chest X-ray.

Because rare diseases are hard to find in real hospitals, researchers use Generative AI to create fake but realistic medical images to help the student practice. To do this efficiently, they use a special "translator" system called an Autoencoder.

The Encoder: Takes a complex medical image and compresses it into a tiny, secret code (the "latent space"). Think of this like translating a 500-page novel into a single, dense paragraph of shorthand notes.
The Decoder: Takes that shorthand note and tries to rebuild the original 500-page novel.
The Diffusion Model: Uses these shorthand notes to generate new, fake stories (images).

The Discovery: The "Learnability Gap"

The researchers found a strange problem they call the Learnability Gap.

Here is the situation:

The Decoder is perfect: When the AI takes the shorthand note and turns it back into an image, the result looks almost identical to the original. If you look at the picture, you can see all the details. The "translator" did a great job of keeping the information safe.
The Student is confused: However, when they try to teach a classifier (the student) to read the shorthand notes directly to spot diseases, the student fails miserably. Even though the information is there in the notes, the notes are written in a way that is incredibly hard to read.

The Analogy:
Imagine a librarian who takes a complex book and writes a summary on a sticky note.

If you give the full book back to a reader, they can find the answer to any question easily.
If you give the sticky note to the reader, they can't find the answer, even though the librarian claims the note contains all the necessary facts.
The problem isn't that the librarian forgot the facts (the image is reconstructed perfectly). The problem is that the way the facts are organized on the sticky note is confusing and messy.

What They Tested

The researchers tested this idea across five different types of "translators" (autoencoders) and four different medical datasets (chest X-rays, skin lesions, CT scans, and heart echocardiograms).

They found that no matter how good the translator was, or how much they tried to "fine-tune" it to understand medical jargon, the sticky note (the latent code) remained hard for the student to learn from.

Fine-tuning didn't help: Even when they trained the translator specifically on medical data, the gap didn't close. It's like hiring a translator who speaks perfect medical Latin but still writes their notes in a code that no one else can decipher.
Image quality didn't matter: A translator that produced a crystal-clear picture (high fidelity) didn't necessarily produce a "readable" note. The clarity of the picture and the readability of the code are two different things.

The Solution They Tried: "Noise-Conditioned" Classifiers

Since they couldn't fix the translator, they tried to make the student smarter at reading the messy notes.

They built a special student that uses noise conditioning.

The Analogy: Imagine trying to learn a language by listening to a radio station with static. If you only listen when the signal is perfect, you might get confused by the silence. But if you practice listening through the static, you learn to ignore the noise and focus on the real words.
They added "static" (noise) to the shorthand notes and trained the student to still find the disease. This made the student more robust.
They also used distillation: They let a "super-teacher" (who looked at the full, clear images) guide the student on how to interpret the messy notes.

The Result:
This new student didn't solve the problem completely (the gap still existed), but it did two great things:

It got better at reading the notes: It narrowed the gap slightly.
It was incredibly fast: Because it was reading the tiny shorthand notes instead of the huge full images, it was 64 times faster and used 120 times less computer memory.

The Main Takeaway

The paper concludes that the biggest bottleneck in using AI to generate medical data isn't that the AI can't make good-looking fake images. The bottleneck is that the internal "language" the AI uses to store these images is structured in a way that is hard for other AI programs to learn from.

Key Insights:

Don't just polish the image: Making the fake images look more realistic (high fidelity) doesn't fix the problem.
Don't just retrain the translator: Training the translator specifically on medical data doesn't fix the problem.
Fix the structure: The real solution lies in changing how the AI organizes the information in its "shorthand notes" so it's easier for other programs to read.

Until we fix the structure of these notes, using AI to generate data for rare diseases might still leave the most critical, rare conditions hard to detect.

Technical Summary: The Learnability Gap in Medical Latent Diffusion

Problem Statement
Generative data augmentation using Latent Diffusion Models (LDMs) is proposed as a solution for class imbalance in medical imaging, particularly for long-tailed distributions where rare but critical conditions are underrepresented. However, current approaches prioritize perceptual fidelity (e.g., FID scores) and domain-specific autoencoder fine-tuning, operating under the assumption that the latent space serves as a neutral intermediary. This paper identifies a fundamental bottleneck: while large-scale pretrained autoencoders faithfully encode discriminative features (evidenced by near-lossless reconstruction), their latent representations are structured in ways that are difficult for downstream classifiers to learn from. This discrepancy, termed the learnability gap, suggests that synthetic data generated from these latents may lack the specific pathological markers required for effective classification, particularly for tail classes.

Methodology
The authors conduct a systematic study across five autoencoder families (Stable Diffusion 1.4, Flux.1 dev, Flux.2 dev, MedVAE, and MedVAE fine-tuned) and four medical benchmarks (MIMIC-CXR, ISIC 2019, CT-RATE, and Cardium). To isolate the cause of performance degradation, they define three evaluation spaces:

Image Space: Classifiers trained on original images.
Reconstruction Space: Classifiers trained on decoded images ( $\tilde{x}_{img} = G(E(x_{img}))$ ).
Latent Space: Classifiers trained directly on latent codes ( $x = E(x_{img})$ ).

If latent-space classifiers underperform reconstruction-space classifiers (which match image-space performance), the deficit is attributed to the structure of the latent space rather than information loss.

To probe and partially mitigate this gap, the authors develop noise-conditioned latent classifiers. These models:

Use a ConvNeXt-Tiny backbone operating directly on latent tensors.
Incorporate FiLM (Feature-wise Linear Modulation) layers to condition predictions on noise levels ( $\sigma$ ), forcing the network to learn robust, noise-invariant features.
Employ image-space distillation, where a strong image-space teacher (trained on reconstructions) guides the latent student via soft logit matching combined with ground-truth supervision.

Key Results

Existence of the Gap: Across all 20 autoencoder-dataset pairs, latent-space classifiers suffer a substantial performance drop compared to reconstruction-space classifiers. On ISIC-2019, the balanced accuracy (bACC) gap exceeds 18 percentage points for every autoencoder. A Wilcoxon signed-rank test confirms this is a systematic phenomenon ( $p = 9.5 \times 10^{-7}$ ).
Fidelity vs. Learnability: High reconstruction fidelity (measured by PSNR/SSIM) does not predict latent-space learnability. For instance, Flux.2 achieves the highest PSNR but exhibits one of the largest gaps, while MedVAE shows lower fidelity but a comparable gap. This rules out information loss as the primary cause.
Ineffectiveness of Fine-tuning: Domain-specific fine-tuning of the autoencoder improves reconstruction quality significantly (e.g., PSNR increase from 29.9 to 38.9 dB) but fails to close the learnability gap. In some cases, it widens the gap, confirming the issue is structural rather than a domain mismatch.
Partial Mitigation: The proposed noise-conditioned latent classifiers with distillation partially narrow the gap and improve robustness (e.g., increasing mean AUC from 0.727 to 0.754), though a residual gap of 4–11 bACC points persists.
Efficiency Gains: Despite the gap, latent classifiers offer significant practical advantages. Operating in the compressed domain yields 64× higher throughput and 120× memory reduction compared to image-space models, making them viable for real-time quality filtering in generative pipelines.

Significance and Claims
The paper claims to provide the first systematic diagnosis of the learnability gap in medical latent spaces. Its primary contribution is shifting the focus of optimization in medical generative AI:

Bottleneck Identification: The primary obstacle to closing the performance gap between real and synthetic medical data is the structure of the latent space, not reconstruction fidelity or domain specificity.
Strategic Redirection: Efforts to improve synthetic medical data should prioritize latent space restructuring (e.g., via class-conditional objectives or alignment with discriminative encoders) rather than expensive autoencoder fine-tuning.
Practical Tooling: The developed noise-conditioned latent classifiers serve a dual purpose: as diagnostic tools to evaluate latent space quality and as efficient components for rejection sampling and quality filtering in generative pipelines.

The authors conclude that while they have diagnosed the gap and provided tools to work around it, the fundamental problem of restructuring latent spaces to be inherently learnable for downstream medical tasks remains an open challenge.