🤖 machine learning

Beyond Generative Priors: Minority Sampling with JEPA-Guided Diffusion

This paper introduces JEPA guidance, a diffusion sampling framework that leverages Joint-Embedding Predictive Architecture (JEPA) world models to define and generate minority samples based on real-world semantic rarity rather than model-specific densities, thereby improving the fidelity and validity of generated low-density instances across various generation tasks.

Original authors: Sol Park, Soobin Um

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sol Park, Soobin Um

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Echo Chamber" of AI Art

Imagine you ask an AI artist to draw a picture of a "rare animal." Most AI models today are like echo chambers. They have been trained on millions of photos from the internet. If they see 1,000 photos of a dog on a white background, they think that's the "normal" way a dog looks. If you ask for a "rare" dog, the AI might just draw a dog with a weird color or a strange pose, but it's still just a dog on a white background.

The paper argues that current AI methods for finding "rare" things are limited because they only look for things that are rare inside the AI's own memory. They don't understand what is actually rare in the real world. For example, a "stealth aircraft" is genuinely rare in the real world, but if the AI has never seen one, it won't know to draw it as "rare." It might just draw a very blurry dog instead.

The Solution: A New Guide (JEPA)

The authors propose a new way to teach the AI what "rare" really means. They introduce a new guide called JEPA (Joint-Embedding Predictive Architecture).

Think of the AI drawing process (Diffusion) as a hiker trying to find a hidden cave in a vast, foggy mountain range.

The Old Way: The hiker only has a map of the area they just walked through (the training data). They look for a cave that is hidden relative to the path they just took. They might find a small hole in a familiar rock, but they miss the massive, unique cave system that exists elsewhere in the world.
The New Way (JEPA Guidance): The hiker is now given a satellite view of the entire planet (the "World Prior"). This satellite view was built by a super-smart observer (JEPA) who has seen billions of images and understands the true geography of the world. This guide tells the hiker: "Don't look for a hole in the rock you're standing on. Look for the place where the terrain is truly unique and untouched, like a stealth aircraft in a sky full of birds."

How It Works: The "Low-Density" Hunt

In the world of AI, "dense" areas are where most things live (like a crowded city square). "Low-density" areas are the empty, quiet places (like a desert or a secret base).

The Goal: The paper wants the AI to generate images from these "low-density" areas that are genuinely rare in the real world, not just rare in the AI's training set.
The Trick: The AI uses the JEPA guide to check its work as it draws. At every step of the drawing process, it asks the JEPA guide: "Is this image common or rare in the real world?"
The Steering: If the image is becoming too "common" (like a standard dog), the guide pushes the drawing toward the "rare" direction (like a dog with wings or a stealth plane). It steers the AI away from the crowded city square and toward the quiet, unique corners of the world.

Making It Fast: The "Shortcut"

Calculating this "real-world rarity" is usually very slow and heavy, like trying to count every single grain of sand on a beach to find a specific one. The paper introduces a mathematical shortcut (using something called Randomized SVD and the Envelope Theorem).

Imagine you need to weigh a giant boulder. Instead of lifting the whole thing, you take a few smart measurements of its surface and use a formula to estimate the weight with incredible accuracy. This allows the AI to use the "World Prior" guide without slowing down the drawing process too much.

The Results: Better, Stranger, and More Real

The paper tested this on several tasks:

Drawing faces: Instead of just making weird-looking faces, it generated faces that were genuinely unique (e.g., specific rare features) rather than just messy.
Drawing animals: It successfully generated things like "stealth aircraft" or "ostriches," which are rare in the real world, whereas other methods just made weird-looking dogs.
Text-to-Image: When asked to draw specific scenes, it created images that were both accurate to the text and surprisingly unique, avoiding the "boring average" look.

Why This Matters

The authors show that by using a "World Prior" (JEPA) instead of just the "AI's Memory" (Generative Prior), we can create AI art that feels more surprising and authentic. It's the difference between an AI that only knows what it has seen before, and an AI that understands the broader, stranger reality of the world.

In short: This paper teaches AI how to stop looking for rare things in its own backyard and start exploring the whole world to find truly unique and interesting images.

Problem Statement

Minority sampling aims to generate low-density instances on a data manifold, which is critical for applications like medical diagnosis, anomaly detection, and creative AI. However, existing approaches define "minority" relative to the generative prior learned by the model itself (i.e., the distribution of the training data). This creates a fundamental limitation: samples identified as "rare" are often merely artifacts of the specific training set or the model's inductive biases, rather than genuinely atypical instances in the real world. For example, a model trained on standard datasets might consider a dog on a white background "rare" simply because it is underrepresented in the training data, whereas a stealth aircraft might be genuinely rare in the real world but not captured as such if the model's prior is narrow. The paper argues that current methods fail to align with real-world priors, which are necessary for capturing true semantic rarity.

Methodology: JEPA Guidance

The authors propose JEPA Guidance, a framework that shifts minority sampling from a generator-centric perspective to a world-centric one. This is achieved by guiding diffusion models using the implicit density of a Joint-Embedding Predictive Architecture (JEPA), a class of world models (e.g., DINOv2) known for encoding broad, semantically rich representations.

Core Mechanism:

World-Centric Definition: Instead of minimizing the probability under the diffusion model's density $p_\theta(x)$ , the method seeks to minimize the probability under the JEPA-induced density $p_\phi(x)$ . Minority samples are defined as those residing in low-density regions of the JEPA representation space.
JEPA-SCORE: The method utilizes JEPA-SCORE, a density measure derived from the Jacobian of a pretrained JEPA encoder $f_\phi$ . Specifically, it is the sum of the logarithms of the singular values of the Jacobian $J_f(x)$ . Lower JEPA-SCORE values indicate lower density (higher atypicality) in the world prior.
Approximation via Randomized SVD: Computing the full Jacobian and its Singular Value Decomposition (SVD) at every diffusion step is computationally prohibitive. The authors employ Randomized SVD to efficiently approximate the top- $k$ singular values of the compressed Jacobian. They provide a theoretical upper bound on the approximation error, decomposing it into randomized SVD estimation error and truncation error.
Envelope Theorem for Efficiency: To avoid backpropagation through the inner optimization loop of the randomized SVD (which depends on the input $x_t$ ), the authors invoke the Envelope Theorem. This allows them to treat the optimal projection matrix $Q^*$ as a constant during gradient computation, significantly reducing memory usage and computational overhead while preserving correct first-order gradients.
Deferred Guidance: To address the domain gap between the noisy intermediate latents of the diffusion model and the clean inputs expected by the JEPA encoder, the authors introduce deferred guidance. JEPA guidance is only applied after an intermediate timestep $\tau T$ (e.g., $t=0.8T$ ), ensuring that the denoised estimates $\hat{x}_{0|t}$ are sufficiently clear for the JEPA encoder to evaluate effectively. This also enables the method to extend to conditional generation (e.g., text-to-image), as the condition is established early in the process before the world-centric guidance begins.

Key Contributions

World-Centric Minority Sampling: A paradigm shift redefining minority samples based on real-world priors (via JEPAs) rather than the narrow priors of the generative model.
JEPA Guidance Framework: A principled diffusion sampling method that steers trajectories toward low-density regions under JEPA-induced priors.
Efficient Computation: The development of principled approximation strategies (Randomized SVD) and the application of the Envelope Theorem to make JEPA guidance computationally practical, accompanied by theoretical error bounds.
Training-Free Integration: The approach is entirely training-free, requiring only pretrained diffusion and JEPA models, and can be integrated at inference time without modifying the underlying models.

Experimental Results

The authors evaluated JEPA guidance across unconditional, class-conditional, and text-to-image generation tasks (using CelebA, ImageNet, Stable Diffusion 1.5, and SDXL-Lightning).

Atypicality: JEPA guidance consistently achieves the lowest JEPA-SCORE (indicating higher atypicality under world priors) compared to baselines like Sehwag et al. (2022), MG, SGMS, and MinorityPrompt.
Fidelity and Validity: Despite targeting rare regions, the method maintains high fidelity. In text-to-image tasks, it achieves competitive text-alignment scores (CLIPScore, PickScore, ImageReward) while generating semantically distinct samples (e.g., elderly military personnel, African-American baseball players) that other methods fail to produce.
Qualitative Comparison: Visualizations show that generator-centric methods often produce samples with atypical contexts (e.g., an eagle near a crane), whereas JEPA guidance produces samples with atypical semantics or behaviors (e.g., mating eagles, unusual animal shapes), aligning better with real-world rarity.
Downstream Utility: In a data augmentation task for classifier training, augmenting with JEPA-guided minority samples yielded superior or comparable performance to other methods using fewer samples, suggesting these samples provide more informative training signals.

Significance and Claims

The paper claims that JEPA guidance offers a robust solution to the limitation of existing minority samplers being trapped within the generative model's own distribution. By leveraging world models (JEPAs), the method generates samples that are genuinely rare in the real world, not just rare relative to a specific dataset.

The authors emphasize that this approach:

Decouples the notion of minority from the generative prior.
Bridges the gap between generative modeling and world modeling, a key component in the pursuit of artificial general intelligence.
Provides a practical, training-free tool for applications requiring diverse and atypical data, such as robustness testing, fairness improvement, and creative content generation.

The authors also acknowledge a potential dual-use concern: reversing the guidance direction could reinforce existing biases in world priors, and they encourage responsible use. They note that while the method introduces computational overhead compared to standard samplers, it remains faster than some complex baselines (like MinorityPrompt) and offers a reasonable trade-off for the significant gain in world-centric atypicality.