💻 computer science

LatentDiff: Scaling Semantic Dataset Comparison to Millions of Images

This paper introduces LatentDiff, a scalable and efficient framework that leverages pretrained vision encoders and density ratio estimation to identify interpretable semantic differences between massive datasets, outperforming existing methods in accuracy and robustness even when only a tiny fraction of images differ.

Original authors: James Flora, Kowshik Thopalli, Akshay R. Kulkarni, Weng-Keen Wong, Shusen Liu

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: James Flora, Kowshik Thopalli, Akshay R. Kulkarni, Weng-Keen Wong, Shusen Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two massive photo albums, each containing millions of pictures. Your job is to figure out: "What is the one thing Album A has that Album B is missing, and vice versa?"

Usually, if you tried to do this by reading a description of every single photo, it would take forever and cost a fortune. If you just looked at the photos randomly, you might miss the tiny, important differences because they are so rare.

The paper introduces LatentDiff, a smart, fast, and cheap way to solve this puzzle. Here is how it works, broken down into simple ideas:

1. The Problem: The "Needle in a Haystack"

Most existing methods for comparing photo albums assume the differences are everywhere (like if Album A was all cats and Album B was all dogs). But in the real world, the differences are often tiny. Maybe Album A has 100 photos of dogs on surfboards, while Album B has zero. The rest of the millions of photos are identical.

Trying to find these rare "missing modes" is like looking for a needle in a haystack. If you just grab a handful of hay (random photos) and look, you probably won't find the needle.

2. The Solution: Two Superpowers

LatentDiff uses two different "superpowers" to find these needles, working together like a detective team.

Superpower A: The "Feature Dictionary" (SAE)

Think of the computer's brain (a pre-trained vision encoder) as a giant library where every photo is converted into a list of ingredients (like "dog," "beach," "sunny").

How it works: LatentDiff uses a tool called a Sparse Autoencoder (SAE) to organize these ingredients into a neat dictionary. It breaks the photos down into very specific, single-meaning concepts (monosemantic features).
The Trick: It simply counts how often each "ingredient" appears in Album A versus Album B. If "surfboard" shows up 500 times in Album A and 0 times in Album B, the system flags it immediately.
The Limit: It can only find things that are already in its dictionary. If the missing concept is something weird or new the dictionary doesn't know, it might miss it.

Superpower B: The "Spotlight" (DRE)

This is the backup plan for when the dictionary fails.

How it works: Instead of counting ingredients, this method acts like a spotlight. It scans the two albums and asks, "Which photos look the most different from the other album?"
The Trick: It finds the top 10 or 20 photos that are the "odd ones out." Once it finds these rare photos, it only then calls in a very expensive, slow AI (a language model) to write a description for just those few photos.
The Benefit: It doesn't waste time describing millions of normal photos. It only describes the weird ones, saving huge amounts of time and money.

3. The Teamwork (Ensembling)

The paper argues that using just one method isn't enough.

If you only use the Dictionary, you might miss new or weird concepts.
If you only use the Spotlight, you might miss obvious differences that are spread out but not "extreme" enough to be the top outliers.

LatentDiff combines them. It takes the list of differences found by the Dictionary and adds the descriptions found by the Spotlight. This creates a "best of both worlds" list that catches almost everything, from common differences to rare, weird ones.

4. The "Noisy-Diff" Test

To prove this works, the authors created a new test called Noisy-Diff.

Old Tests: Were like comparing a box of apples to a box of oranges. The difference was obvious and everywhere.
Noisy-Diff: Is like taking a box of 1,000 apples and secretly swapping out just 10 for pears. It's a "needle in a haystack" test.
The Result: Old methods (like VisDiff) got confused and missed the pears because they relied on random guessing. LatentDiff found the pears every time, even when they made up less than 1% of the data.

5. Why It's a Big Deal (Scale)

The paper shows that LatentDiff can handle millions of images (like the entire ImageNet dataset) in a few hours.

Old way: To compare millions of photos, you'd have to describe every single one. That would take weeks and cost a lot of computing power.
LatentDiff way: It does a quick "scan" of the whole library (which takes a few hours) and then only does the expensive "description" work on a tiny handful of photos. It's like scanning a library with a metal detector instead of reading every book cover-to-cover.

Summary

LatentDiff is a new tool that compares huge collections of images to find what's missing. It uses a dictionary to spot common differences and a spotlight to hunt down rare, weird differences. By combining these two, it finds the "needles in the haystack" that other methods miss, all while being fast, cheap, and able to handle millions of photos at once.

Technical Summary: LatentDiff

Problem Statement

The paper addresses the challenge of semantic dataset comparison: given two image datasets ( $D_A$ and $D_B$ ), the goal is to identify and describe the "missing modes"—semantic concepts, visual attributes, or scene configurations that are prevalent in one dataset but absent or underrepresented in the other.

Existing approaches face two primary limitations:

Scalability vs. Interpretability Trade-off: Caption-based methods (e.g., VisDiff) generate human-readable descriptions but require expensive per-image inference, making them infeasible for large-scale datasets. Statistical methods on embeddings scale well but often lack interpretability.
Benchmark Realism: Current benchmarks assume pervasive differences between datasets. In contrast, real-world distribution shifts are often sparse and subtle, affecting only a small fraction of images (e.g., <1% to 5%), a scenario where existing methods relying on random sampling often fail.

Methodology: LatentDiff

The authors propose LatentDiff, a framework that operates directly in the latent space of pretrained vision encoders (specifically CLIP) to compare datasets efficiently. It employs two complementary strategies that are aggregated to maximize coverage:

1. SAE-Based Concept Identification (Interpretability)

Mechanism: The method utilizes Sparse Autoencoders (SAEs) to decompose CLIP embeddings into sparse, approximately monosemantic features. These features act as a fixed dictionary of visual concepts.
Process:
- Images are embedded and projected into the SAE latent space.
- Activation distributions for each neuron (concept) are compared between $D_A$ and $D_B$ using Jensen-Shannon Divergence (JSD).
- Concepts are ranked by divergence magnitude and signed mean gap to determine which are over-represented in one dataset versus the other.
Strengths: Provides direct, interpretable labels for differences without generating new text.
Limitations: Restricted to the SAE's learned vocabulary; may miss concepts not represented in the dictionary.

2. DRE-Based Localization (Open-Vocabulary Discovery)

Mechanism: To address vocabulary gaps and sparse shifts, the method uses Density Ratio Estimation (DRE).
Process:
- A lightweight classifier is trained in the latent space to estimate the density ratio $r(h) = p_A(h) / p_B(h)$ .
- Samples are ranked by their log density ratio to identify the most discriminative images (those most characteristic of $D_A$ vs. $D_B$ ).
- Only the top-ranked, highly discriminative samples (e.g., top 10) are selected for targeted captioning (using BLIP-2) and aggregation (using an LLM) to generate natural language descriptions.
Strengths: Effective for rare, sparse, or novel concepts outside the SAE vocabulary; robust to noise.
Limitations: Requires downstream captioning/LLM steps, though applied only to a tiny subset of data.

3. Aggregation Strategy

The final output is a union of hypotheses from both branches. The system combines the top-ranked SAE-derived concepts with the top-ranked DRE-derived captions. This ensembling strategy ensures robustness: if a concept is missing from the SAE dictionary, DRE can still surface it via targeted sampling.

Key Contributions

LatentDiff Framework: A scalable approach for interpretable dataset comparison that operates in the latent space of pretrained encoders, avoiding the computational cost of per-image captioning for the entire dataset.
Noisy-Diff Benchmark: A new benchmark constructed from MS-COCO and ImageNet to evaluate methods under sparse, realistic distribution shifts. Unlike prior benchmarks where differences are pervasive, Noisy-Diff introduces "needle-in-a-haystack" scenarios where missing modes constitute as little as 1% of the data.
Empirical Validation: Demonstration that LatentDiff achieves superior accuracy compared to caption-based baselines (VisDiff) while operating at a fraction of the computational cost.

Experimental Results

The paper evaluates LatentDiff against VisDiff (the current state-of-the-art) on the Noisy-Diff benchmark.

Performance on Sparse Shifts:
- LatentDiff (SAE) achieved an average cosine similarity of 0.548 vs. VisDiff's 0.298 on a subset of 15 pairs per category.
- LatentDiff showed robustness to scarcity: As missing modes became rarer (from 50% down to <5%), VisDiff's performance degraded significantly (negative correlation with scarcity), whereas LatentDiff's performance remained stable.
Complementarity:
- The combined approach (SAE + DRE) achieved an average cosine similarity of 0.678, representing a 23% improvement over using either method in isolation.
- Qualitative analysis showed that DRE successfully recovered missing modes (e.g., "surfboards" with dogs) when the SAE lacked the corresponding neuron.
Scalability:
- Efficiency: On equivalent hardware, LatentDiff took 0.96 hours compared to VisDiff's 5.4 hours for the same task.
- Large-Scale Application: The method was tested on the ImageNet training set (>1 million images). LatentDiff successfully identified missing modes (e.g., specific building types) with a cosine similarity of 0.41. The embedding pass took ~10 hours, with subsequent comparison taking only minutes, demonstrating linear scalability.

Significance and Claims

The paper claims that LatentDiff represents a significant step forward in making semantic dataset comparison scalable to millions of images while maintaining interpretability.

Efficiency: By leveraging the geometric structure of pretrained latent spaces and sparse autoencoders, the method avoids the prohibitive cost of generating captions for every image.
Robustness to Real-World Noise: The introduction of Noisy-Diff highlights that existing methods are brittle when differences are sparse. LatentDiff's hybrid approach (SAE for breadth, DRE for targeted depth) specifically addresses this "needle-in-a-haystack" problem.
Practical Utility: The framework allows practitioners to detect subtle distribution shifts (e.g., generative model failures on specific categories) in massive datasets without requiring prior knowledge of what those shifts might be.

The authors note that the approach is dependent on the quality and coverage of the underlying SAEs and that future work could involve training task-specific SAEs or integrating human-in-the-loop feedback. However, the current results establish a new baseline for efficient, large-scale semantic analysis.