🤖 AI

Rethinking FID Through the Geometry of the Reference Dataset

This paper demonstrates that the reliability of the Fréchet Inception Distance (FID) metric in evaluating image generators is fundamentally influenced by the geometric properties of the reference dataset, specifically its distributional density and effective rank, which can cause FID to worsen even as sample quality improves in dispersed datasets.

Original authors: Yunghee Lee, Byeonghyun Pak

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunghee Lee, Byeonghyun Pak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a cooking competition. Your job is to rate how good the chefs' new dishes are by comparing them to a "Gold Standard" cookbook of perfect recipes.

For years, the AI world has used a specific scoring system called FID (Fréchet Inception Distance) to judge AI image generators. The rule was simple: The lower the score, the better the AI. A low score meant the AI's pictures looked very similar to the "Gold Standard" cookbook.

However, the authors of this paper discovered a confusing glitch in the judging system. Sometimes, when they made the AI's pictures look sharper and more realistic to human eyes, the FID score actually got worse (higher). This is like a judge giving a lower score to a chef who just improved their dish, simply because the cookbook they were comparing it to was different.

The Real Culprit: The Shape of the Cookbook

The paper argues that the problem isn't the AI or the scoring math itself, but rather the geometry of the reference dataset (the "cookbook" or collection of real images used for comparison).

The authors used two main metaphors to describe these cookbooks:

The "Crowded Room" (Concentrated Datasets):
Imagine a room full of people who all look very similar (e.g., a room full of professional models). Everyone is standing close together.
- The Result: If the AI tries to make a picture of a person, it's easy for it to land in that crowded cluster. The FID score drops nicely as the picture gets better.
- Datasets like this: FFHQ and CelebA-HQ (mostly faces).
The "Huge Park" (Dispersed Datasets):
Imagine a massive park with people doing everything: skiing, swimming, eating, sleeping, and wearing all kinds of clothes. They are spread out everywhere.
- The Result: Even if the AI makes a better picture, it might accidentally move slightly away from the specific cluster of people it was aiming for, or it might fail to cover the vastness of the park. In this "Huge Park," making the picture better can actually cause the FID score to go up (worsen).
- Datasets like this: COCO, ImageNet, and Flickr30K (all kinds of objects and scenes).

The Experiment: Turning Up the Volume

To prove this, the researchers took a single AI generator and told it to create images using different settings (like turning up the "denoising" steps, which usually makes images clearer).

On the "Crowded Room" datasets: As the images got clearer, the FID score went down (improved).
On the "Huge Park" datasets: As the images got clearer, the FID score went up (got worse).

This proved that FID doesn't just measure "how good the image is." It measures "how well the image fits into the specific shape of the reference dataset."

The "Precision vs. Recall" Explanation

The paper also broke down why this happens using two concepts:

Precision (Fidelity): How accurate is the image? Does it look like a real photo?
Recall (Coverage): Does the AI cover all the different types of things in the dataset?
In Crowded Rooms, FID cares mostly about Precision. If the image looks sharp and real, the score is good.
In Huge Parks, FID cares mostly about Recall. If the AI makes a perfect image of a dog, but the dataset has 1,000 other things (cats, cars, trees) and the AI isn't covering them well, the score gets punished. You can have a perfect dog picture, but if the "park" is too big and the AI isn't exploring it enough, the score drops.

The Bottom Line

The paper concludes that you cannot trust the FID score in isolation. It's like trying to judge a swimmer's speed without knowing if they are swimming in a small pool or the open ocean.

The authors' advice:

If you are using a "Crowded Room" dataset (like faces), FID is a reliable judge.
If you are using a "Huge Park" dataset (like general scenes), you must look at the "shape" of that dataset (how spread out it is) before you trust the FID score.
Don't just report the number; report the geometry of the dataset you used to get it.

In short: A low FID score doesn't always mean a better AI; it might just mean the AI is playing in a smaller, easier playground.

Technical Summary: Rethinking FID Through the Geometry of the Reference Dataset

Problem Statement

The Fréchet Inception Distance (FID) is the de facto standard for evaluating generative image models, operating under the assumption that a lower score indicates higher sample quality. However, recent evidence suggests this assumption is unreliable. Studies have shown that improvements in perceptual quality (e.g., sharper images via increased computation) can lead to worse FID scores, and conversely, stronger image distortions can yield better FID scores. While prior work has attributed these inconsistencies to the fragility of the Inception-v3 feature extractor or limitations of the Fréchet distance itself, this paper argues that a fundamental aspect remains underexplored: the geometry of the reference dataset.

Because FID measures the distance between a generated distribution and a chosen real-image reference distribution, its behavior is inherently dependent on the structural properties of that reference set. The paper posits that the mismatch between FID trends and actual sample quality arises because different datasets occupy distinct regions of feature space, yet benchmarking practices rarely justify or account for these geometric differences.

Methodology

Geometric Characterization of Datasets

The authors characterize six reference datasets (FFHQ, CelebA-HQ, MJHQ-30K, ImageNet, Flickr30K, and COCO) using two scalar descriptors computed within the feature space used by the metric (Inception-v3):

Distributional Density ( $\langle -\log d_k \rangle$ ): A measure of how tightly the reference distribution concentrates around its points. It is calculated as the mean of the negative logarithm of the Euclidean distance to the $k$ -th nearest neighbor ( $k=80$ ). High values indicate concentrated, clustered data (e.g., face datasets), while low values indicate dispersed data.
Effective Rank ($erank(A)$): A measure of the number of linear directions the feature distribution spans. It is derived from the Shannon entropy of the singular values of the centered feature matrix. High values indicate a broad, heterogeneous support (e.g., open-domain datasets), while low values indicate a collapse onto fewer dimensions.

Experimental Design

The study employs a controlled empirical setup using Stable Diffusion 1.5 with a fixed random seed per prompt.

Independent Variable: Sample quality is manipulated by sweeping the number of denoising steps ( $N$ ) and by using ImageReward as a proxy for perceptual quality.
Dependent Variable: FID scores are calculated for generated samples against each of the six reference datasets.
Statistical Analysis: The authors use hierarchical linear models to test two hypotheses:
- Omnibus Test: Whether the slope of the relationship between sample quality and FID varies significantly across datasets.
- Moderation Test: Whether the geometric descriptors (density and effective rank) significantly explain the variance in these slopes.
Ablation Studies: The analysis is repeated using alternative feature spaces (DINOv2) and alternative distance metrics (Kernel MMD/KID) to ensure the findings are not specific to Inception-v3 or the Fréchet distance.
Decomposition: FID is decomposed into Precision (fidelity) and Recall (diversity/coverage) to diagnose which component drives the observed trends.

Key Results

1. Dataset-Dependent FID Trends

The relationship between sample quality and FID is not uniform; it reverses based on the reference dataset geometry:

Concentrated Datasets (FFHQ, CelebA-HQ): As sample quality improves (more denoising steps, higher ImageReward), FID decreases (improves). The slope is negative.
Dispersed Datasets (ImageNet, Flickr30K, COCO): As sample quality improves, FID increases (worsens). The slope is positive.
MJHQ-30K: Exhibits a trend intermediate between the two regimes.

2. Geometric Moderation

Statistical analysis confirms that dataset geometry significantly moderates the quality-FID relationship:

Distributional Density: Explains 70.7% of the between-dataset variance in slopes. Denser datasets correlate with more negative slopes (FID improves with quality).
Effective Rank: Explains 66.1% of the variance. Higher effective rank correlates with positive slopes (FID worsens with quality).

3. Precision vs. Recall Attribution

The decomposition reveals that the correlation between FID and its components shifts with dataset geometry:

Concentrated Datasets: FID is strongly correlated with Precision ( $R^2 \approx 0.95$ for CelebA-HQ).
Dispersed Datasets: FID is strongly correlated with Recall ( $R^2 \approx 0.85$ for COCO).
This suggests that for dispersed datasets, FID rewards the coverage of the real distribution (Recall) more than the fidelity of individual samples (Precision). Consequently, if increasing sample quality reduces diversity (Recall), FID may worsen even if the images look better.

4. Robustness

The observed effects persist when replacing Inception-v3 with DINOv2 and the Fréchet distance with Kernel MMD (KID). In fact, geometric descriptors explain even more variance with DINOv2 features, indicating the phenomenon is not an artifact of Inception-v3 limitations.

Contributions and Significance

The paper makes three primary contributions:

Empirical Evidence: It provides a controlled study demonstrating that reference dataset geometry (quantified by density and effective rank) significantly moderates FID behavior.
Generalizability: It shows through ablations that this dependency is not specific to Inception-v3 features or the Fréchet estimator, but is a fundamental property of distributional metrics.
Practical Guidance: The authors argue that distributional metrics cannot be interpreted in isolation. They recommend:
- Using concentrated datasets (like FFHQ) when FID is the primary evaluation criterion, as FID aligns better with precision in these settings.
- Reporting FID alongside geometric descriptors (density and effective rank) for open-domain or dispersed datasets to provide context for the metric's behavior.

Conclusion

The paper concludes that the "fragility" of FID is not solely a flaw in the metric's calculation but a consequence of the interaction between the generated distribution and the specific geometry of the reference dataset. For reliable benchmarking, researchers must understand and report the geometric properties of the reference dataset, as these properties determine whether FID will faithfully reflect improvements in sample quality or obscure them.