💻 computer science

Fréchet inception distance under finite sampling between real image sets

This paper demonstrates that the Fréchet Inception Distance (FID) exhibits a systematic dependence on sample size and yields non-zero values even between subsets of real images from the same distribution, thereby questioning its reliability for evaluating generative models in practical, finite-sample scenarios.

Original authors: Jessica E. Goldring, Elizabeth A. Cooke, Carlijn Roozemond, Ruben van Engen, Spencer A. Thomas

Published 2026-10-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jessica E. Goldring, Elizabeth A. Cooke, Carlijn Roozemond, Ruben van Engen, Spencer A. Thomas

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, computers are learning to create images that look startlingly real. From generating portraits of people who never existed to designing new architectural landscapes, these systems are becoming powerful tools for artists and scientists alike. However, this progress brings a critical question: how do we know if the computer is doing a good job? To answer this, researchers rely on a standard measuring stick called the Fréchet inception distance. Think of this metric as a sophisticated ruler that compares two groups of pictures. It works by translating every image into a complex list of numbers that describe its visual features, such as shapes, textures, and colors. The system then calculates the distance between the average list of numbers for a set of real photos and the average list for a set of computer-generated photos. If the two groups are very similar, the distance is small; if they are different, the distance is large. For years, this method has been the gold standard for judging whether an artificial intelligence is successfully mimicking reality.

A team of researchers from the National Physical Laboratory in the United Kingdom and the Dutch Expert Centre for Screening decided to test the reliability of this ruler. Their goal was not to compare real and fake images, but to see what happens when the ruler measures two groups of real images against each other. Logically, if you take two different handfuls of real photographs from the same world, the distance between them should be zero, or at least so small as to be negligible. If the ruler shows a large gap between two sets of genuine photos, it suggests the tool itself might be flawed. To investigate this, the researchers gathered a diverse collection of images, ranging from simple handwritten digits and common objects to complex medical scans like skin lesions and mammograms. They split these large collections into smaller, balanced groups and measured the distance between them, carefully tracking how the results changed as they added more or fewer pictures to each group.

The results revealed a surprising and persistent pattern. Even when comparing two sets of real images drawn from the exact same source, the measuring tool consistently reported a non-zero distance. This gap did not disappear simply by adding more images. In fact, the researchers found that the reported distance decreased only very slowly as the number of images grew. When they used mathematical methods to predict what would happen if they had an almost infinite number of images—billions or even trillions—the tool still predicted a noticeable distance between the two sets of real photos. This finding suggests that the tool is not perfectly calibrated to recognize that two groups of real images are, in essence, the same. The researchers observed that this behavior varied depending on the type of image. For simpler, well-known datasets, the distance dropped in a predictable way, but for more complex, real-world images like medical scans, the drop was much slower and less consistent.

A particularly troubling discovery was that the tool often gave lower scores for images that were quite different from what it had been trained on. The system was originally trained on a massive database of natural photographs, such as cats, cars, and landscapes. When the researchers tested it on medical images, which look very different from those natural photos, the tool reported that the two groups of medical scans were more similar to each other than the tool reported for groups of natural photos. The researchers suspect this is because the tool's internal "vision" is not optimized for medical data, causing it to blur the unique details of those images into a generic, homogenized view. This creates a risk where a poor-quality artificial image might receive a good score simply because it shares the same lack of detail as the real medical images, misleading developers into thinking their system is performing well when it is not.

The study also examined the mathematical stability of the calculation, specifically looking at what happens when the number of images is smaller than the number of features the system analyzes. There is a theoretical point where the calculation becomes unstable, but the researchers found that the tool's output did not show a dramatic shift or warning sign when crossing this threshold. The distance values and their variability remained consistent regardless of whether the calculation was mathematically "under-determined" or fully stable. This lack of a clear warning signal means that users might not realize their results are based on shaky mathematical ground when working with smaller datasets.

Ultimately, the research challenges the assumption that this measuring tool provides a perfect, unbiased view of image quality. The fact that it registers a significant difference between two sets of real images implies that it may not be sensitive enough to distinguish between a truly high-quality artificial image and a mediocre one, especially in specialized fields like medicine. The authors conclude that while the tool remains a useful benchmark, its scores should be interpreted with caution. The distance it measures is not just a reflection of how well an artificial intelligence is copying reality, but also a reflection of the tool's own limitations and the specific nature of the images being tested. For scientists and developers, this means that a low score does not guarantee perfection, and a high score does not always mean failure, particularly when the images being generated fall outside the realm of the natural world the tool was designed to understand.

Problem Statement
The Fréchet Inception Distance (FID) is the current standard metric for quantifying the similarity between two sets of images, predominantly used to evaluate the quality of synthetically generated images against a reference set of real images. The underlying assumption is that the FID can robustly measure the similarity of sample distributions drawn from the same population; if two sets are identical (e.g., real images from the same distribution), the FID should ideally be zero. However, there is a lack of verification regarding whether the FID is a robust measure between two sets of real images drawn from the same distribution. Furthermore, the FID is known to be a biased estimator sensitive to sample size, and its reliance on an Inception-v3 model pre-trained on natural images (ImageNet1K) may render it sub-optimal for non-natural or out-of-distribution data, such as medical imaging. This study addresses the gap in understanding how FID behaves when applied to real-world image sets of varying sizes and distributions.

Methodology
The authors evaluated the FID metric by applying it to sets of real images partitioned from a single superset, effectively creating a "best-case" scenario where the FID should theoretically approach zero. The study utilized three benchmark datasets (MNIST, CIFAR-10, ImageNet1K) and three real-world medical imaging datasets (HAM-10,000, PathMNIST, and a mammography dataset from the Dutch Expert Centre for Screening).

The experimental design involved two primary approaches:

Class-Wise FID: For datasets with class labels, the authors split each class into two equal-sized subsets and computed the FID between them.
All Sample FID: Multiple classes were combined to create larger, balanced subsets, and the FID was computed between these aggregated sets.

Experiments were conducted across a range of set sizes, starting from 50 or 100 images up to 30,000 images per subset. To assess uncertainty, the splitting process was repeated five times to calculate mean and variance. The authors also analyzed the rank and condition number of the covariance matrices used in the FID calculation to determine numerical stability, particularly in the "under-determined" regime where the number of images is less than the feature dimensionality (2048) of the Inception-v3 model. Finally, following the recommendation of Chong et al., the authors extrapolated FID values to extremely large set sizes ( $10^9$ and $10^{10}$ ) by fitting a power law to the relationship between FID and the inverse of the sample size ( $1/N$ ).

Key Results

Sample Size Dependence: FID values decrease approximately linearly with the inverse of the number of images on a log-log scale. However, the rate of decrease (gradient) varies significantly by dataset. Simple benchmark datasets like MNIST and CIFAR-10 exhibited super-linear and linear gradients, respectively, while real-world and out-of-distribution datasets (HAM-10,000, PathMNIST, Mammography) showed sub-linear gradients.
Non-Zero Baseline: Even when comparing extremely large subsets of real images from the same distribution, the extrapolated FID values ( $FID^*$ ) remained appreciably larger than zero for most datasets (except MNIST and CIFAR-10, where the extrapolated values were extremely small but non-zero).
Out-of-Distribution Effects: The FID was generally lower for out-of-distribution data (medical images) compared to in-distribution data (ImageNet1K) at the same sample size. The authors hypothesize this is due to the sub-optimal feature embedding of the Inception-v3 model for non-natural images, which effectively homogenizes the latent features.
Covariance Matrix Properties: A clear step change in the conditioning of the covariance matrix was observed when transitioning from an under-determined problem (sample size < 2048) to a determined one. However, this transition did not correspond to a measurable difference in the FID value, its variance, or the rank of the covariance matrix. The rank behavior varied by dataset, with some approaching full rank quickly and others continuing to increase with sample size.
Inconsistency Across Classes: For sample-wise assessments, FID values at the same subset size could differ by a factor of two or three between different classes within the same dataset, indicating that FID values are not directly comparable between different datasets or even different classes within a dataset.

Significance and Claims
The paper claims that the FID metric has limitations when used to measure the similarity of real and generated images in practical applications. The observation that FID values do not converge to zero even for massive sets of real images from the same distribution questions the effectiveness of FID as a definitive measure of similarity for real-world image set sizes.

The authors conclude that the FID is not a robust measure for assessing the quality of synthetic images if the assumption that "real images from the same distribution yield a zero FID" is false. Additionally, the finding that out-of-distribution data yields lower FID values suggests that low FID scores in medical or specialized imaging contexts may be misleading, potentially resulting from feature homogenization rather than true image quality. The study highlights that FID values are highly dependent on dataset characteristics, sample size, and the specific distribution of the data relative to the pre-trained feature extractor, urging caution in their interpretation.

More like this