💻 computer science

The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation

This paper reveals that the Frechet Inception Distance (FID) exhibits significant hidden randomness primarily driven by training seeds rather than sampling variations, prompting a recommendation to report FID with error bars and treat small performance gaps as statistically inconclusive.

Original authors: Nicolas Dufour, Alexei A. Efros, Patrick Pérez

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nicolas Dufour, Alexei A. Efros, Patrick Pérez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a cooking competition. Every chef (a generative AI model) presents a dish, and you give them a score based on how delicious it looks and tastes. In the world of AI image generation, this score is called FID (Fréchet Inception Distance). The lower the score, the better the dish.

For years, the community has treated this score like a perfect, unchangeable fact. If Chef A gets a score of 34.0 and Chef B gets 33.5, everyone assumes Chef B is definitively better.

This paper, "The FID Lottery," argues that this is a dangerous illusion. The authors claim that the score you see isn't just a measure of the chef's skill; it's also a measure of luck.

Here is the breakdown of their findings using simple analogies:

1. The Two Lotteries

The authors say that every time an AI generates an image, two different "lotteries" are being played:

The Training Lottery (The Big One): Before the chef even starts cooking, they roll the dice on three things:
1. The Ingredients: How the data is shuffled and ordered.
2. The Pantry Setup: How the AI's brain (weights) is initialized at the start.
3. The Cooking Process: A specific type of "noise" (random static) is added to the recipe at every single step of training.
- The Result: Even if two chefs follow the exact same recipe, the one who got the "lucky" dice roll during training will end up with a slightly different (and often better) dish.
The Generation Lottery (The Small One): Once the dish is cooked, the chef has to plate it. They have to pick a random starting point for the final garnish.
- The Result: If you ask the same chef to plate the dish 10 times, the scores will vary slightly, but not much.

The Big Discovery: The authors found that retraining the model (playing the Training Lottery again) changes the score 3.2 times more than just re-plating the same dish (playing the Generation Lottery).

2. The "Hidden" Noise Floor

The paper reveals that there is a "noise floor" for these scores.

Imagine the score is a thermometer. The authors found that the temperature fluctuates naturally by about 1% to 2% just because of luck, even if the chef does nothing different.
The Problem: Many recent AI papers claim to have improved the score by tiny amounts (e.g., going from 34.0 to 33.8). The authors argue that if the improvement is smaller than this 1–2% "luck gap," it might not be a real improvement at all. It might just be that the lucky dice rolled in their favor that time.

3. Bigger Isn't Always Better (Regarding Luck)

You might think that if you build a bigger, more powerful AI (a bigger kitchen), the luck factor would disappear.

The Finding: No. Whether the AI is small or huge, the "luck gap" stays roughly the same percentage (1–2%).
The Analogy: It's like rolling dice. Whether you roll a single die or a thousand dice, the randomness is still there. Making the model bigger doesn't make the dice less random.

4. The "Golden Ticket" (Luck of the Draw)

The authors found that some training runs are just incredibly lucky.

The Finding: A "lucky" training seed (a lucky start) can reach the same high-quality score as an "unlucky" seed, but it can do it twice as fast.
The Implication: If a researcher claims their new method made training 2x faster, they might just be comparing their "unlucky" old method against a "lucky" new run. They might not have actually improved the code; they just got lucky with the dice.

5. Tuning the Guidance (The "Secret Sauce")

The paper also looked at a setting called "Classifier-Free Guidance" (CFG), which is like a dial that tells the AI how strictly to follow a prompt.

The Finding: If you tune this dial perfectly for every single training run, you can reduce the noise gap by half.
The Catch: Doing this changes the rankings. The "lucky" seed that was #1 before might drop to #5 after you tune the dial. It's like if you adjusted the oven temperature for every single cake; the one that was best at 350°F might not be the best at 360°F.

The New Rules of the Game

Based on these findings, the authors suggest a new way to report results so we stop being fooled by luck:

Don't trust a single number: Don't just report one score. Report an "error bar" (a range) based on running the training multiple times with different seeds.
Ignore tiny wins: If a new method only improves the score by less than ~1.3%, treat it as "inconclusive." It's likely just noise.
Tune the dial: If you are using guidance, tune it specifically for each run, but remember that this changes which runs are considered "best."

In summary: The paper tells us that in the world of AI image generation, luck plays a massive role. We have been treating random fluctuations as scientific breakthroughs. To know if a new method is actually better, we need to run the experiment many times and see if the improvement holds up against the "noise" of the lottery.

Technical Summary: The FID Lottery: Quantifying Hidden Randomness in Generative Model Evaluation

Problem Statement

The Fréchet Inception Distance (FID) serves as the de facto standard for evaluating image generation models. However, the community typically reports a single FID number derived from a single trained model using a single sampling seed. This paper argues that such single-number reporting obscures a significant "reproducibility gap" caused by the inherent randomness in the training and generation pipelines.

The authors identify two distinct "lotteries" that introduce variance into FID scores:

The Training Lottery: Occurs once per training run, driven by random weight initialization, data ordering (minibatch shuffling), per-step Gaussian noise injected by the flow-matching loss, and hardware non-determinism (bitwise drift in multi-GPU execution).
The Generation Lottery: Occurs during sampling, driven by the random initial noise ( $x_T \sim \mathcal{N}(0, I)$ ) drawn for each generated image.

Conventional practice often mitigates the generation lottery by averaging over multiple sampling seeds but ignores the training lottery. The paper posits that the variance introduced by retraining a model (even with the same recipe) is significantly larger than the variance introduced by resampling from a fixed model, rendering single-seed FID claims potentially inconclusive.

Methodology

The authors treat FID as a random variable defined over a two-axis panel of $N$ independent training seeds and $K$ sampling seeds.

Experimental Setup: The study utilizes Scalable Interpolant Transformers (SiT) trained on class-conditional ImageNet (256×256) using conditional flow matching.
Scale: Experiments cover several hundred SiT networks across four model sizes (S, B, L, XL) and training budgets up to 2 million steps.
Variance Decomposition: The authors systematically isolate sources of randomness by creating controlled conditions where only one source varies at a time (e.g., varying only initialization, only data order, or only the per-step training noise) while holding others constant.
Metrics:
- $\sigma_{within}$ : Variance across sampling seeds for a fixed trained model.
- $\sigma_{between}$ : Variance across the means of different training seeds.
- Coefficient of Variation (CoV): $\sigma/\mu$ , used to normalize variance across different FID magnitudes (e.g., guided vs. unguided).
GS-FID Protocol: The authors introduce a "Golden-Section FID" protocol where Classifier-Free Guidance (CFG) scales are individually tuned for every (training seed, sampling seed) pair using golden-section search to minimize FID per cell.

Key Results

1. Training Variability Dominates Evaluation Variability

Retraining a model with a different seed moves the FID score 3.2× more than redrawing samples from a fixed network.

In a SiT-B/2 experiment, the between-seed standard deviation ( $\sigma_{between} \approx 0.44$ ) was 3.2 times larger than the within-seed standard deviation ( $\sigma_{within} \approx 0.14$ ).
The CoV for training seeds is approximately 1.3%, whereas the CoV for sampling seeds is only ~0.4%.
Increasing the sampling budget (more $K$ ) reduces the within-seed jitter but leaves the dominant between-seed envelope untouched.

2. Sources of Training Variance

The training lottery is driven by three primary factors, ordered by contribution to variance:

Per-step Training Noise: The fresh Gaussian noise injected at every gradient step by the flow-matching loss is the largest contributor (reproducing 77% of the baseline variance).
Random Initialization: Contributes significantly (67% of baseline).
Data Ordering: Contributes the least but introduces a distinct distribution shape (tight bulk with a long upper tail), suggesting occasional "bad runs" rather than continuous variance.

Hardware Non-Determinism: Bitwise drift in multi-GPU execution was found to be negligible, contributing less variance than the sampling lottery itself.

3. The 1–2% CoV Floor is Scale-Invariant

Increasing compute (training steps) or model size (S to XL) does not eliminate the variance.

The CoV remains stable within a 1–2% band (median 1.30%) across all model sizes and training checkpoints.
Larger models do not yield proportionally tighter FID spreads; the relative noise floor is a property of the metric and loss function, not the scale.
Rank stability (correlation of seed rankings over time) is weak before ~1M steps, meaning early checkpoint rankings are poor predictors of final performance.

4. The Impact of Classifier-Free Guidance (CFG)

GS-FID: Tuning CFG individually for each seed pair (GS-FID) halves the relative spread (CoV drops from 1.26% to 0.67%).
Rank Reshuffling: However, this tuning reshuffles which seeds perform best. The Spearman correlation between unguided and guided rankings is only 0.73. A seed selected as "best" via unguided FID is not reliably the best under optimized guidance.
Learning Rate (LR) Windows: Under GS-FID, the optimal LR is not a single point but a flat "window" (approx. 1.7× wide) where adjacent LRs yield indistinguishable FID scores.

5. The "Luck of the Draw"

The choice of training seed significantly impacts efficiency. A "lucky" seed can reach the same FID target as an "unlucky" seed with up to 2× less compute (e.g., reaching a target at 1M steps that an unlucky seed only reaches at 2M). This implies that single-seed papers claiming speedups may simply be benefiting from a favorable seed rather than architectural improvements.

Significance and Recommendations

The paper concludes that the current practice of reporting a single FID number is insufficient for rigorous evaluation. The authors propose a new evaluation protocol:

Report Error Bars: Instead of a single number, report FID over several training seeds (e.g., 5–10) to capture the training lottery.
Inconclusive Thresholds: Treat any FID gap below the empirically measured ~1.3% CoV (or ~2% to be safe) as inconclusive. Claims of improvement must exceed this noise floor to be considered real.
Per-Cell Optimization: When using CFG, employ per-cell optimal guidance (GS-FID) rather than a fixed global value, acknowledging that this changes the ranking of seeds.
Avoid Cherry-Picking: Recognize that selecting the single best seed from a large pool can manufacture headline-grabbing improvements that are merely statistical artifacts of the seed lottery.

The authors emphasize that these findings are specific to the SiT/Flow-Matching/ImageNet/Inception-V3 combination but suggest that the "1–2% floor" may be a general characteristic of modern generative evaluation metrics, urging the community to adopt more robust statistical reporting standards.