The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation

TL;DR

Quantifies hidden randomness in FID evaluation and proposes a new protocol for reproducible assessment.

Problem

FID is the standard metric for generative models, but most papers report a single number from one training seed and one sampling seed. The reproducibility of that number under retraining or resampling is unknown.

Approach

Treat FID as a random variable on a two-axis grid of training and generation seeds, and measure its variance directly on hundreds of SiT networks trained on class-conditional ImageNet 256x256. Decompose variance contributions and analyze the effect of classifier-free guidance tuning.

Results & Contribution

Retraining moves FID 3.2x more than resampling, driven by initialization, data ordering, and Gaussian noise in the flow-matching loss. Increasing compute or model size barely reduces spread (CoV 1-2%). Per-cell optimal guidance halves spread but reshuffles best seeds; a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Recommends evaluating under per-cell optimal guidance, treating FID gaps below ~1.3% CoV as inconclusive, and reporting error bars over several training seeds.