Fair Benchmarking of Emerging One-Step Generative Models Against Multistep Diffusion and Flow Models

1. Problem Statement

The field of text-to-image generation has seen rapid advancements in native one-step models (e.g., MeanFlow, Improved MeanFlow, SoFlow) that aim to generate images in a single inference step, drastically reducing computational costs compared to traditional multi-step diffusion or flow models. However, fair benchmarking between these emerging one-step models and established multi-step systems (e.g., Stable Diffusion 3.5, FLUX.1) is currently hindered by three critical issues:

• Mismatched Inference Protocols: Comparisons often use inconsistent settings. One-step models are typically trained and evaluated under class-conditional (ImageNet label-ID) settings, while state-of-the-art (SOTA) systems are text-conditioned.
• Hyperparameter Sensitivity (CFG): Results are heavily dependent on Classifier-Free Guidance (CFG) scales. Different studies use empirically tuned CFG values optimized for specific metrics (usually FID), making cross-paper comparisons invalid.
• Metric Limitations: The field relies heavily on Fréchet Inception Distance (FID). The authors argue that optimizing solely for FID can lead to misleading conclusions, as it often trades off against text-image alignment, diversity, and human preference, sometimes resulting in images with lower FID scores but visible structural distortions.
• Lack of OOD Evaluation: There is a scarcity of standardized Out-of-Distribution (OOD) evaluation datasets for label-ID conditioned generators beyond the standard ImageNet validation set.

2. Methodology

To address these gaps, the authors established a controlled benchmarking protocol evaluating eight models across three datasets.

A. Models Evaluated

The study covers a spectrum of architectures:

• Native One-Step Models: MeanFlow, Improved MeanFlow (iMF), SoFlow.
• Flow/Diffusion Baselines: Scalable Interpolant Transformer (SiT), Representation Autoencoder (RAE), Scale-RAE.
• SOTA Text-to-Image Systems: Stable Diffusion 3.5 Large, FLUX.1-dev.

B. Controlled Experimental Setup

• Datasets:
  1. ImageNet Validation: Standard in-distribution test.
  2. ImageNetV2: Frequency-matched test set to probe overfitting.
  3. reLAIONet (New Contribution): A new, manually proofread OOD dataset. It maps web-sourced images (from reLAION-400M) to ImageNet classes, filtering for mislabeled/ambiguous samples. This allows for OOD evaluation while maintaining the label-ID conditioning required by the one-step models.
• Inference Protocols:
  • Steps: Evaluated at 1 step (native one-step) and 25 steps (multi-step) to test scalability and iteration effects.
  • Guidance (CFG): Two settings were used:
    1. Fixed CFG = 7: Matches the default for SD-based pipelines.
    2. Native CFG: The specific CFG value reported in the original papers for each model.
• Conditioning Standardization: Text-conditioned models (SD3.5, FLUX) were prompted with "a photo of a {class name}" to match the class-conditional nature of the one-step models.

C. Evaluation Metrics & The MMHM Proposal

The authors utilize four metrics:

1. FID: Distributional similarity (Inception features).
2. Inception Score (IS): Diversity and class separability.
3. CLIP Score: Text-image alignment.
4. Pick Score: Human preference proxy.

Key Innovation: MinMax Harmonic Mean (MMHM)
Recognizing that optimizing one metric often degrades others, the authors propose MMHM as a composite proxy. It is the harmonic mean of the four normalized metrics (with a small $\epsilon$ to prevent division by zero):
$$S(m) = \frac{4}{\sum_{x \in \{FID, IS, CLIP, PICK\}} (\epsilon + \text{norm}(x, m))^{-1}}$$
MMHM is designed to stabilize hyperparameter selection (CFG and steps) by balancing realism, diversity, alignment, and preference.

3. Key Results

A. The Failure of FID-Only Optimization

• Trade-offs: Minimizing FID often leads to a systematic degradation in IS, CLIP Score, and Pick Score.
• Qualitative Mismatch: Configurations with the lowest FID often exhibit visible distortions (e.g., facial artifacts, unnatural textures) compared to configurations with slightly higher FID but higher MMHM.
• CFG Sensitivity: In few-step regimes (1–5 steps), FID is highly sensitive to CFG. Lower CFG often improves FID but harms alignment/preference, while intermediate CFG (3–9) yields the best MMHM.

B. One-Step vs. Multi-Step Performance

• Scaling: Native one-step models (MeanFlow, iMF) show substantial improvement when iterated to 25 steps, narrowing the quantitative gap with SOTA multi-step baselines.
• Degradation: Conversely, forcing multi-step systems (SD3.5, FLUX) into a 1-step regime causes a sharp performance drop, confirming they are not yet optimized for single-step inference.
• Qualitative Gap: Despite competitive quantitative scores (under MMHM), one-step models still suffer from characteristic local distortions (particularly in faces and fine structures) that persist even at 25 steps, lagging behind SOTA in perceived photorealism.

C. Generalization (reLAIONet)

• Trends observed on ImageNet and ImageNetV2 persisted on reLAIONet.
• While absolute FID scores shifted due to distribution changes (reLAIONet generally had higher FID), MMHM remained a stable criterion for selecting the best hyperparameters across datasets, whereas FID-based selection was inconsistent.

4. Key Contributions

1. Fair Benchmarking Protocol: A rigorous, controlled evaluation framework that standardizes CFG, step counts, and conditioning (class vs. text) to enable fair comparison between one-step and multi-step models.
2. reLAIONet Dataset: Introduction of a new, proofread OOD evaluation dataset aligned with ImageNet classes, enabling robust generalization testing for label-ID conditioned generators.
3. MMHM Metric: Proposal of the MinMax Harmonic Mean as a superior selection criterion over FID alone, demonstrating that multi-metric optimization yields better qualitative results and more stable hyperparameter choices.
4. Empirical Insights:
   • Demonstration that FID-optimized settings often degrade human preference and alignment.
   • Evidence that one-step models can scale to multi-step inference to become quantitatively competitive, though qualitative fidelity (especially local geometry) remains a challenge.

5. Significance and Future Directions

• Paradigm Shift: The paper argues that the community must move away from "FID-chasing" as the primary optimization target. Relying on a single metric leads to brittle models that look good statistically but fail perceptually.
• Practical Utility: MMHM provides a practical tool for researchers to select hyperparameters that balance multiple aspects of generation quality, offering a more holistic view of model performance.
• Open Challenges:
  • Qualitative Fidelity: One-step models still struggle with local distortions (e.g., faces) even when quantitative metrics are strong.
  • Metric Design: There is a critical need for a new, principled objective function that better correlates with human-perceived realism across diverse model families, rather than just aggregating existing metrics.
  • Text-Conditioning: Future work should test if these findings and scaling laws hold when one-step models are trained and evaluated under full text-conditioning rather than just class-labels.

In conclusion, the paper establishes that while native one-step models are rapidly closing the quantitative gap with SOTA systems, current evaluation practices (FID-centric) obscure significant qualitative deficiencies. A multi-metric approach (MMHM) is essential for fair benchmarking and guiding future model development.