Technical Summary: ConsistentRFT

1. Problem Statement

Reinforcement Fine-Tuning (RFT) on flow-based generative models is essential for aligning image generation with human preferences. However, existing flow-based RFT methods (e.g., DDPO, DPO, GRPO variants) frequently introduce visual hallucinations. These manifest in three primary ways:

1. Detail Over-Optimization: The generation of irrelevant details, over-sharpening, and grid artifacts.
2. Semantic Misalignment: A disconnect between the generated image and the text prompt.
3. Cross-Step Inconsistency: Semantic degradation and divergence when using few-step sampling.

The authors identify that current methods fail because they operate under a unified perspective of Exploration (SDE rollouts) and Exploitation (policy gradient updates) that is fundamentally flawed for flow models:

• Limited Exploration: Stochastic Differential Equation (SDE) rollouts often yield a narrow exploration domain. When using the same initial noise with varying process noise, models generate highly similar images that differ only in local details. This leads to fine-grained optimization that over-emphasizes local textures while neglecting global semantics.
• Trajectory Imitation Distortion: Policy gradient methods (like GRPO) are reinterpreted as reward-weighted trajectory imitation. They compel the model to imitate stochastic, non-linear SDE trajectories to maximize rewards. This conflicts with the pre-training objective of learning a consistent, deterministic velocity field, thereby distorting the model's foundational vector field and causing instability.

2. Methodology: ConsistentRFT

To mitigate these hallucinations, the authors propose ConsistentRFT, a general framework compatible with online RFT methods (DDPO, DPO, GRPO). It consists of two core components:

A. Dynamic Granularity Rollout (DGR)

This mechanism addresses the Exploration limitation by balancing global semantics and local details through dynamic noise scheduling.

• Inter-Group Schedule: The training process is divided into periods containing both fine-grained and coarse-grained rollouts.
  • Fine-Grained: Uses progressive noise injection (varying process noise) to refine local details.
  • Coarse-Grained: Uses different initial noise sources to explore global semantic variations.
  • This prevents the model from getting stuck in local optima focused solely on details.
• Intra-Group Schedule: To enhance diversity within fine-grained groups without excessive computation, the method employs progress-aware rollout with clustering.
  • It performs an intermediate single-step prediction to assess the state.
  • Samples are clustered in latent space to select representative groups (one for detailed refinement, one for coarse perception), ensuring diversity while reducing computational overhead.

B. Consistent Policy Gradient Optimization (CPGO)

This mechanism addresses the Exploitation distortion by preserving the model's intrinsic consistency.

• Core Insight: Instead of purely imitating stochastic SDE trajectories, CPGO enforces single-step prediction fidelity via Ordinary Differential Equation (ODE) constraints.
• Implementation: It aligns the current model's prediction at step $t$ with the old model's prediction at step $t-1$ using an ODE-based solver.
  • The objective function adds a consistency term: $J_{CPGO} = -\mathbb{E}[\|\pi^{ODE}_{\theta}(x_t, t) - \pi^{ODE}_{\theta_{old}}(x_{t-1}, t-1)\|^2]$.
• Effect: This constrains the optimization to follow smoother, more consistent trajectories, preventing the model from imitating hallucinated samples interpolated from the pre-trained distribution.

C. Visual Hallucination Evaluator (VH-Evaluator)

The authors introduce a new evaluation metric combining low-level objective metrics (e.g., Laplacian variance, grid pattern detection) and high-level assessments via pre-trained Multimodal Large Language Models (MLLMs) to specifically quantify detail over-optimization and perceptual hallucinations.

3. Key Contributions

1. Analysis of Hallucination Causes: The paper provides a preliminary analysis attributing visual hallucinations in flow-based RFT to two specific design choices: limited exploration domains in SDE rollouts (leading to detail over-optimization) and the distortion of the vector field caused by trajectory imitation in policy gradients.
2. Dynamic Granularity Rollout (DGR): A novel scheduling mechanism that dynamically balances global semantic exploration and local detail refinement, preventing the model from focusing exclusively on fine-grained artifacts.
3. Consistent Policy Gradient Optimization (CPGO): A policy gradient formulation that enforces temporal self-consistency and alignment with a stable prior, mitigating the inconsistencies introduced by imitating stochastic trajectories.
4. Comprehensive Evaluation: The introduction of the VH-Evaluator and extensive experiments demonstrating significant reductions in hallucinations across multiple metrics.

4. Experimental Results

Extensive experiments were conducted on the FLUX.1-dev model using the HPDv2 dataset and HPS-v2 benchmark.

• Hallucination Reduction: ConsistentRFT achieved an average reduction of 49% in low-level perceptual hallucinations (e.g., grid patterns, noise) and 38% in high-level perceptual hallucinations compared to baseline methods.
• Out-of-Domain Generalization: Unlike baselines that often sacrifice out-of-domain performance for in-domain gains (e.g., MixGRPO improved HPS but dropped CLIP scores by 10.8%), ConsistentRFT improved out-of-domain metrics. Specifically, it showed a 5.1% improvement in comprehensive scores on FLUX1.dev, whereas the baseline decreased by 0.4%.
• Performance vs. SOTA: The method outperformed state-of-the-art RFT baselines (DDPO, DPO, GRPO variants) in human preference scores (ImageReward, PickScore) and aesthetic quality while maintaining semantic alignment.
• Efficiency: The method is compatible with standard policy gradients and offers speedups. For instance, the Intra-Group DGS reduced training time by ~18.6% compared to standard DanceGRPO while maintaining or improving performance.
• Few-Step Sampling: The method demonstrated superior stability in few-step sampling scenarios, where other methods typically degrade.

5. Significance and Claims

The paper claims that ConsistentRFT offers a general solution to the persistent issue of visual hallucinations in flow-based reinforcement fine-tuning. By decoupling the optimization of global semantics and local details (via DGR) and enforcing vector field consistency (via CPGO), the framework allows models to align with human preferences without sacrificing the structural integrity and realism of the generated images.

The authors position this work as a preliminary investigation into the specific causes of hallucinations in the post-training phase of flow models. They emphasize that while their method provides a robust mitigation strategy, the challenge of visual hallucinations also requires more powerful and scalable reward models, which they anticipate the community will develop in the future. The work does not claim to solve all hallucination issues but demonstrates a significant step forward in balancing preference alignment with visual fidelity.