Technical Summary: FullFlow

Problem Statement

Modern text-to-image (T2I) diffusion and rectified-flow models possess rich visual priors and strong generative capabilities but operate strictly in a unidirectional manner: text in, image out. While unified vision–language models (VLMs) exist, they are typically constructed by conditioning a pretrained language model with a visual encoder or by performing expensive joint pretraining from scratch. These approaches often discard the strong image priors already encoded in T2I backbones or require substantial retraining that is computationally prohibitive. The central question addressed by this work is whether a pretrained T2I flow model can be upgraded into a bidirectional vision–language generator without full multimodal pretraining or sacrificing the original image prior.

Methodology

The authors propose FullFlow, a parameter-efficient recipe that upgrades a pretrained rectified-flow T2I model (specifically Stable Diffusion 3 and FLUX.1-dev) into a bidirectional generator. The core innovation lies in treating the two modalities asymmetrically but compatibly within a single shared backbone.

1. Native-Space Multimodal Extension

• Image Modality: Images remain in their native continuous rectified-flow latent space. The model preserves the pretrained continuous image process.
• Text Modality: Text is handled via a discrete insertion process built on Edit Flows. Unlike mask-based discrete diffusion (which introduces out-of-vocabulary tokens incompatible with frozen encoders), FullFlow uses a deletion-based forward process. This ensures that at any noise level, the corrupted text state is a valid token subsequence that can be decoded and re-tokenized for auxiliary encoders (e.g., CLIP).
• Architecture: A single shared backbone (MM-DiT) processes both modalities. The model is conditioned on two decoupled timesteps: $t$ for image time and $\tau$ for text time.
  • Frozen Components: The backbone weights and all auxiliary text encoders (T5, CLIP) remain frozen.
  • Trainable Components: Only LoRA adapters (applied to transformer linear layers), lightweight text heads (for insertion counts and token identities), and new timestep conditioning pathways are trained. This amounts to training only ~5% of the parameters for SD3 and ~2.5% for FLUX.1.

2. Dual-Timestep Training Objective

The training objective involves jointly predicting the image velocity and missing text insertions based on the pair of timesteps $(t, \tau)$.

• Corruption Schedule: The authors compare an "alternating-clean" schedule (training only on endpoint-conditioned tasks) with a "mixed-corruption" schedule where $(t, \tau) \sim \text{Unif}([0, 1]^2)$. They find that a hybrid approach—pretraining with mixed corruption to learn cross-modal correspondence, followed by refinement with alternating-clean—yields the best results.
• Stabilization:
  • Adaptive Gradient Balancing: To address the scale mismatch between the MSE-based image loss and the cross-entropy/count-based text loss, the authors dynamically adjust the text loss weight $\lambda_{txt}$ using an exponential moving average (EMA) of the gradient ratio.
  • Teacher Matching: To prevent degradation of the pretrained image prior during fine-tuning on new data distributions, the image velocity regression is replaced with a "same-noise teacher matching" objective. The teacher is the same backbone with LoRA layers disabled, requiring no additional memory.

3. Inference Modes

Decoupling $t$ and $\tau$ transforms inference into trajectory selection in a 2D generative space:

• Text $\to$ Image: Fix $\tau=1$ (clean text), evolve $t$ from 0 to 1.
• Image $\to$ Text: Fix $t=1$ (clean image), evolve $\tau$ from 0 to 1.
• Joint Sampling: Evolve both $t$ and $\tau$ simultaneously.
• Partial-Text Generation (VQA): Fix the image and question tokens, restricting the insertion process to the answer span. This enables Visual Question Answering without a separate decoder.

Key Contributions

1. C1: Converting a pretrained T2I rectified-flow model into a bidirectional generator by training only LoRA adapters and lightweight text heads, successfully preserving the pretrained image prior.
2. C2: Introducing a dual-timestep framework where images stay in continuous flow and text uses a discrete insertion process, enabling conditional, joint, and partial-text generation as trajectories in a unified $(t, \tau)$ space.
3. C3: Demonstrating superior performance over a matched Dual Diffusion-LoRA baseline on SD3 and FLUX.1-dev in both text-to-image retention and image-to-text quality, while supporting downstream VQA with minimal computational resources.

Experimental Results

The authors evaluate FullFlow on SD3 and FLUX.1-dev using a matched parameter count and LoRA rank against a re-implemented Dual Diffusion baseline.

• Efficiency: FullFlow trains on two RTX A5000 GPUs in under 24 hours, reducing peak VRAM from ~84 GB (Dual Diffusion) to ~38 GB and increasing throughput by ~8$\times$.
• Text-to-Image (SD3): At matched wall-clock time, FullFlow improves FID from 62.7 to 31.6 (and 29.3 at matched steps) and CMMD from 0.870 to 0.121, demonstrating strong retention of the image prior.
• Image-to-Text (SD3): FullFlow achieves a CIDEr score of 99.4 (vs. 2.0 for the baseline) at matched wall-clock time, and 13.4 at matched steps.
• Downstream Tasks: On VQA benchmarks (VQAv2, VizWiz, OKVQA), FullFlow (with only 130M trainable parameters) achieves competitive accuracy, notably outperforming the Dual Diffusion baseline on VizWiz (50.7 vs 29.9).
• Transferability: The recipe successfully transfers to FLUX.1-dev, achieving a CIDEr of 73.6 after 36 hours of training.

Significance and Claims

The paper claims that FullFlow demonstrates that strong bidirectional vision–language capabilities can be unlocked from pretrained T2I flow models without the need for full multimodal pretraining. The results suggest that pretrained T2I priors encode richer visual–semantic structure than commonly assumed, capable of supporting inverse tasks (image-to-text) and joint generation through lightweight adaptation.

The authors position this work as a "compute-feasible proof of concept" rather than a claim of peak end-to-end performance. They emphasize that their method makes bidirectional rectified-flow models accessible to the broader research community by requiring only commodity GPUs and a small fraction of the parameters typically needed for unified models. They acknowledge limitations, including the current focus on 256$\times$256 resolution and the lack of scaling to larger datasets or instruction tuning for complex dialogue, noting these as future directions.