💻 computer science

SeFi-Image: A Text-to-Image Foundation Model with Semantic-First Diffusion

SeFi-Image is a novel text-to-image foundation model utilizing a semantic-first diffusion paradigm that achieves state-of-the-art performance across multiple benchmarks with significantly reduced training compute (125K A800 GPU hours) compared to prior models, while offering scalable variants and distilled few-step versions for diverse deployment needs.

Original authors: SeFi-Team

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: SeFi-Team

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Building a House Before Painting the Walls

Imagine you are trying to teach a robot to paint a masterpiece based on a description you give it.

The Old Way (Traditional AI):
Usually, these robots try to learn everything at once. They have to figure out where the tree goes, what color the sky is, how the leaves look, and how the bark feels, all in the same split second. It's like trying to build a house, paint the walls, and install the plumbing all at the exact same time. It takes a massive amount of time, energy, and money (computing power) to get it right.

The SeFi-Image Way (Semantic-First Diffusion):
The SeFi-Image team realized that humans don't paint this way. We usually sketch the outline first, then fill in the details.

Step 1: The Blueprint (Semantics): First, the AI figures out the "big picture." Where is the sun? Is there a cat? Is the cat on the roof? It creates a clean, structural skeleton of the image.
Step 2: The Details (Texture): Once the skeleton is solid, the AI fills in the fur on the cat, the clouds in the sky, and the texture of the roof.

This paper introduces a new method called Semantic-First Diffusion. It forces the AI to solve the "blueprint" part before it starts worrying about the "paint." Because the blueprint is already clear, the part that adds the details has an easier job.

Why This is a Big Deal

1. It's a "Budget-Friendly" Superstar
Usually, to get an AI to draw really well, you need a supercomputer running for months.

The Comparison: The paper mentions a famous model called Z-Image that cost a fortune to train (using 314,000 GPU hours).
The SeFi-Image Result: Their biggest model (5 billion parameters) achieved similar or even better results using only 125,000 GPU hours. That's like doing the same amount of work with only 10–20% of the electricity bill. They proved you don't need to burn as much cash to get a high-quality result if you organize the learning process better.

2. The "Three Sizes" Approach
The team didn't just build one giant robot; they built three:

The Tiny One (1B): Small, fast, and surprisingly smart. It can follow instructions well even though it's small.
The Medium One (2B): A balanced option.
The Big One (5B): The heavy lifter that handles the most complex requests.
This is great because different people have different computers. If you have a powerful server, you use the Big One. If you have a laptop, you might use the Tiny One.

3. It's Good at Reading and Writing Text
One of the hardest things for AI is drawing text inside an image (like a sign on a store or a book cover).

The Problem: Most AIs get the letters jumbled or misspelled.
The SeFi-Image Fix: They fed the AI a special diet of "synthetic data." Imagine a machine that generates millions of images of text on plain backgrounds or complex layouts, teaching the AI exactly how letters look and where they should sit. Because of this, SeFi-Image is very good at rendering text accurately, even in long, complicated sentences.

How They Trained It (The "Curriculum")

The paper describes a smart training schedule, like a student going from elementary school to college:

Elementary (Pre-training): They showed the AI millions of images with simple descriptions to learn the basics of shapes and objects.
High School (Continual Training): They switched to higher-quality images to teach it how to follow more specific instructions.
College (Fine-Tuning): They used a very strict filter to only show the AI the best possible images, teaching it to be an artist rather than just a sketcher.

They also created a "Turbo" version of the model. Think of this as a "fast-forward" button. Usually, AI takes 50 steps to draw an image. They used a technique called distillation to teach the model how to do it in just 4 steps, making it much faster for real-time use without losing too much quality.

What They Found (The Results)

It works: The model passed many tests (benchmarks) where it had to follow complex instructions (like "put a red cat on a blue chair next to a tree").
It scales: Even though the "Big One" (5B) is the best, the "Tiny One" (1B) performed almost as well as much larger models from other companies. This proves that the "Blueprint First" method is very efficient.
It's bilingual: It works well in both English and Chinese.

The Limitations (What They Didn't Do)

The authors are honest about what their model can't do yet:

Size Cap: Their biggest model is 5 billion parameters. They wish they could make it bigger, but they ran out of computer power (specifically, they were limited to NVIDIA A800 GPUs).
Style Variety: Because they focused heavily on natural images and text, the model isn't as good at very specific artistic styles (like complex graphic design or infographics) as it is at general pictures.
Video & Editing: They built this for making new images from text. They haven't tested it yet on editing existing photos or making videos.

Summary

SeFi-Image is a new way to teach AI to draw. Instead of trying to learn everything at once, it teaches the AI to draw the "skeleton" first and the "skin" second. This simple change allows them to build a model that is cheaper to train, faster to run, and just as good (or better) than the most expensive models currently available. They have released their code and models so anyone can try them out.

Technical Summary: SEFI-Image

Problem Statement

Training foundational text-to-image (T2I) models has become increasingly resource-intensive, with state-of-the-art models requiring hundreds of thousands of GPU hours. While previous methods attempted to leverage semantic guidance to accelerate training, these approaches were largely validated only on simple, low-resolution datasets (e.g., ImageNet 256×256) using small-scale models (<1B parameters). A critical open question remains: whether semantic guidance mechanisms remain effective at larger model scales, higher resolutions, and within the complex, practical T2I setting. Furthermore, existing methods often struggle with the fundamental trade-off in latent diffusion: high reconstruction fidelity requires complex latent distributions that are difficult to model, while simplified latent spaces often sacrifice image quality.

Methodology: Semantic-First Diffusion (SFD)

The paper proposes SEFI-Image, a T2I foundation model built upon Semantic-First Diffusion (SFD), a novel latent diffusion paradigm. The core innovation is the decoupling of the denoising process into asynchronous semantic and texture streams.

1. Core Mechanism

Unlike conventional latent diffusion models that denoise all information under a single timestep schedule, SFD separates semantic factors (global structure, object layout) from texture factors (high-frequency details).

Asynchronous Timesteps: The model assigns distinct timesteps $t_s$ (semantic) and $t_z$ (texture). The semantic latent is denoised slightly ahead of the texture latent ( $t_s \ge t_z$ ), ensuring that texture generation is always conditioned on a "cleaner" structural anchor.
Three-Phase Inference:
1. Semantic Initialization: Only semantic latents are denoised to establish global structure.
2. Asynchronous Generation: Both streams denoise, but semantics lead to guide texture refinement.
3. Texture Completion: Semantics are fully denoised; texture latents refine fine-grained details.
Dual-Timestep Conditioning: The Diffusion Transformer (DiT) backbone accepts separate timestep embeddings for both streams, allowing the model to be aware of the asynchronous noise levels at each step.

2. Architecture Components

Composite Latent Construction:
- Semantic Latent ( $s_1$ ): Extracted via a frozen visual foundation model (DINOv2-Large) and compressed by a Semantic VAE (SemVAE). This captures object identity and scene structure.
- Texture Latent ( $z_1$ ): Encoded directly from the image by a Texture VAE (a fine-tuned FLUX.2 VAE). This captures low-level reconstruction details.
Text Encoder: Utilizes the LLM backbone of Qwen3-VL (2B or 4B variants depending on the generation model size) to handle complex, long, and bilingual (Chinese/English) prompts.
DiT Backbone: A FLUX.2-style architecture incorporating double-stream MMDiT blocks and single-stream blocks, adapted to accept the composite latent and dual timesteps.

3. Training Pipeline

The training process involves three stages:

Pre-training: Uses 450M internal image-text pairs and 28M synthetic text-rendered data. The curriculum progresses from 256px to 1024px. The synthetic data focuses on strict character-level alignment for text rendering.
Continual Training: Trained on a curated 9M high-quality mixture (natural scenery, UI, anime, etc.) at 1024px to boost instruction following.
Supervised Fine-Tuning (SFT): Uses ~650K high-quality images with strict filtering based on aesthetics, technical quality, and captionability. This stage emphasizes hard prompts and accurate text rendering.
Distillation & RL:
- Few-Step Distillation: DMD2 is used to distill the SFT model into 4-step "Turbo" variants while preserving the semantic-first offset.
- RL Post-training: DiffusionNFT is applied to sharpen prompt following, visual quality, and artifact suppression.

Key Contributions

Scalable Semantic-First Diffusion: The paper demonstrates that SFD is not limited to small models or toy datasets. It successfully scales to 1B, 2B, and 5B parameter models, maintaining effectiveness in high-resolution T2I generation.
Resource Efficiency: The largest 5B model was trained with only 125K A800 GPU hours, representing roughly 10–20% of the compute used by Z-Image, yet achieving comparable or superior performance.
Reconstruction-Generation Trade-off Resolution: By separating semantics and textures, SFD allows the Texture VAE to be aggressively fine-tuned for reconstruction fidelity (improving PSNR on Kodak from 33.18 to 36.40) without sacrificing the convergence speed typically associated with complex latent spaces.
Comprehensive Model Family: The release of three model scales (1B, 2B, 5B) and their corresponding Turbo variants provides flexible deployment options for varying hardware budgets.

Performance Results

SEFI-Image was evaluated on multiple benchmarks against strong baselines (Qwen-Image, Z-Image, FLUX.2-Klein-9B, JoyAI-Image):

GenEval & DPG: The 5B model achieves an overall GenEval score of 0.88, matching Qwen-Image and surpassing FLUX.2-Klein-9B (0.85) and Z-Image (0.84). Notably, the 1B variant also achieves 0.87, demonstrating strong compositional reasoning even at small scales.
Long-Text Rendering (LongTextBench): The 5B model achieves the highest average score (0.978), outperforming Qwen-Image and JoyAI-Image.
Visual Text Generation (CVTG-2K): The 5B model achieves 0.8947 Word Accuracy and 0.9434 NED, improving over the strongest baselines.
Bilingual Instructions (OneIG): The 5B model achieves the best overall scores on both English (0.5606) and Chinese (0.5379) benchmarks.
Scaling Behavior: Experiments show that a 2B SFD model can outperform a 4B non-SFD model, indicating improved parameter efficiency.

Significance and Claims

The authors claim that SEFI-Image offers a principled and resource-efficient paradigm for building T2I foundation models. The work challenges the notion that semantic guidance is only effective for small-scale or class-conditional tasks. By decoupling semantic layout from texture synthesis, the model achieves a superior reconstruction-generation trade-off, enabling rapid convergence and high-quality generation with significantly reduced compute costs.

The paper positions SEFI-Image as a practical, readily deployable solution that provides useful insights into semantic-guided diffusion modeling. It explicitly notes that while the current models are strong, they are limited by the 5B parameter cap (due to hardware constraints) and the scale of the training data, leaving room for future scaling in model size, data diversity, and application to video generation and image editing.