💻 computer science

Trajectory Forcing: Structure-First Generation with Controllable Semantic Trajectories

The paper proposes Trajectory Forcing (TF), a structure-first generative framework that transforms image synthesis into an explicit, editable sequence of semantically structured stages from global layout to fine details, enabling localized control and inspection of intermediate states while maintaining competitive sample quality.

Original authors: Merve Kocabas, Gege Gao, Bernhard Schölkopf, Andreas Geiger

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Merve Kocabas, Gege Gao, Bernhard Schölkopf, Andreas Geiger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician pull a rabbit out of a hat. In most modern AI image generators, the magic happens in a "black box." You give the AI a prompt (like "a cat"), and it instantly produces the final picture. You have no idea how it got there, and you can't stop the process halfway to say, "Wait, make the ears bigger before you finish the fur." The intermediate steps are just invisible math calculations that get thrown away once the final image appears.

Trajectory Forcing (TF) is a new method that throws open the curtains on that black box. Instead of a magic trick, it turns image generation into a step-by-step painting process, much like how a human artist works.

Here is how it works, broken down into simple concepts:

1. The "Speed Painting" Analogy

Think of how an artist paints a scene. They don't start by painting every single hair on a dog's head.

First, they sketch the big shapes and where the main objects are (the "Global Layout").
Next, they define the major parts (the dog's head, body, and legs).
Then, they refine the sub-parts (the snout, the ears).
Finally, they add the tiny details (the texture of the fur, the shine in the eyes).

Current AI skips straight to step 4. Trajectory Forcing forces the AI to stop at every single step. It generates a "rough draft" of the layout, then a "sketch" of the parts, then a "detailed drawing," and finally the "finished photo."

2. The "Editable Blueprint"

The coolest part of this paper is that every single step is a real, viewable image.

In normal AI, the intermediate steps are just numbers.
In Trajectory Forcing, the intermediate steps are decoded into actual pictures.

This means you can look at the "rough draft" stage and say, "I don't like where the dog is standing." You can move the dog in that rough draft, and the AI will automatically re-paint the rest of the picture (the parts, the details) based on your new position. You are editing the path the AI takes, not just the final destination.

3. How They Built the "Teacher"

To teach the AI to do this, the researchers didn't just guess what "parts" and "sub-parts" look like. They used a smart "teacher" system based on a pre-trained visual brain (called DINOv2).

Imagine you have a pile of Lego bricks.

Old methods tried to sort them by size or color, which often led to messy piles.
Trajectory Forcing uses a smart teacher that looks at the bricks and naturally groups them by meaning: "These bricks make a wheel," "These make a door," "These make the whole car."

The AI learns to build the image by following this logical hierarchy: First the "car" (object), then the "wheels and doors" (parts), then the "tires and handles" (sub-parts).

4. The "One-Step" Speed Trick

Usually, making an image in stages is slow because the computer has to do a lot of work at every stage.

The Problem: If you have 4 stages and each takes 10 steps, that's 40 steps total.
The Solution: The researchers used a clever math trick (called One-Step Flow Matching) that lets the AI jump from one stage to the next in a single leap.
The Result: It takes the same amount of time to generate a 4-stage image as it does to generate a normal, single-stage image. You get the control of a slow process with the speed of a fast one.

5. Why This Matters (According to the Paper)

The paper claims that this method gives us control without sacrificing quality.

Control: You can fix mistakes early. If the "object" stage is wrong, you fix it there, and the AI doesn't waste time generating details for a wrong object.
Consistency: The paper shows that if you change a small part (like a sub-part), the rest of the image stays stable. If you change a big part (like the whole object), the whole image changes naturally.
Speed: It achieves high-quality images on the ImageNet dataset (a standard test for AI) in just 4 "leaps" (stages), which is very fast compared to older multi-stage methods.

Summary

Trajectory Forcing changes image generation from a "black box" magic trick into a transparent, editable assembly line. It forces the AI to build images from the "big picture" down to the "tiny details," allowing humans to inspect and fix the work at every single stage, all while keeping the process fast.

Technical Summary: Trajectory Forcing (TF)

1. Problem Statement

Current diffusion and flow-based generative models excel at producing high-quality images but operate as "endpoint-centric" systems. Users specify conditions (e.g., class labels) and receive a final output, while the intermediate generative dynamics remain hidden, unstructured, and computationally opaque. Although recent methods have utilized generation order to improve sample quality, they treat intermediate states merely as internal computational by-products (e.g., frequency components or token completions) rather than as semantic objects available for human inspection or intervention.

This contrasts with human visual creation, which follows a coarse-to-fine workflow: establishing global composition before refining local details, with each intermediate stage being inspectable and revisable. The paper posits that generative trajectories themselves can be made interpretable, semantically structured, and editable, shifting the focus from the final image to the generative path.

2. Methodology: Trajectory Forcing (TF)

Trajectory Forcing is a trajectory-centric framework that organizes image synthesis as a sequence of semantically structured stages, progressing from global layout to object-, part-, and detail-level representations.

2.1 Representation Space and Hierarchy Construction

Instead of operating in raw pixel space or standard latent spaces (like VQ-VAE) where geometry may not align with semantic structure, TF operates in the feature space of DINOv2, a pretrained visual encoder.

Teacher Hierarchy: The framework derives a coarse-to-fine hierarchy by clustering DINOv2 latent tokens via unsupervised methods.
- Object/Background: K-means ( $K=2$ ) separates the image into object and background based on a center prior.
- Parts and Subparts: Agglomerative clustering using a combined semantic-spatial distance metric ( $\mathcal{D} = \cos \langle \mathbf{z}_i, \mathbf{z}_j \rangle + \alpha \|\mathbf{p}_i - \mathbf{p}_j\|_2$ ) decomposes the object into parts and subparts.
Level Canvases: These clusters define "level canvases" ( $z^{(l)}$ ). Coarser levels ( $l=0$ ) represent piecewise-constant mean features of regions, while finer levels ( $l=L-1$ ) represent the full-resolution latent features.

2.2 Hierarchy-Conditioned One-Step Flow Matching

To avoid the high inference cost of progressive multi-step diffusion ( $L \times T$ ), TF employs a hierarchy-conditioned one-step flow-matching model.

Architecture: A single shared network (DiT) is trained across all hierarchy levels.
Conditioning: At each training step, a level $l$ $l$ is sampled. The network receives:
1. The noisy current-level canvas $z^{(l)}(t)$ .
2. The previous-level canvas $z^{(l-1)}$ as a spatial condition (fused via channel concatenation and projection).
3. Standard global conditions (class label, time interval, guidance scale).
Training Objective: The model minimizes a combined loss:
- Flow Loss: Uses $x$ -prediction (predicting denoised latents) with a $v$ -loss objective based on MeanFlow (iMF) to enable one-step generation.
- Structural Loss: A cosine distance penalty ensuring that tokens within a predicted region align with the ground-truth mean feature of that region, enforcing structural consistency.

2.3 Hierarchical Sampling and Editing

Inference: Generation proceeds sequentially from $l=0$ to $L-1$ . Each level requires only a single network function evaluation (NFE), resulting in a total cost of $L$ -NFE (e.g., 4 NFEs for $L=4$ ). Every intermediate output is decodable into a visual image via a shared Representation Autoencoder (RAE) decoder.
Interactive Editing: Users can inspect decoded images at any level $l^*$ , modify the canvas (via feature swapping or shape/region reassignment), and re-generate subsequent levels. Due to the Markovian nature of the process, edits at level $l^*$ propagate to finer levels but leave coarser levels unchanged, allowing for controlled scope of modification.

3. Key Contributions

Trajectory-Centric Framework: Proposes TF, which models synthesis as a structured, interpretable coarse-to-fine trajectory, enabling inspection and editing at every semantic level.
Hierarchy-Conditioned One-Step Flow: Extends one-step generation to multi-level trajectories via level-wise conditioning and structural consistency supervision, achieving competitive quality with $L$ -NFE.
Trajectory-Aware Metrics: Introduces metrics beyond FID to measure Local Invariance (how edits affect unedited regions) and Structural Consistency (coherence between parent and child regions).
Empirical Validation: Demonstrates on ImageNet that TF achieves competitive sample quality (FID) while exposing coherent intermediate states and supporting localized edits.

4. Experimental Results

Experiments were conducted on ImageNet at $256 \times 256$ resolution using DINOv2 features and a ViT-XL decoder.

Sample Quality: TF achieves competitive FID scores. For example, the TF-L/16+FD-loss variant achieves an FID of 2.02 and IS of 259.3 with only 80 training epochs and 4 NFEs, comparable to multi-step baselines requiring hundreds of epochs and NFEs.
Efficiency: Unlike hierarchical baselines like NVG (which requires ~184 NFEs), TF requires only $L=4$ NFEs.
Trajectory Metrics:
- Local Invariance Score (LIS): Shows that edits at coarser levels (object/background) propagate more strongly to the final image, while finer edits (subparts) remain localized.
- Parent Misrouting Rate (PMR): Both spatial and feature PMR remain low, indicating that child regions are spatially contained within parent regions and maintain semantic consistency.

5. Significance and Claims

The paper claims that by shifting the focus from the final image to the generative path, TF opens a route toward controllable, trajectory-aware image synthesis.

Interpretability: Unlike standard diffusion where intermediate states are discarded, TF makes every intermediate state visually decodable and semantically meaningful.
Editability: It enables users to intervene at specific semantic granularities (global layout vs. local details) with predictable effects on the final output.
Efficiency: It demonstrates that high-quality, structured generation does not require expensive multi-step iterative denoising at every level, leveraging one-step flow matching instead.

The authors acknowledge limitations, noting that the hierarchy relies on unsupervised clustering with a center prior (which may struggle with non-centered objects) and that current results are based on early training epochs (40–80), with further scaling analysis deferred. However, the core contribution remains the establishment of a framework where the generative trajectory itself serves as a user-facing interface for control.