Technical Summary: IV-CoT for Structure-Aware Text-to-Image Generation

Problem Statement

Unified multi-modal large language models (MLLMs) have demonstrated strong capabilities in generating realistic images from open-ended instructions. However, they frequently struggle with structure-aware prompt following, where prompts require precise adherence to object counts, spatial relations, attribute bindings, and coarse layouts.

The authors attribute this limitation to the entanglement of structural planning and appearance rendering within a single conditioning stream. In existing unified MLLM-based generators, scene structure, object identity, and visual details are compressed into a flat query sequence. This lack of separation makes it difficult for the diffusion generator to distinguish which factors should determine the scene's geometry versus its texture or color, leading to errors such as attribute swapping, omitted objects, or violated layouts.

While recent works have explored Chain-of-Thought (CoT) reasoning to address this, existing methods suffer from complementary limitations:

1. Explicit Textual CoT: Generates intermediate verbal reasoning or layouts but lacks the spatial bandwidth to represent continuous 2D geometry, boundaries, and occlusion effectively.
2. Explicit Interleaved CoT: Incorporates visual intermediates (masks, drafts) but requires multi-stage pipelines, explicit intermediate decoding, and suffers from error accumulation and high latency.

The paper posits a need for an intermediate planning mechanism that preserves visual-spatial information without requiring decoded reasoning states at inference time.

Methodology: Implicit Visual Chain-of-Thought (IV-CoT)

The authors propose IV-CoT, a structure-first latent reasoning framework built upon a query-conditioned MLLM-DiT (Diffusion Transformer) generator. Instead of externalizing reasoning as text or images, IV-CoT internalizes the planning process into the latent query representations.

1. Structural-to-Semantic Query Cascade

IV-CoT partitions the learnable visual query inputs into two ordered groups:

• Structural Queries ($Q_s$): Placed first in the sequence. They encode a "latent visual plan" including object form, count, layout, and coarse spatial relations.
• Semantic Queries ($Q_m$): Placed after structural queries. They attend to the prompt and the structural query states to render appearance, fine-grained details, and textures.

Due to the causal self-attention mechanism in the MLLM, this ordering enforces a one-way dependency: semantic queries can access the structural plan, but structural queries cannot access semantic details. This creates an implicit chain where the model first plans the structure and then renders the appearance conditioned on that plan.

2. Training-Only Sketch Supervision

To guide the structural queries without requiring sketches at inference, the authors employ a two-stage training scheme using sketch supervision:

• Stage 1 (Structural Training): The MLLM and DiT are frozen. Only the structural query inputs ($Q^0_s$) are optimized to denoise a sketch latent ($z_s$) derived from edge-detected training images. This forces $Q_s$ to encode contours, shapes, and layouts while suppressing appearance factors like color and texture.
• Stage 2 (Image Generation Training): The model is optimized for full image generation. The structural loss ($L_{struct}$) is retained as a regularizer alongside the image generation loss ($L_{img}$). This ensures $Q_s$ remains aligned with the latent visual plan while $Q_m$ learns to render identity, color, and material details.

3. Inference

At inference time, IV-CoT operates in a single forward pass. It takes only the text prompt and the learned query inputs, producing the combined conditioning sequence $[Q_s, Q_m]$. No sketches are extracted, no intermediate images are decoded, and no explicit reasoning traces are generated. The diffusion generator then performs standard denoising conditioned on this sequence.

Key Contributions

1. Formulation of IV-CoT: The paper introduces a framework where intermediate visual planning is internalized in latent query representations rather than externalized as explicit text or visual states.
2. Causal Query Cascade & Sketch Supervision: The authors instantiate this with a causal structural-to-semantic query cascade and a training-only sketch supervision strategy. This shapes structural queries into latent visual plans while preserving text-only, single-pass inference.
3. Performance and Efficiency: Using the OpenUni-L-1024 backbone, IV-CoT improves GenEval scores from 0.86 to 0.88 and T2I-CompBench scores from 0.5448 to 0.5743. Crucially, it achieves these results with 9–15× lower latency than explicit CoT methods (e.g., T2I-R1, GoT-R1) by avoiding multi-stage decoding.
4. Interpretability: Visualizations and cross-prompt recombination experiments demonstrate that structural queries encode recoverable and manipulable latent visual plans, while semantic queries handle appearance, confirming the complementary roles of the two query groups.

Experimental Results

• Benchmarks: Evaluated on GenEval and T2I-CompBench. IV-CoT outperforms both standard unified models (e.g., Janus-Pro, OpenUni) and explicit CoT-based methods (e.g., T2I-R1, TWIG-RL) in structure-sensitive dimensions such as position, color attribution, spatial relations, and shape.
• Efficiency: IV-CoT achieves a T2I-CompBench score of 0.5743 with an inference time of 1.693 seconds per sample. In contrast, explicit CoT methods require between 15.2 and 25.3 seconds per sample.
• Ablation Studies:
  • Removing the structural-to-semantic cascade or the sketch supervision leads to performance drops, confirming that both the ordered dependency and the structural guidance are necessary.
  • Simply increasing the number of queries without the specific cascade design yields limited gains, indicating the improvement stems from the architectural organization, not just capacity.
• Analysis: Cross-attention analysis reveals that structural queries receive higher attention around object contours and boundaries, while semantic queries are more active in object interiors and appearance regions.

Significance and Claims

The paper claims that IV-CoT successfully addresses the "entanglement" problem in unified MLLM generators by organizing conditioning queries into a structure-first, semantic-second hierarchy. By internalizing the reasoning process into latent states, the method achieves superior structure-aware prompt following without the computational overhead of explicit intermediate decoding.

The authors emphasize that IV-CoT offers a practical alternative to explicit CoT methods, providing a "single-pass" inference capability that is significantly faster while maintaining high compositional accuracy. The work suggests that latent reasoning, when properly structured (e.g., via sketch supervision and causal ordering), can effectively bridge the gap between high-level planning and low-level rendering in text-to-image generation.

Limitations: The authors note that IV-CoT is not specifically optimized for rendering readable text within images (which requires fine-grained character-level alignment) and has not yet been explored for image editing scenarios.