DivCon: Divide and Conquer for Complex Numerical and Spatial Reasoning in Text-to-Image Generation

DivCon introduces a divide-and-conquer framework that decouples text-to-image generation into simplified numerical/spatial reasoning and progressive object synthesis steps, enabling lightweight models to achieve superior layout accuracy and perceptual quality for complex multi-object prompts without relying on closed-source large language models.

The Gist

Imagine you are an artist trying to paint a picture based on a very specific, complicated description someone gives you.

The Problem:
If you ask a standard AI artist (like the popular "Stable Diffusion") to draw: "Five red apples, two blue birds, and a cat sitting between a chair and a tree," it often gets confused. It might draw three apples instead of five, put the cat on top of the tree, or forget the birds entirely. It's like asking a talented painter to juggle too many specific instructions at once; they get overwhelmed and drop the ball.

The Old Solution:
Previous attempts to fix this involved hiring a "super-intelligent project manager" (a massive, expensive AI like GPT-4) to first draw a rough sketch (a layout) of where everything goes, and then hand that sketch to the painter.

• The Downside: This "manager" is incredibly expensive to run, hard to access, and requires a lot of computer power. Also, the painter still tries to paint the whole scene in one go, which leads to mistakes with the tricky parts.

The New Solution: DivCon (Divide and Conquer)
The authors of this paper, Yuhao Jia and Wenhan Tan, came up with a smarter way called DivCon. Instead of trying to do everything at once, they break the job into smaller, manageable chunks. Think of it like building a house: you don't try to build the roof, walls, and plumbing all in one second. You do it step-by-step.

Here is how DivCon works, using simple analogies:

Phase 1: The "Smart Sketch" (Layout Prediction)

Instead of asking a super-expensive AI to draw the whole plan, DivCon uses a small, lightweight AI (like a smart intern) and gives it a two-step checklist:

1. Step A: The Math & Logic Check. First, the intern just reads the sentence and counts the items. "Okay, I hear 'five apples' and 'two birds'." It doesn't try to draw them yet; it just does the math and notes the positions.
2. Step B: The Drawing Plan. Once the intern knows the exact numbers and positions, it draws the rough boxes (like a blueprint) for where those items should go.

Why this is cool: By splitting "counting" from "drawing," even a small, cheap computer program can do a better job than a giant, expensive one. It's like giving a calculator to a student to solve the math, and then letting them draw the picture. The result is a perfect blueprint without the high cost.

Phase 2: The "Progressive Painting" (Image Generation)

Now that we have the blueprint, we need to paint the picture. Standard AI tries to paint the whole image at once. DivCon does it differently:

1. Round 1: The "Easy Stuff" First. The AI paints the whole scene, but it pays extra attention to the "easy" things (like a big, simple chair).
2. The Check-Up. The AI looks at its own work. "Hmm, the chair looks great. But the 'five apples' look blurry, and the 'cat' looks weird."
3. Round 2: Fixing the Hard Parts. The AI freezes the good parts (the chair) and re-paints only the messy parts (the apples and the cat). It focuses all its energy on fixing the difficult details without messing up the good stuff.

The Analogy: Imagine you are editing a photo. Instead of trying to fix the whole picture at once, you use a "clone stamp" tool to fix just the blemishes on the face while leaving the background untouched. DivCon does this automatically.

The Results

• Better Accuracy: It gets the numbers right (5 apples, not 3) and the positions right (cat between chair and tree, not on top).
• Cheaper: It uses small, open-source AI models instead of needing a billion-dollar supercomputer.
• Faster: Because it fixes problems in two focused steps rather than one chaotic step, it actually runs efficiently.

In a Nutshell:
DivCon is like hiring a project manager who breaks a big, scary task into small, easy steps, and then a painter who fixes their mistakes one by one. The result is a masterpiece that follows the instructions perfectly, without needing a massive budget to create it.

Technical Summary

Here is a detailed technical summary of the paper "DivCon: Divide and Conquer for Complex Numerical and Spatial Reasoning in Text-to-Image Generation."

1. Problem Statement

While large-scale Text-to-Image (T2I) models (e.g., Stable Diffusion) excel at generating general images, they struggle significantly with numerical reasoning (generating exact object counts) and spatial reasoning (accurate positioning and relationships between objects).

Existing solutions that combine Large Language Models (LLMs) with diffusion models to improve these capabilities face two major limitations:

1. Dependency on Massive LLMs: Current methods rely on closed-source, large-scale LLMs (like GPT-4) for layout prediction. This creates high computational costs, limits accessibility, and prevents the use of lightweight, open-source models.
2. Simultaneous Generation Issues: Layout-to-image diffusion models typically synthesize all objects in a single forward pass. This fails to account for the varying difficulty of generating different object types (e.g., complex textures vs. simple shapes), leading to inconsistent counts, missing objects, or spatial distortions in complex prompts.

2. Methodology: DivCon

The authors propose DivCon, a training-free framework that applies a divide-and-conquer strategy to both the layout prediction and image generation stages.

Stage 1: Layout Prediction (Decoupled Reasoning & Planning)

Instead of asking an LLM to directly output bounding boxes (which is unstable for smaller models), DivCon splits this task into two explicit steps:

1. Numerical & Spatial Reasoning: A lightweight, open-source LLM (e.g., TinyLlama, Phi-2, Mistral-7B) is prompted to extract structured information from the text prompt.
   • It outputs numerical constraints (e.g., "5 spoons") or spatial constraints (e.g., "cat in the middle of chair and dog").
2. Bounding Box Visual Planning: The same LLM is prompted again to generate the actual bounding box coordinates, but this time it is guided by the reasoning output using constrained decoding.
   • Mechanism: The model uses prefix tokens (e.g., apple: repeated 5 times for count, or apple(top-left): for position) to enforce strict adherence to the extracted constraints. This allows lightweight models to achieve layout accuracy comparable to massive models like GPT-4.

Stage 2: Layout-to-Image Generation (Progressive Refinement)

The generation process is divided into two rounds to handle objects of varying difficulty:

1. First-Round Generation: The diffusion model generates an image based on the text and the predicted layout.
2. Consistency Evaluation: The system calculates the CLIP similarity between the generated object crops and their text descriptions.
   • High Consistency (Easy): Objects that match the prompt well are considered "good enough."
   • Low Consistency (Hard): Objects that are distorted, missing, or incorrect are flagged.
3. Second-Round Refinement:
   • The latent representations of the "easy" objects from the first round are retrieved and preserved.
   • A binary mask is applied to the noise input of the second forward pass, effectively "freezing" the high-fidelity regions.
   • The diffusion model is forced to focus its denoising capacity solely on the "hard" objects (the low-consistency regions), refining them without disrupting the already correct parts of the image.

3. Key Contributions

• DivCon Framework: A novel approach that decouples T2I generation into manageable subtasks, significantly improving numerical and spatial reasoning without requiring model retraining.
• Lightweight LLM Optimization: Demonstrates that by decoupling reasoning from visual planning, small, quantized, open-source LLMs (e.g., 1.1B–8B parameters) can match or exceed the layout prediction performance of GPT-4 (>70B parameters).
• Progressive Synthesis: Introduces a two-step image generation process that identifies and re-synthesizes difficult objects while preserving easy ones, leading to higher fidelity in complex scenes.
• Computational Efficiency: Achieves state-of-the-art results with significantly lower computational costs compared to methods relying on GPT-4 or multi-pass diffusion without masking.

4. Experimental Results

The method was evaluated on the HRS and NSR-1K benchmarks, which test numerical and spatial reasoning.

• Layout Prediction:
  • Mistral-7B + DivCon achieved a 24.56% gain in numerical accuracy on HRS compared to its baseline.
  • Qwen3-8B + DivCon surpassed GPT-4 in HRS numerical reasoning metrics (F1 score +0.94%, Accuracy +0.72%) and matched GPT-4 on NSR-1K.
• Image Generation:
  • DivCon outperformed previous state-of-the-art methods (e.g., Attention-Refocusing, GLIGEN) by significant margins.
  • On HRS, it improved numerical F1 by 8%+ and spatial accuracy by 40%+.
  • On NSR-1K, spatial accuracy improved by 50%+.
• Efficiency:
  • DivCon (2 iterations) takes 72.01s, which is significantly faster than LLM-Grounded (134.98s) and Mixture-of-Diffusion (126.41s), while offering better quality.
• User Study:
  • In a blind ranking study, DivCon achieved the best average rank (1.74), indicating superior perceptual quality and prompt alignment compared to Stable Diffusion and other baselines.

5. Significance

DivCon represents a paradigm shift in complex T2I generation by proving that task decomposition is more effective than simply scaling up model size.

• Accessibility: It democratizes high-quality, reasoning-capable T2I generation by enabling the use of lightweight, open-source models, removing the barrier of expensive proprietary APIs.
• Scalability: The divide-and-conquer approach is highly scalable, allowing for the generation of complex scenes with many objects and intricate spatial relationships that were previously impossible for standard diffusion models.
• Quality: By explicitly handling "hard" objects in a second pass, it solves the common issue of object omission or distortion in multi-object prompts, setting a new standard for prompt fidelity.