Technical Summary: Images in Sentences (Inset)

Problem Statement

Recent advancements in multimodal language models have enabled image generation from expressive, multi-image instructions. However, existing methods struggle to maintain performance under complex, interleaved instructions involving multiple reference images. This limitation arises from two primary factors:

1. Indirect Referencing Mechanisms: Current paradigms typically treat images as external references retrieved via explicit indices (e.g., "the dog in Image 1"). This forces models to bridge difficult long-range dependencies to align abstract indices with distant visual features, often leading to attribute binding failures or the neglect of specific image inputs as sequence length increases.
2. Scarcity of Complex Data: Existing interleaved datasets are often limited in scale and complexity, lacking the rich, long-horizon sequences necessary to teach models how to handle intricate compositional reasoning with dense visual contexts.

Methodology: Inset

To address these challenges, the authors propose Images in Sentences (Inset), a unified generation model that fundamentally rethinks how visual information is integrated into textual instructions.

1. Unified Interleaved Modeling

Instead of treating images as external references requiring retrieval, Inset embeds images directly as native vocabulary within the text instruction.

• Native Formulation: Visual features are positioned directly at their corresponding semantic slots within the sentence (e.g., "A [Image1] robot holds a [Image2] flower vase"). This leverages the contextual locality of transformers to directly bind textual descriptions with visual targets, eliminating the need for the model to resolve long-range dependencies between text and distant image tokens.
• Architecture: Built upon the BAGEL model, Inset utilizes a Mixture-of-Transformer architecture. Crucially, it discards pixel-level VAE latent features from the input, relying solely on semantic ViT embeddings. The authors argue that including VAE latents biases models toward "image-pasting" (rigid copying) rather than semantic integration.
• Inference Strategy: To prevent visual modalities from dominating the generation process, a two-stage guidance strategy is employed. This involves calibrating the text-visual balance (boosting text influence) and applying classifier-free guidance.

2. Scalable Data Engine

Recognizing the lack of high-quality interleaved data, the authors developed a scalable data engine to synthesize 15 million high-quality samples from standard image and video datasets.

• From Images: The pipeline uses Vision-Language Models (VLMs) to generate global captions and extract fine-grained object instances (masks and captions) via the Segment Anything Model (SAM) and Describe Anything Model (DAM). An LLM then synthesizes these elements into natural text sequences with visual embeddings explicitly placed at their semantic positions.
• From Videos: To teach dynamic state manipulation, the engine processes video corpora. It identifies object correspondences across frame pairs with significant temporal variance using VLMs. It filters for pairs exhibiting meaningful semantic changes (action, pose, morphology) and constructs instructions where the visual token comes from a source frame, and the generation target is a transformed state in a target frame.

3. InterleaveBench

The authors introduce InterleaveBench, a rigorous benchmark designed to evaluate complex multi-image scenarios. It features multi-image compositions with intricate instructions requiring logical spatial reasoning and adaptive attribute modification. Evaluation utilizes a dual-perspective LLM-as-Judge framework to measure:

• Image Consistency: Identity preservation (1–5 scale), penalizing drift while tolerating instruction-driven variations.
• Text Consistency: Semantic alignment via VQA-based binary questions targeting specific attributes and relationships.

Key Results

Experiments on InterleaveBench demonstrate that Inset significantly outperforms state-of-the-art methods:

• Performance Gaps: Inset achieves superior performance in both image and text consistency compared to open-source models (e.g., DreamOmni2, FluxKontext, Qwen Image).
• Scalability: The performance advantage widens as input complexity increases. In the "Five Objects" setting, Inset leads open-source baselines by 0.29 in image consistency and 0.24 in text consistency.
• Competitive Standing: Despite having fewer parameters (7B+7B) than many competitors, Inset achieves performance comparable to powerful closed-source models (e.g., GPT-4o, Seedream 4.0) in image consistency.
• Ablation Studies: Removing VAE features and utilizing video-based data were identified as critical components for preventing "image-pasting" and enabling dynamic state manipulation, respectively.

Significance and Claims

The paper claims that treating visual content as dense, expressive language tokens offers a promising direction for building more intuitive and unified multimodal systems. By seamlessly embedding images as native vocabulary, Inset overcomes the structural separation of images and text in current paradigms.

Beyond standard generation, the authors highlight that this interleaved format naturally extends to multimodal image editing. The model can integrate visual content as part of the instruction, facilitating highly expressive and creative visual manipulations where both text and specific visual references guide the editing process. The work suggests that precise object binding is best achieved by leveraging the transformer's contextual locality rather than relying on indirect, index-based referencing.