How Long Can Unified Multimodal Models Generate Images Reliably? Taming Long-Horizon Interleaved Image Generation via Context Curation

This paper introduces UniLongGen, a training-free inference strategy that improves long-horizon interleaved image generation by dynamically curating context to discard accumulated visual noise, thereby overcoming the reliability collapse caused by dense visual token interference in unified multimodal models.

The Gist

Imagine you are a master storyteller who can also draw. You want to tell a long, epic story where you write a paragraph, draw a picture, write another paragraph, draw another picture, and keep going for 40 or 50 turns.

This is what Unified Multimodal Models (AI that handles both text and images) are trying to do. But there's a big problem: The longer the story gets, the worse the drawings become.

By the time the AI reaches the 20th picture, the characters start looking like melted wax, the style changes randomly, and the story makes no sense. It's like a painter who starts with a masterpiece but, after 20 paintings, can't remember what the main character looks like anymore.

This paper, UniLongGen, figures out why this happens and gives the AI a simple trick to fix it.

The Problem: The "Cluttered Desk" Effect

Usually, when AI fails at long tasks, we think it's because it's "forgetting" things (like running out of memory). The authors say: No, that's not it.

Think of the AI's memory like a desk where it keeps all the reference photos and notes for the story.

• Text is like notes: If you have 1,000 pages of notes, the AI might get a little confused, but it can still find the right one.
• Images are like giant posters: Every time the AI draws a picture, it adds a massive, high-resolution poster to the desk.

The Real Issue:
When you have 20 posters on your desk, they start fighting for your attention. The AI looks at the current task ("Draw a cat") and gets distracted by a random poster from 15 turns ago that happens to have a blurry shape that looks slightly like a cat.

Because the AI is so good at finding patterns, it latches onto these "accidental matches." It's like trying to listen to a friend in a crowded room, but suddenly 20 people start shouting random words that sound kind of like what your friend said. The AI gets hijacked by these "noise" signals, and the drawing goes off the rails.

The authors call this "Active Pollution." The old images aren't just fading away; they are actively corrupting the new ones.

The Solution: The "Smart Curator"

The paper proposes a method called UniLongGen. Instead of trying to remember everything (which causes the clutter), the AI learns to curate its memory.

Imagine the AI has a Smart Curator (a helpful assistant) who stands by the desk. Here is how the Curator works:

1. The "One-Shot" Check: Before drawing the next picture, the Curator quickly glances at the entire history of the story (all the text and all the old pictures).
2. The "Relevance" Test: The Curator asks the AI: "Which of these old pictures actually helps me draw the NEXT one?"
   • It ignores pictures that are just "there."
   • It ignores pictures that are too old to matter.
   • It picks only the top 4 or 5 most important images that define the character's face or the story's style.
3. The "Trash Bin" (Crucial Step): This is the most important part. The Curator doesn't just hide the other pictures; it throws them away (removes them from the AI's immediate memory).
   • Why? If you just hide them, the AI might still peek at them and get distracted. If you throw them away, the AI cannot be distracted by them.

The "Two-Layer" Trick

The paper also discovered that the AI brain works in layers, like a factory assembly line:

• Early Layers: These are good at reading the text instructions ("Draw a cat wearing a hat").
• Late Layers: These are good at the actual drawing (the pixels and colors).

UniLongGen uses a clever split:

• For the text instructions, it keeps the relevant text history.
• For the drawing, it keeps only the relevant image history.
  It doesn't mix them up. It gives the "text brain" the text it needs and the "drawing brain" the pictures it needs, and nothing else.

The Result

By using this "Smart Curator" approach:

• Quality stays high: The AI can draw 40+ pictures in a row, and the 40th picture looks just as good as the 1st.
• Characters stay consistent: The main character doesn't turn into a monster halfway through.
• It's faster: Because the AI isn't trying to look at 1,000 old posters, it works much faster and uses less computer power.

In a Nutshell

Old Way: "Remember everything! Keep every single note and picture we ever made!" -> Result: The desk gets so messy the AI can't work.

UniLongGen Way: "Only keep the 5 most important pictures and the 5 most important notes. Throw the rest in the trash so we don't get distracted." -> Result: The AI stays focused, creative, and consistent for as long as the story goes on.

It's not about having a bigger memory; it's about having a cleaner workspace.

Technical Summary

Here is a detailed technical summary of the paper "UniLongGen: Taming Long-Horizon Interleaved Image Generation via Context Curation."

1. Problem Statement: The Reliability Gap in Long-Horizon Generation

Unified Multimodal Models (UMMs) aim to generate long-form narratives by interleaving text and images in a single autoregressive stream. While these models perform well on short sequences, they suffer from a critical reliability collapse as the sequence length increases.

• The Phenomenon: As the number of interleaved image events grows (e.g., beyond 20 images), generation quality rapidly deteriorates. Images develop severe artifacts, structural distortions, and identity/style drift, eventually becoming unrecognizable.
• The Misconception: Conventional wisdom attributes this to a generic "long-context problem" (token overflow or memory limits).
• The Reality: The authors argue this is a modality-specific failure. Unlike text, where long history causes passive dilution (vague outputs), dense visual history causes active pollution. The accumulation of visual tokens creates a "heavy-tailed" noise distribution in the attention mechanism, where spurious high-similarity outliers hijack the attention budget, injecting incorrect details into the synthesis.

2. Key Mechanistic Insights

Through systematic diagnosis, the authors identified three core mechanisms driving this collapse:

1. The Event Bottleneck: Degradation is governed by the number of discrete image events, not the raw token count. A model can handle 150k text tokens but collapses after ~20 images. The "Effective Multimodal Context Length" is limited by how many distinct visual events the Softmax mechanism can resolve before saturation.
2. Active Pollution vs. Passive Dilution:
   • Text History: Causes passive dilution; relevant signals are missed or under-weighted.
   • Visual History: Causes active pollution. Because historical visual tokens share the same representational space as the target image, irrelevant features are not just ignored; they are mixed into the conditioning. Spurious high-frequency matches in historical keys capture disproportionate attention mass (due to Softmax amplification), corrupting the current synthesis step.
3. Depth-Wise Specialization: Attention allocation in unified models is specialized by layer depth:
   • Early Layers: Dominated by text and ViT tokens (grounding and multimodal routing).
   • Late Layers: Dominated by VAE tokens (direct image synthesis).
   • There is a zero-sum competition between text grounding and visual synthesis across layers.

3. Methodology: UniLongGen

UniLongGen is a training-free, inference-time context curation strategy designed to prevent visual attention hijacking. Instead of retaining all history or compressing it (which preserves noise), it actively curates the model's memory.

Core Components:

1. One-Shot Context Profiling: Before generating a new image, the model performs a single forward pass with the full dense history to probe internal attention signals.
2. Dual-Depth Relevance Scoring:
   • Text Selection (Grounding): Uses an early layer (e.g., Layer 1) to score historical text blocks. This ensures the model remains grounded in the narrative instructions.
   • Image Selection (Synthesis): Uses a late layer (e.g., Layer 15) to score historical VAE image blocks. This selects images that are most relevant for visual consistency (identity/style) based on the model's own synthesis dynamics.
3. Layer-Split KV Visibility Policy:
   • The model enforces a fixed visibility policy during generation:
     • Early Layers: Attend to the text-filtered history.
     • Late Layers: Attend to the image-filtered history.
   • Eviction over Compression: Non-selected tokens are directly dropped from the KV cache rather than compressed (e.g., via pooling). This is crucial because compression often retains the "spurious competitors" that cause pollution, whereas dropping them removes the source of the heavy-tailed noise.

4. Experimental Results

The authors evaluated UniLongGen on a benchmark of 40-image interleaved sequences using the BAGEL model (a hybrid AR-diffusion UMM).

• Performance Metrics:
  • Quality: Significant improvement in HPS v3 (from ~3.17 to 7.57) and PickScore.
  • Consistency: Dramatic gains in Subject Identity (DINOv2) and Style Consistency (GPT-5.2 based evaluation).
  • Stability: While baseline models collapse after ~20 images, UniLongGen maintains high fidelity and coherence up to 40 images.
• Efficiency: By evicting non-relevant history, UniLongGen reduces the KV cache footprint and inference time. It achieves up to 11x speedup compared to dense KV baselines at long context lengths (e.g., 350k tokens).
• Ablation Findings:
  • Event-Level vs. Token-Level: Selecting whole image events is superior to token-level pruning. Token-level pruning leaves fragments of dense keys that still cause attention hijacking.
  • Model Alignment: Attention-based selection (using the model's internal signals) outperforms "Semantic Oracles" (human-selected relevant images), proving that semantic relevance does not always equate to generative utility.
  • Compression vs. Dropping: Direct dropping of tokens outperforms compression methods (AvgPool, MaxPool, Interpolation), which can amplify artifacts.

5. Significance and Contributions

• Redefining Long-Context Limits: The paper shifts the paradigm from "fitting more tokens" to "managing event-level competition." It establishes that visual history has a unique, active destructive potential in unified generation that differs fundamentally from text.
• Mechanistic Diagnosis: It provides a rigorous explanation of why long-horizon generation fails (tail-risk hijacking via Softmax) and identifies the structural vulnerability in unified autoregressive paradigms.
• Practical Solution: UniLongGen offers a plug-and-play, training-free solution that significantly extends the reliable generation horizon of existing models without architectural changes. It demonstrates that active forgetting (curating memory) is more effective than total recall for stability.
• Generalizability: While tested on hybrid AR-diffusion models, the principles of model-aligned curation and depth-aware selection are shown to be adaptable to pure autoregressive models, though the gains vary based on the model's inherent attention dynamics.

In summary, UniLongGen solves the long-horizon collapse in multimodal generation by recognizing that "more context" is not always better. By dynamically curating the history to remove visual noise and aligning the curation with the model's internal layer specialization, it enables the generation of coherent, high-quality, 40+ image narratives.