InfinityStory: Unlimited Video Generation with World Consistency and Character-Aware Shot Transitions

The paper presents InfinityStory, a novel framework, dataset, and model that overcome key limitations in long-form video generation by ensuring background and character consistency across shots while enabling seamless multi-subject transitions for hour-long narratives.

The Gist

Imagine you are a director trying to film a full-length movie, but you only have a magic camera that can take 5-second clips. You want to tell a story about a knight, a dragon, and a princess, but every time you try to stitch these clips together, two terrible things happen:

1. The Background Glitch: In one clip, the castle is made of stone with a blue sky. In the very next clip, the castle suddenly looks like it's made of jelly with a purple sky. The world keeps changing its mind.
2. The Teleporting Actors: The knight walks into the frame in one clip, but in the next clip, he just poofs into existence in the middle of the room, or vanishes instantly without walking out. It looks like a magic trick gone wrong, not a movie.

InfinityStory is a new AI system designed to fix exactly these two problems. It's like a super-smart production team that ensures the world stays consistent and the actors move naturally from scene to scene.

Here is how it works, broken down into simple concepts:

1. The "Location Library" (Fixing the Background)

Most AI video makers treat every scene as a brand-new painting. If you ask for a "forest," the AI might draw a forest with pine trees in one shot and oak trees in the next.

InfinityStory's Solution:
Think of the AI as having a digital "Location Library." Before the movie starts, it creates a few permanent "sets" (like a specific castle, a specific forest, a specific room).

• Once the "Castle" set is built, the AI locks it in.
• Every time the story returns to the castle, the AI doesn't redraw it; it pastes the characters onto the exact same background image.
• The Analogy: Imagine you are making a stop-motion movie. Instead of rebuilding the set every time you move the camera, you build the set once and keep it there. You just move the clay figures around. This ensures the castle never changes its shape or color, no matter how many scenes you film.

2. The "Choreographer" (Fixing the Actors)

Usually, when AI switches from one clip to another, it doesn't know how to handle characters entering or leaving. It's like a play where actors run off stage and the next scene starts with them already standing in the center, looking confused.

InfinityStory's Solution:
The system uses a special "Choreographer" (a smart AI agent) that plans the movement before the video is made.

• It creates a special dataset of 10,000 practice videos showing exactly how characters should walk in, walk out, or swap places.
• It teaches the AI a specific rule: "No teleporting allowed." If a character leaves the frame, they must walk out. If they enter, they must walk in.
• The Analogy: Think of a dance instructor. A normal AI just snaps its fingers and the dancers appear. InfinityStory's AI is the instructor who says, "Okay, the knight needs to walk from the left door to the center, and the dragon needs to fly in from the right." It fills the gap between the clips so the movement feels smooth and continuous.

3. The "Assembly Line" (How the Movie is Made)

The system uses a team of AI "agents" (specialized robots) that work together like a movie studio:

• The Screenwriter: Breaks the story into chapters and scenes.
• The Location Manager: Assigns a specific "set" to every scene so the background never drifts.
• The Director: Decides who is in the shot and what they are doing.
• The Editor: Uses two different tools to stitch the movie together:
  • Tool A (The Storyteller): Generates the main action clips (e.g., the knight fighting the dragon).
  • Tool B (The Bridge Builder): Generates the transition clips. This is the secret sauce. It takes the last frame of the previous clip and the first frame of the next clip and creates a smooth video that connects them, ensuring the characters move naturally between the two moments.

Why Does This Matter?

Before this, making a long AI video was like trying to build a house out of sand; the walls would crumble and shift as you added more rooms.

InfinityStory is the first system that can build a stable, hour-long movie where:

• The world feels real and consistent (the castle stays a castle).
• The characters feel alive (they don't teleport; they walk and talk).
• The transitions are smooth, like a real Hollywood movie, not a jarring slideshow.

In short, it takes the "magic" out of the glitches and replaces it with the "logic" of a real film production, allowing us to finally tell long, coherent stories with AI.

Technical Summary

1. Problem Statement

The paper addresses three critical limitations in current long-form video generation (storytelling) systems:

• Background Inconsistency (Scene Drift): Existing methods often regenerate backgrounds for every shot, leading to subtle but accumulating deviations in lighting, texture, and spatial layout. This breaks visual coherence in long narratives (e.g., hours long).
• Jarring Shot-to-Shot Transitions: Most pipelines generate independent clips and concatenate them. This results in abrupt character appearances/disappearances (e.g., a character suddenly vanishing or appearing without movement) and lacks cinematic fluidity, especially in multi-subject scenarios.
• Scalability: Current state-of-the-art models struggle to maintain consistency over hundreds of shots or hour-long narratives, often limited to short clips (4–16 seconds).

2. Methodology

InfinityStory proposes an end-to-end framework built on a hierarchical multi-agent system and two core technical innovations: Location-Grounded Background Injection and Cinematic Multi-Subject Transition Synthesis (CMTS).

A. Multi-Agent Narrative Planning

The system decomposes a high-level story specification into a hierarchical structure: Chapters $\to$ Locations $\to$ Scenes $\to$ Shots.

• Chapter Agent: Breaks the story into major plot beats and defines character relationships.
• Location Agent: Creates a fixed library of reusable locations (e.g., "Castle," "Forest") with specific background descriptions. Crucially, it binds specific scenes to specific locations to ensure background stability.
• Scene Agent: Assigns characters and locations to scenes, enforcing constraints (e.g., alternating locations between adjacent scenes).
• Shot Agent: Decomposes scenes into individual shots. It alternates between two generation types:
  • Odd Shots (Narrative): Generated via Image-to-Video (I2V) using a keyframe.
  • Even Shots (Transitions): Generated via First-Last-Frame-to-Video (FLF2V) to bridge consecutive narrative shots.

B. Background Consistency via Location Injection

To prevent scene drift, the authors decouple background generation from foreground dynamics:

1. Canonical Backgrounds: A Text-to-Image (T2I) model generates a fixed, high-quality background image ($B_\ell$) for each location in the library.
2. Keyframe Fusion: For every shot, an Image-to-Image (I2I) model fuses the fixed background ($B_\ell$) with character reference images to create a consistent keyframe ($K_k$).
3. Video Generation:
   • Narrative Shots: Generated from the fused keyframe ($K_k$) using I2V.
   • Transition Shots: Generated using FLF2V, taking the last frame of the previous shot and the next shot's fused keyframe ($K_{k+1}$) as endpoints. Since both endpoints share the same fixed background, the transition naturally preserves the scene identity.
4. Memory Gating: A cross-shot memory module accumulates identity and layout information to limit drift over long sequences.

C. Cinematic Multi-Subject Transition Synthesis (CMTS)

To solve abrupt character changes, the authors introduce a specialized transition model:

• Transition Logic: Even-indexed shots are treated as transitions with explicit metadata ($\tau_k$) describing character entry, exit, or replacement (e.g., "Character A walks in from the left").
• Synthetic Dataset Creation: Since real-world data for specific multi-subject transitions is scarce, they generated a synthetic dataset of 10,000 transition videos.
  • Pipeline: Multi-agent planning $\to$ Video generation (Wan2.2) $\to$ VLM Filtering (using Qwen3-VL to strictly count characters in first/last frames to ensure 0-to-N or N-to-0 transitions are valid).
  • Result: A curated dataset of ~3,980 high-quality videos covering entry, exit, and replacement scenarios.
• Model Training: The FLF2V model (based on Wan2.1-14B) is fine-tuned using LoRA on this dataset to learn smooth, choreographed movements rather than sudden cuts.

3. Key Contributions

1. Cinematic Multi-Subject Transition Synthesis (CMTS): The first method to systematically address smooth, multi-subject shot transitions (entry/exit/replacement) with explicit movement supervision, moving beyond single-subject limitations.
2. Background-Consistent Generation Pipeline: A novel approach using "Location Injection" that anchors scenes to fixed backgrounds, effectively eliminating scene drift in long-form generation.
3. Large-Scale Synthetic Dataset: A curated dataset of 10,000 annotated multi-subject transition sequences, filtered by VLMs, to train robust transition models.
4. Scalable Framework: An end-to-end system capable of generating hour-long narratives with coherent story structure, consistent characters, and stable environments.

4. Experimental Results

The model was evaluated on the TinyStories dataset (10 multi-character stories) using VBench metrics.

• Overall Performance: InfinityStory achieved the best average rank (2.80) among all baselines.
• Consistency Metrics:
  • Background Consistency: 88.94 (Highest among all models, significantly outperforming MovieAgent at 79.84).
  • Subject Consistency: 82.11 (Highest, outperforming the next best by ~3.5 points).
• Comparisons: It outperformed strong baselines like MovieAgent, VideoGen-of-Thought, and StoryAdapter in maintaining visual coherence.
• Trade-offs: Image Quality and Aesthetic scores were slightly lower than some baselines. The authors attribute this to the 480p resolution used (vs. 720p for base generators) and minor artifacts introduced by the image-editing (I2I) fusion step.
• Ablation Studies: Removing background injection caused a significant drop in Subject Consistency (-3.6) and Background Consistency (-1.6). Disabling multi-character transitions reduced motion smoothness and aesthetic quality.

5. Significance

InfinityStory represents a significant leap forward in generative AI for storytelling. By solving the "scene drift" problem and introducing smooth, multi-character transitions, it moves video generation from short, disjointed clips toward cinematic, long-form narratives.

• Practical Impact: It enables the creation of consistent, hour-long videos suitable for education, entertainment, and digital media without manual frame-by-frame editing.
• Technical Advancement: It demonstrates that explicit structural constraints (location binding) and specialized training data (synthetic transition datasets) are more effective than relying solely on prompt engineering or long-context attention mechanisms.
• Future Direction: The work lays the foundation for future systems capable of generating complex, multi-hour movies with rich character dynamics and stable world-building.