💬 NLP

CANVAS: Continuity-Aware Narratives via Visual Agentic Storyboarding

The paper introduces CANVAS, a multi-agent framework that explicitly plans visual continuity through character consistency, persistent background anchors, and location-aware scene planning to overcome the coherence challenges in long-form visual storytelling, demonstrating significant improvements over existing baselines on multiple benchmarks.

Original authors: Ishani Mondal, Yiwen Song, Mihir Parmar, Palash Goyal, Jordan Boyd-Graber, Tomas Pfister, Yale Song

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ishani Mondal, Yiwen Song, Mihir Parmar, Palash Goyal, Jordan Boyd-Graber, Tomas Pfister, Yale Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are directing a movie. You have a script that tells a story about a detective, a thief, and a golden artifact in a museum. You ask an AI to draw the story, scene by scene.

Here is the problem: Most current AI artists are like amateur sketchers who forget what they drew five minutes ago.

In Scene 1, the detective wears a blue trench coat and has a scar on his left cheek.
In Scene 2, the AI draws him again, but now he's wearing a red suit and the scar is on his right cheek.
In Scene 3, the museum background changes from a marble floor to a wooden one, even though the story says they never left the room.
In Scene 4, the golden artifact vanishes, even though the thief hasn't stolen it yet.

This is called a continuity error. It breaks the magic of the story.

Enter CANVAS: The "Super-Producer" AI

The paper introduces CANVAS (Continuity-Aware Narratives via Visual Agentic Storyboarding). Think of CANVAS not just as a drawer, but as a Super-Producer who runs a tiny, highly organized movie studio.

Instead of just drawing one picture at a time, CANVAS uses a team of "agents" (specialized AI helpers) to manage the entire production. Here is how it works, using a simple analogy:

1. The Script Supervisor (Global Planning)

Before a single line is drawn, the Script Supervisor reads the whole story and creates a "World Bible."

Character File: "Detective Nora always wears a blue trench coat unless she changes clothes in Scene 5."
Location Map: "The Museum Gallery has marble floors and a specific glass case. If we return to this room later, it must look exactly the same."
Prop Tracker: "The Golden Artifact starts in the case. In Scene 4, the thief steals it. In Scene 5, the case must be empty."

Most other AIs don't have this Bible; they just guess what to draw based on the previous picture. CANVAS knows the whole story before it starts drawing.

2. The Memory Bank (Visual Anchors)

Imagine the AI has a digital photo album that it updates after every single shot.

Character Album: It keeps a perfect reference photo of Detective Nora's face and coat. Every time she appears, the AI checks this album to make sure she looks the same.
Location Album: It saves a photo of the Museum Gallery. If the story jumps back to the museum later, the AI pulls this photo out to ensure the walls and floor don't magically change.
Prop Album: It tracks the Golden Artifact. If the thief steals it, the AI updates the album to show the artifact is now "gone" from the room.

3. The Quality Control Team (Selection)

When the AI needs to draw a new scene, it doesn't just draw one picture. It acts like a director with a camera crew:

It generates multiple drafts (candidates) of the scene.
It sends these drafts to a Quality Control Team (a smart AI judge).
The team asks questions: "Does the detective still have his blue coat? Is the museum floor still marble? Is the artifact still in the case?"
It picks the best draft that answers "Yes" to everything, and throws the rest away.

Why is this a big deal?

The paper tested CANVAS against other top AI tools on a "Hard Continuity Challenge." This was a story designed to be tricky, with characters changing clothes, objects disappearing, and locations reappearing after a long time.

Other AIs: Got confused. Characters looked like different people; rooms changed shape; objects appeared out of thin air.
CANVAS: Kept the story consistent. The detective looked the same, the museum stayed the same, and the stolen artifact stayed stolen.

The Bottom Line

Think of existing AI story generators as improvisational actors who make up the story as they go, often forgetting what they said in the first act.

CANVAS is like a professional film crew with a strict script, a dedicated continuity person, and a memory of every detail. It ensures that if you watch the whole story from start to finish, the world feels real, consistent, and logical, rather than a confusing dream where everything changes every few seconds.

This is a massive step forward for creating long, coherent visual stories, whether for comics, movies, or interactive games.

1. Problem Statement

Long-form visual storytelling requires generating a sequence of images that form a coherent narrative world. While existing generative models (e.g., diffusion-based text-to-image) can produce high-quality individual frames, they fail to maintain long-range continuity across multiple shots. Common failure modes include:

Character Drift: Inconsistent facial features, clothing, or hairstyles when a character reappears.
Unstable Environments: Backgrounds and room layouts changing arbitrarily when a location is revisited.
Logical Inconsistencies: Objects appearing, disappearing, or changing state (e.g., a stolen artifact remaining in a display case) without narrative justification.

Existing approaches typically rely on short-range conditioning (using the previous frame or reference images) or attention mechanisms. These methods lack an explicit representation of the narrative world state, causing errors to accumulate over time and failing to model how characters, environments, and objects evolve logically throughout a story.

2. Methodology: CANVAS Framework

The authors propose CANVAS (Continuity-Aware Narratives via Visual Agentic Storyboarding), a training-free, multi-agent framework that treats storyboard generation as an explicit world-state modeling problem. The system enforces coherence through three core components:

A. Global Story Planning

Before generating any images, a planner agent constructs a global continuity plan ( $P$ ) based on the input story script. This plan explicitly tracks:

Character States ( $C$ ): A mapping of characters to their specific appearance states (face, clothing, hair) at each shot. It handles identity preservation and tracks intentional changes (e.g., a character changing clothes).
Location Assignments ( $L$ ): Clustering shots by location identity to ensure that when a scene reappears, the same spatial structure is reused. It also identifies "fresh" locations vs. "reappearances."
Object State Transitions ( $O$ ): Tracking the state of props (e.g., present, stolen, broken, absent) across the timeline to ensure logical consistency (e.g., if an artifact is stolen in Shot 3, it must be absent in Shot 5).

B. Sequential Generation with Memory

The system generates shots sequentially while maintaining a persistent Visual State Memory ( $M$ ) consisting of:

Character Anchors ( $M_c$ ): Canonical images of characters in specific outfits.
Location Anchors ( $M_l$ ): Background images of specific environments.
Frame Memory ( $M_f$ ): The most recently generated frame for immediate continuity.

For each shot, the system performs Anchor Retrieval:

Queries the global plan to determine required states.
Retrieves relevant anchors from memory (e.g., if returning to a museum, it retrieves the museum background anchor; if a character changed clothes, it retrieves the new outfit anchor).
Constructs a generation prompt combining the shot description with these retrieved visual anchors.

C. QA-Based Candidate Selection

To ensure the generated frame satisfies the continuity constraints, CANVAS employs a Candidate Generation and Selection loop:

Generation: The image generator produces $N$ candidate frames for the current shot.
Evaluation: A Vision-Language Model (VLM) acts as a judge, evaluating each candidate against the global plan using specific Visual Questions (QA). The VLM checks:
- Does the character match the planned appearance?
- Is the background geometry consistent with the location anchor?
- Do the props reflect the correct state (e.g., is the artifact missing if it was stolen)?
Selection: The candidate with the highest consistency score is selected as the final frame ( $I^*_t$ ).
Memory Update: The selected frame is used to extract and update the visual anchors in memory for future shots.

3. Key Contributions

CANVAS Framework: A novel, training-free multi-agent system that explicitly models narrative world states (characters, locations, objects) to enforce long-range visual continuity, moving beyond simple frame-to-frame conditioning.
HardContinuityBench: A new, challenging benchmark designed to stress-test long-range narrative consistency. Unlike existing benchmarks, it features frequent recurring locations with large temporal gaps, character appearance changes, and complex object state transitions.
ContinuityEval: A fine-grained, VLM-based evaluation metric that quantifies continuity across three dimensions: character identity, background geometry, and prop state consistency, showing strong correlation with human judgments.

4. Experimental Results

The authors evaluated CANVAS on ViStoryBench, ST-Bench, and the new HardContinuityBench, comparing it against state-of-the-art training-free baselines (AutoStudio, Story-Iter, Story2Board, and Gemini-CT).

Quantitative Performance:
- Background Continuity: Improved by 21.6% on HardContinuityBench compared to the best baseline.
- Character Consistency: Improved by 9.6%.
- Prop Consistency: Improved by 7.6%.
- CANVAS consistently outperformed all baselines across consecutive and non-consecutive scene transitions.
Human Evaluation:
- In a pairwise user study (240 comparisons), CANVAS achieved win rates of up to 90.9% for background and prop consistency and 86.7% for character consistency against strong baselines like AutoStudio.
Downstream Impact:
- When used as keyframes for Image-to-Video (I2V) generation (using Veo-3.1), CANVAS-generated storyboards resulted in significantly higher video coherence and narrative quality compared to other methods.

5. Significance and Future Work

Significance:
CANVAS addresses a critical bottleneck in AI-driven visual storytelling: the inability to maintain a coherent "world" over long sequences. By decoupling planning (world state) from generation (image synthesis) and using memory to bridge them, the framework enables the creation of complex, multi-shot narratives that were previously impossible with standard generative models. The introduction of HardContinuityBench sets a new standard for evaluating long-range consistency.

Limitations & Future Work:

Computational Cost: The candidate selection process (generating multiple frames and VLM evaluation) increases inference time compared to single-pass generation.
Complex Dynamics: The system currently models static state transitions but does not fully model complex physical interactions or motion dynamics between shots.
Ambiguity: The framework relies on clear text extraction of entities; highly abstract or ambiguous narratives may pose challenges.
Future Direction: The authors plan to explore interactive, human-in-the-loop storytelling where users can refine scenes while maintaining global continuity.

In summary, CANVAS represents a paradigm shift from "generating images" to "simulating a narrative world," offering a robust solution for consistent long-form visual storytelling.