💻 computer science

MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration

This paper introduces MUSE, a multi-agent framework that addresses the challenges of generating long-form audio-visual stories by employing a closed-loop cognitive orchestration process to iteratively plan, execute, verify, and revise content, thereby ensuring narrative coherence and identity consistency across extended sequences.

Original authors: Wenzhang Sun, Zhenyu Wang, Zhangchi Hu, Chunfeng Wang, Hao Li, Wei Chen

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenzhang Sun, Zhenyu Wang, Zhangchi Hu, Chunfeng Wang, Hao Li, Wei Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to tell a long, complex story to a friend, but you only have a single sentence to start with: "A boy goes on an adventure in a forest."

If you ask a standard AI to turn that sentence into a movie, it might make a beautiful first scene. But by the time it gets to the fifth or tenth scene, the boy's face might look like a different person, the forest might suddenly turn into a desert, or the story might forget what happened in the first scene. The AI gets "drunk" on the immediate task of making the next picture, losing the big picture of the whole story.

MUSE is a new system designed to fix this. Think of MUSE not as a single artist, but as a highly organized film production crew working together to ensure the story stays consistent from the first frame to the last.

Here is how MUSE works, broken down into simple parts:

1. The Problem: The "Drunk Artist"

Current AI video generators are like a talented but short-sighted artist. They are great at painting one beautiful picture based on a prompt. But if you ask them to paint a whole movie, they forget the rules. The main character might change clothes, the lighting might flip from day to night randomly, or the character's voice might sound like a different person in the next scene. This is called "semantic drift."

2. The Solution: A "Closed-Loop" Film Crew

MUSE solves this by treating storytelling like a closed-loop factory rather than a one-way street. It uses a team of specialized AI agents (digital workers) that constantly check and correct each other.

The process happens in three main stages, like a real movie production:

Pre-Production (The Casting Director & Scriptwriter):
Before any video is made, MUSE creates a "Character Bible." It locks in exactly what the main character looks like (age, clothes, style) and how they sound (voice pitch, tone). It creates a digital "ID card" for the character. This ID card is the rulebook that must be followed for the entire movie.
- Analogy: Imagine a casting director who takes a perfect photo and a voice recording of an actor and says, "This is who we are using. Do not change a thing."
Production (The Director & Set Designer):
Now, the system starts generating the scenes. But it doesn't just guess. It uses the "ID card" to ensure the character looks the same. If the script says the character is holding a sword, MUSE checks the layout to make sure the sword is actually there and not floating in mid-air.
- Analogy: The director is constantly shouting, "Wait! That actor looks different! Fix it!" before the camera even rolls.
Post-Production (The Editor & Continuity Supervisor):
Once the scenes are made, MUSE checks the "seams" between them. Did the character's hand move smoothly from the last shot to this one? Did the story flow logically? If the AI accidentally made the character walk through a wall or if the voice sounds too happy for a sad scene, MUSE catches it.
- Analogy: An editor who watches the movie and says, "This scene doesn't match the last one. Let's fix the transition before we show it to the audience."

3. The "Plan-Execute-Verify-Revise" Loop

The secret sauce of MUSE is that it doesn't just generate and hope for the best. It follows a strict cycle:

Plan: Decide what needs to happen next.
Execute: Generate the video/audio.
Verify: A "critic" agent checks if the result matches the plan and the character ID.
Revise: If something is wrong (e.g., the character's hat is missing), MUSE fixes only that specific part, rather than throwing away the whole scene and starting over.

4. The New Test: MUSEBench

How do you know if a story is good if there is no "correct" answer? The authors created a new testing ground called MUSEBench.
Instead of comparing the AI's output to a perfect reference video (which doesn't exist for creative stories), MUSEBench uses a smart AI judge to grade the story on things like:

Consistency: Did the character stay the same?
Story Flow: Did the plot make sense?
Audio-Visual Harmony: Did the voice match the mood of the scene?

The Result

The paper shows that MUSE is much better at keeping long stories coherent than previous methods. It keeps characters looking and sounding the same, maintains the right mood, and ensures the story doesn't fall apart as it gets longer.

In short: MUSE is like a super-organized film crew that refuses to let the story get messy. It plans ahead, checks its work constantly, and fixes mistakes immediately, ensuring that a simple prompt can turn into a long, immersive, and consistent movie.

Technical Summary: MUSE – A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration

1. Problem Definition

The paper addresses the fundamental challenge of generating long-form, coherent audio-visual stories from short, unstructured user prompts. Current multimodal generation systems typically rely on feed-forward pipelines or prompt-only refinement. As the generation horizon extends, these approaches suffer from an intent–execution gap: high-level narrative intent (character identity, spatial relationships, causal progression) fails to persist across dozens of generation steps. This leads to semantic drift, identity inconsistency (characters changing appearance or voice), and temporal discontinuity between shots. The authors argue that these failures stem not merely from imperfect generative models, but from the lack of persistent, machine-interpretable constraints that bridge abstract narrative intent and stochastic, local execution.

2. Methodology: The MUSE Framework

MUSE reframes long-form storytelling not as a single-pass generation task, but as a closed-loop constraint enforcement problem. It employs a modular multi-agent system coordinated by an omni-modal controller that executes an iterative Plan–Execute–Verify–Revise loop.

2.1 Core Architecture

The framework decouples high-level planning from low-level execution, maintaining a shared state memory ( $H$ ) that stores persistent information (character identities, shot constraints, accepted layouts) across steps.

Explicit Control Representation: Instead of relying on natural language retries, MUSE translates narrative intent into explicit, machine-executable controls (e.g., identity anchors, bounding box layouts, temporal boundaries).
Structured Feedback: Verification is performed via structured, typed multimodal signals (e.g., identity mismatch, layout violation) rather than heuristic self-refinement. This enables targeted, bounded revisions rather than unconstrained resampling.

2.2 Three-Phase Orchestration

The generation process is divided into three specialized phases managed by distinct agent teams:

Pre-production (Identity Anchoring):
- Goal: Establish global narrative states that remain invariant (character identity, style).
- Mechanism: The system decomposes the script and constructs persistent multimodal identity states ( $z_c = \{z^{(c)}_{vis}, z^{(c)}_{voc}\}$ ).
- Innovation: It introduces Vocal Trait Synthesis (VTS), which derives stable vocal representations directly from semantic descriptors (age, gender, timbre) without requiring reference audio. This "locks" acoustic identity prior to generation, preventing voice drift.
- Verification: Checks instruction alignment and inter-character consistency before any rendering occurs.
Production (Spatial Composition):
- Goal: Synthesize shot-level visual and audio assets while enforcing spatial and cinematic constraints.
- Mechanism: Uses Dynamic Routing to choose between direct generation and layout-guided synthesis. When spatial control is needed, it synthesizes a coarse layout ( $L^{(i)}_{bbox}$ ) as a hard structural prior for the visual generator.
- Verification: Evaluates entity presence, visual coherence, and integration quality. Revisions are localized to failure regions (e.g., adjusting layouts or modifying generation routes) rather than regenerating the entire shot.
Post-production (Temporal Synthesis):
- Goal: Assemble static assets into temporally coherent video shots.
- Mechanism: Models temporal generation as a state-conditioned process. Each shot is generated conditioned on the current script segment and the terminal state of the previous shot ( $Tail(v_{i-1})$ ).
- Innovation: Utilizes a Context Sliding Window to prevent state resets and motion discontinuities.
- Verification: Audits boundary compliance and narrative leakage (e.g., ensuring the next scene's events do not appear prematurely).

3. Evaluation Protocol: MUSEBench

Recognizing that open-ended storytelling lacks ground-truth references, the authors introduce MUSEBench, a reference-free evaluation protocol.

Scope: Contains 30 curated narrative prompts across five genres (Thriller, Daily Life, Period, Sci-Fi, Fantasy) with varying complexity.
Metrics: Evaluates script quality, visual consistency, visual-script alignment, and audio generation. It utilizes Large Multimodal Models (LMMs) for scoring, validated against human judgments.
Key Metrics: Includes Narrative State Resolution (NSR), Story Expansion Richness (SER), Creative Elaboration Score (CES), and a novel Narrative Effectiveness Score (NES) to measure audio-visual synergy (penalizing tautological narration).

4. Experimental Results

The paper compares MUSE against representative baselines (Vlogger, AnimDirector, MMStoryAgent, V-GOT, MovieAgent) on both ViStoryBench (reference-dependent) and MUSEBench (reference-free).

Quantitative Performance: MUSE substantially outperforms baselines in long-horizon narrative coherence and cross-modal identity consistency. On MUSEBench, it achieves higher scores in script quality (NSR, SER, CES) and visual-script alignment.
Identity Consistency: MUSE demonstrates significant improvements in Character Identity Score (CIDS) and reduced Character Style Distance (CSD), particularly in maintaining non-human character identities and preventing structural hallucinations.
Audio-Visual Synergy: Unlike baselines that often lack audio or use generic TTS, MUSE's VTS module enables text-driven customized audio synthesis, achieving superior consistency in age, emotion, and prosody compared to models like CosyVoice2 and F5-TTS in continuation tasks.
Ablation Studies: Experiments confirm that both the Planning module (for rational script decomposition) and the Feedback module (for targeted revisions) are critical. The feedback mechanism specifically drives improvements in identity preservation and style coherence.

5. Significance and Claims

The paper claims three primary contributions:

Reformulation of the Problem: It shifts the paradigm from unconstrained generation to closed-loop constraint enforcement, explicitly modeling narrative intent as machine-executable constraints to bridge the intent–execution gap.
The MUSE Framework: It proposes a multi-agent system that enforces global consistency through a structured loop, using explicit control representations and typed multimodal feedback to enable targeted corrections across vision, audio, and time.
MUSEBench: It introduces a reference-free, multi-dimensional evaluation protocol that enables holistic assessment of open-ended storytelling systems without restricting generation to predefined scripts.

The authors conclude that MUSE demonstrates that enforcing global narrative constraints via closed-loop orchestration substantially improves the cinematic quality and coherence of long-form audio-visual stories. They acknowledge limitations, such as difficulties in maintaining identity in crowded/occluded scenes and the need for richer control over expressive speech, but assert that their approach provides a scalable path toward reliable, long-horizon storytelling.