💻 computer science

UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating

UnityShots is a memory-driven multi-shot audio-video generation system built on LTX-2.3 that ensures cross-shot coherence through a boundary-aware gating mechanism for fixed-size visual memory and reference speaker tokens for audio, outperforming open-source baselines on a newly released multi-cultural benchmark.

Original authors: Jiehui Huang, Yuechen Zhang, Bin Xia, Jiahao Wang, Xu He, Zhenchao Tang, Meng Chu, Xin Tao, Pengfei Wan, Jiaya Jia

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiehui Huang, Yuechen Zhang, Bin Xia, Jiahao Wang, Xu He, Zhenchao Tang, Meng Chu, Xin Tao, Pengfei Wan, Jiaya Jia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are directing a movie. You have a script with many different scenes (shots). The biggest challenge in making a movie with AI is keeping the story consistent. If you ask an AI to make Scene 1, then Scene 2, then Scene 3, it often forgets what the main character looked like in Scene 1 by the time it gets to Scene 3. The character's hair might change color, their clothes might shift, or their voice might sound like a different person.

UnityShots is a new AI system designed to solve this "forgetting" problem. Think of it as a super-organized film director who never loses track of the story. Here is how it works, using simple analogies:

1. The Two-Memory System (The "Anchor" and the "Step")

Most AI video generators are like a person with very short-term memory. They only remember the last few seconds. UnityShots is different because it carries two specific types of memory in its back pocket for every new scene:

The Long-Term "Anchor" (LTM): Imagine a photo of the main character pinned to the wall at the very beginning of the movie. No matter how many scenes pass, the AI always glances at this photo to remember: "This is who the character is." This ensures the character's face, clothes, and identity stay the same from the first shot to the last.
The Short-Term "Step" (STM): Imagine the AI also keeps a mental note of exactly how the previous scene ended. Did the character just stand up? Was the camera moving left? This "step" memory helps the new scene flow naturally from the old one, so the movement doesn't look jerky or broken.

2. The "Smart Gatekeeper"

In a normal movie, a director decides when to cut from one scene to the next based on the action or the music. UnityShots has a Smart Gatekeeper that does this automatically.

Visual Cues: It watches for visual changes (like a sudden cut or a new location).
Audio Cues: It listens to the rhythm of the music or the beat of the audio.
The Decision: When the Gatekeeper sees a big change (like a hard cut in the script), it knows to update the "Short-Term Step" memory to match the new action. But, crucially, it never lets go of the "Long-Term Anchor." It knows that even if the scene changes completely, the main character must remain the same.

3. The "Voice Anchor"

Just as the character's face needs to stay consistent, their voice needs to too. UnityShots doesn't try to remember the whole audio track of the movie. Instead, it keeps a single "Voice ID" card for each character. Every time a new scene starts, it shows this card to the AI, saying, "Remember, this is the same person speaking." This prevents the character's voice from sounding like a different person in the next shot.

4. The "Strata" (The Layered Bookshelf)

To keep all this information organized without getting confused, UnityShots uses a special filing system called Strata-RoPE.

Imagine a bookshelf with three distinct, separated shelves:

Top Shelf: Holds the "Long-Term Anchor" (the opening shot).
Middle Shelf: Holds the "Short-Term Step" (the immediate past).
Bottom Shelf: Holds the "Current Scene" (what is being made right now).

Because these shelves are physically separated, the AI never mixes up the old character details with the new scene details. It knows exactly which shelf to look at for which piece of information.

5. The Result: A Consistent Movie

The paper tested this system on a "multi-cultural benchmark" (a collection of 200 different short stories with characters from six different cultural backgrounds and many languages).

Old Methods: Often resulted in "drift," where the character looked different or the voice changed halfway through the story.
UnityShots: Successfully kept the characters looking and sounding the same across multiple cuts, even in long sequences. It performed better than other open-source tools and matched the quality of the best closed-source commercial systems, but with the added ability to handle audio and video together seamlessly.

In summary: UnityShots is like a director who has a permanent photo of the cast on the wall (Long-Term Memory), a notepad of the last action (Short-Term Memory), and a strict rulebook on how to switch scenes (The Gatekeeper). This allows it to generate long, multi-scene videos where the story, the characters, and the voices stay consistent from start to finish.

Technical Summary: UnityShots

Problem Statement

Generating coherent multi-shot audio-video sequences requires maintaining subject appearance, scene context, and speaker identity across shot boundaries. Existing approaches suffer from structural limitations:

End-to-end methods train on fixed-length sequences, limiting scalability to long-form storytelling due to GPU memory constraints.
Shot-by-shot methods often rely on sliding windows or keyframe banks that grow linearly with sequence length, leading to identity drift as the conditioning window moves past the opening shot.
Agent-orchestrated methods use LLM planners to chain pretrained generators, but the underlying models lack native cross-shot coherence training, making signal propagation difficult.

Furthermore, current designs treat memory as a monolithic context window, failing to distinguish between long-term narrative anchors (the opening shot) and short-term local continuity (the immediate preceding tail). Additionally, no existing model jointly utilizes visual and audio boundary signals to dynamically control memory updates at shot cuts.

Methodology

UnityShots is a memory-driven multi-shot audio-video generation system built upon the LTX-2.3 22B dual-stream diffusion transformer. It introduces a structured memory bank and boundary-conditioned gating mechanisms to solve cross-shot coherence.

1. Dual-Stream Memory Bank

The system maintains fixed-size memory slots for both video and audio streams:

Video Memory: Consists of two fixed-size slots updated at every cut:
- Long-Term Memory (LTM): Anchored to the opening shot ( $P_\ell=2$ latent frames). It preserves the global narrative identity. Its contribution is governed by a rule-based coefficient that ensures the anchor is never fully erased, regardless of cut strength.
- Short-Term Memory (STM): Holds the immediately preceding tail ( $P_v=4$ latent frames). It captures local motion dynamics and scene state. Its weight is refined by a content-adaptive MLP conditioned on the outgoing and incoming shot content.
Audio Memory: Instead of a sliding window, the system injects a fixed reference speaker token derived from the reference audio ( $A_{ref}$ ) at every shot. This preserves vocal timbre without maintaining a sliding audio bank.

2. Boundary-Conditioned Memory Gating

Memory updates are driven by a composite boundary score ( $b_N$ ) that fuses visual and audio signals:
$b_N = \tau_N \cdot (\alpha s_{vis} + \beta s_{aud} + \gamma s_{beat})$
Where $\tau_N$ is a discrete cut-type prior (FIRST, CONTINUE, HARD), $s_{vis}$ is the visual-cut probability, $s_{aud}$ is RMS energy change, and $s_{beat}$ encodes beat position.

Gating Coefficients: Two monotone-increasing coefficients ( $g_{ltm}$ and $g_{stm}$ ) are derived from the clipped boundary score. The LTM coefficient is strictly bounded above zero to prevent anchor loss, while the STM coefficient adapts to bridge boundaries.
Update Mechanism: The memory slots are updated before each denoising pass using a bounded update rule that prevents abrupt overwriting of the opening-shot anchor.

3. Strata-RoPE Position Encoding

To distinguish between the current shot, the STM, and the LTM without extra depth tokens, the system employs Strata-RoPE. The temporal RoPE axis is partitioned into three non-overlapping strata:

Current Shot: $[0, T_v - 1]$
STM Band: $[f_{lim} - P_v, f_{lim} - 1]$
LTM Band: $[f_{lim} - 2P_v, f_{lim} - P_v - 1]$
This design allows the rotary kernel to automatically attenuate cross-strata interactions based on distance.

4. Training and Inference

Training: The model is trained on 146k annotated cinematic and music-video clips in two stages. Stage 1 fine-tunes the backbone on single shots. Stage 2 introduces memory slots and gating, training on multi-shot chunks ( $k \in [3, 9]$ ) under mixed conditioning (I2V, T2V, R2V).
Inference: For each shot, the STM receives the tail of the previous shot, and the LTM is updated via the gating mechanism. The backbone denoises the current shot conditioned on the reference identity, memory slots, and reference audio.

Key Contributions

Memory-Driven Dual-Stream Generation: An end-to-end trained backbone with a two-slot video memory bank (LTM/STM) and a reference speaker anchor, enabling control over transition strength via a cut-type prior learned through AdaLN.
Boundary-Conditioned Memory Gating: A mechanism that updates memory banks at every cut using a joint visual-audio boundary signal, distinguishing between global anchors and local continuity.
Multi-Cultural Multi-Shot Benchmark: A release of 200 multi-shot sequences spanning six cultural groups and ten languages, featuring reference identities, audio, and per-boundary transition labels.

Results

Evaluated on the multi-cultural benchmark across I2V, T2V, and R2V modes, UnityShots demonstrates superior performance:

Coherence: It leads open-source baselines on every cross-shot coherence metric (Narrative Coherence, Story, Character consistency). For instance, in I2V mode, it achieves +0.40 NC and +0.69 Story over the strongest open-source baseline.
Comparison with Closed-Source: While closed-source systems like Kling show high per-frame fidelity, UnityShots matches or exceeds them on multi-shot axes (NC, Story, Pace) and significantly outperforms them in audio continuity (AES-A, CLAP) due to its speaker anchor.
Long-Sequence Robustness: Ablation studies confirm that both memory slots are necessary. The full system maintains identity consistency even in sequences of 6+ shots, whereas single-slot variants or baselines suffer from drift.
Human Evaluation: A user study with 32 participants showed UnityShots receiving majority preference for identity consistency and overall quality against all open-source baselines.

Significance

The paper posits that treating memory as a monolithic window is insufficient for long-form generation. By explicitly separating long-term identity anchors from short-term continuity and gating their updates via joint audio-visual boundary signals, UnityShots provides a scalable solution for coherent multi-shot generation. The work bridges the gap between single-shot generation quality and the structural requirements of cinematic storytelling, offering a system compatible with both manual and agent-driven production pipelines. The release of a diverse, multi-cultural benchmark aims to facilitate further research into demographic diversity and cross-shot coherence.