💻 computer science

OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

This paper introduces OmniScript, an efficient 8B-parameter omni-modal language model and a new human-annotated benchmark designed to tackle the novel video-to-script task by generating detailed, temporally grounded, hierarchical scripts for long-form cinematic videos.

Original authors: Junfu Pu, Yuxin Chen, Teng Wang, Ying Shan

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junfu Pu, Yuxin Chen, Teng Wang, Ying Shan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a 2-hour movie. Now, imagine asking a computer to watch that entire movie and write a perfect, detailed screenplay for it, scene by scene, including exactly what the characters are doing, saying, feeling, and what the background music sounds like, all with precise timestamps.

That is the massive challenge this paper tackles. The authors introduce OmniScript, a new AI system designed to do exactly that.

Here is the breakdown of their work using simple analogies:

1. The Problem: The "Short-Attention Span" AI

Current AI models are like tourists on a bus. They can look out the window and say, "Oh, that's a mountain!" or "Look, a dog!" for a 30-second clip. But if you ask them to watch a whole movie and write a story about it, they get lost. They forget who the characters are, mix up the plot, or miss the subtle emotional shifts.

Writing a script from a video is hard because:

It's too long: A movie has thousands of moments.
It's complex: You need to track who is who, what they are saying, and how they feel, all while the camera moves and music changes.
It's expensive: Writing a script for a 2-minute clip takes a lot of computer power; doing it for a whole movie is currently impossible for most AIs.

2. The Solution: OmniScript (The "Super-Editor")

The team built OmniScript, an AI that acts like a super-intelligent film editor and screenwriter rolled into one.

It's "Omni-modal": Unlike other AIs that just "see" the video, OmniScript has ears too. It listens to the dialogue, the sound effects, and the music. It understands that a character's "angry" face combined with "shouting" and "drumming music" creates a specific mood.
It's Efficient: Even though it's very smart, it's surprisingly small (only 8 billion parameters). Think of it as a F1 race car: it's lightweight and fast, yet it beats the massive, heavy trucks (larger AI models) in the race.

3. How They Trained It: The "Memory Gym"

To teach OmniScript, the researchers had to solve three big problems:

A. The Data Problem (The "Blank Canvas")
There were no existing datasets where humans had written detailed scripts for full movies.

The Fix: They built a massive, high-quality dataset from scratch. They used a "human-in-the-loop" system where a powerful AI generated a draft, and human experts polished it, ensuring the character names and plot points were perfect.

B. The Memory Problem (The "Who's Who?")
In a long movie, a character might wear a different coat in Scene 100 than in Scene 1. A normal AI would think they are two different people.

The Fix: They gave OmniScript a "Character Profile Manager." Imagine a detective's corkboard with photos and notes. As the AI watches the movie, it updates this board: "Okay, that's still John, but now he's wearing a red hat and looks tired." This keeps the story consistent from start to finish.

C. The Training Method (The "Thinking Step")
Instead of just guessing the next word, the AI was taught to think first.

The Fix: They used a "Chain of Thought" approach. Before writing the script, the AI writes a mental summary: "Okay, this is the climax. The hero is angry because the villain lied. The music is tense." Only after this "thinking" phase does it write the actual script. This prevents the AI from getting confused.

4. The Evaluation: The "Strict Film Critic"

How do you grade a script? You can't just use a spellchecker.

The Fix: They created a new grading system that checks four things separately:
1. Did it happen at the right time? (Timing)
2. Was it the right character? (Identity)
3. Did they say the right thing? (Dialogue)
4. Did they do the right action? (Action)
They even built a system to handle "open vocabulary." If the AI says "The guy in the blue suit" and the ground truth says "John Smith," the system knows they are the same person and gives credit, rather than marking it wrong.

5. The Results: Beating the Giants

When they tested OmniScript:

It beat the giants: It performed as well as (and sometimes better than) massive, expensive, closed-source models like Google's Gemini 3-Pro.
It handled long videos: While other AIs started to hallucinate and forget the plot after 20 minutes, OmniScript kept its cool.
The "Two-Stage" Trick: For very long movies (like 40+ minutes), they used a strategy called Two-Stage Script Generation.
- Stage 1: A planner breaks the movie into chunks and outlines the plot.
- Stage 2: OmniScript writes the detailed script for each chunk based on that outline.
- Analogy: It's like writing a novel. You don't write the whole book in one breath; you write an outline, then write chapter by chapter, and finally stitch them together.

The Bottom Line

OmniScript is a breakthrough because it finally allows computers to understand long, complex stories visually and audibly, turning raw video footage into structured, readable scripts. It proves that you don't need a "giant" brain to understand a movie; you just need the right memory, the right listening skills, and a good plan.

This technology could revolutionize how movies are archived, searched, and even how new stories are created by helping human writers visualize their ideas.

1. Problem Definition

The paper addresses the Video-to-Script (V2S) task, defined as the generation of detailed, temporally grounded, hierarchical scripts from long-form cinematic videos (movies, TV episodes, short dramas). Unlike existing video understanding tasks that focus on short clips, captioning, or question-answering, V2S requires:

Long-Range Context: Processing videos ranging from several minutes to hours.
Hierarchical Structure: Generating a script organized by Scenes (location, time, mood) and Events (timestamped actions, dialogues, expressions, and audio cues).
Multimodal Fidelity: Accurately capturing not just visual actions but also off-screen dialogue, background music (BGM), sound effects, character emotions, and subtext.
Challenges: The field suffers from a lack of high-quality training data (due to the labor intensity of granular annotation), the absence of robust evaluation metrics for open-ended narrative generation, and the computational inefficiency of autoregressive models generating thousands of tokens for long videos.

2. Methodology

A. The OmniScript Model Architecture

OmniScript is an 8B-parameter omni-modal language model built on the Qwen3-VL backbone, enhanced with an AV-DeepStack fusion strategy.

Multimodal Temporal Alignment: Visual frames and raw audio are encoded separately (using a Vision Encoder and Whisper for audio) and aligned at the timestamp level to create paired $(v_t, a_t)$ representations.
AV-DeepStack Injection: Unlike standard models that inject features only at the input, OmniScript injects paired audio-visual tokens into multiple layers of the LLM via residual adapters. This enables deep cross-modal interaction throughout the reasoning process, crucial for understanding dialogue, off-screen narration, and acoustic events.
Reasoning-Guided Decoding: The model employs a Chain-of-Thought (CoT) paradigm. Before generating the final structured script, it produces an intermediate reasoning trace that summarizes plot progression and character relationships. This scaffold ensures global consistency and mitigates long-context ambiguity.

B. Data Construction: Memory-Augmented Annotation Pipeline

To overcome data scarcity, the authors constructed the first large-scale, human-annotated V2S benchmark using a novel pipeline:

Segmentation: Videos are split into semantic segments (<5 mins).
Character Profile Manager (CPM): A memory module tracks characters across segments, updating profiles with appearance and relationship changes. It uses a "lazy naming" strategy to resolve identities (e.g., merging "man in black" to "John" once named).
CoT Generation: A strong LLM (Gemini-2.5-Pro) generates plot descriptions and character reasoning based on the CPM.
Fine-Grained Scripting: A second MLLM translates these high-level plots into detailed, timestamped scripts.
Thinking Data: A distillation step extracts "intermediate thinking" from the scripts to create a Video $\to$ Thinking $\to$ Script dataset for training.

C. Training Strategy

OmniScript is trained via a progressive four-stage pipeline:

Modality Alignment: Aligning audio features with the visual-language space using timestamped ASR data.
Multimodal Pretraining: Large-scale training on 2.4M bilingual videos for temporal grounding and narrative understanding.
Supervised Fine-Tuning (SFT): Training on 45K curated videos with CoT traces to enforce schema adherence and narrative coherence.
Reinforcement Learning (RL): Using GRPO with a temporally segmented reward. Unlike global rewards, this evaluates the script at the event level, penalizing fine-grained hallucinations and misses in specific fields (dialogue, action, etc.).

D. Long-Video Processing Strategies

To handle videos longer than the model's native context window, two strategies are proposed:

Strategy 1 (OmniScript-LCE): Long-Context Extension. Directly scales the context window using pseudo-long videos constructed by concatenating short clips with merged annotations.
Strategy 2 (OmniScript-TSG): Two-Stage Script Generation. A plot-segmentation model first predicts scene boundaries and character lists. OmniScript then generates scripts for each segment conditioned on these structural prompts, which are finally merged. This explicitly handles long-range dependencies.

E. Evaluation Framework

The paper introduces a Temporally-Aware Hierarchical Evaluation Framework with four stages:

Event Alignment: Uses dynamic programming and semantic similarity (dialogue/action) rather than strict temporal IoU to match predicted events to Ground Truth (GT).
Character Mapping: Resolves open-vocabulary identity mismatches (e.g., "officer" vs. "John") using an LLM.
Field-Level Evaluation: Assesses semantic correctness of specific fields (Action, Dialogue, Expression, Audio Cue) using LLM-based similarity scoring.
Temporal Boundary Evaluation: Computes tIoU Hit Rate on aligned groups to measure localization accuracy.

3. Key Contributions

First V2S Benchmark: A comprehensive dataset of 10 full-length cinematic works (19.9 hours) with 16.8k+ structural events, featuring dense annotations for actions, dialogues, expressions, and audio cues.
OmniScript Model: The first omni-modal (audio+visual) model specifically designed for long-form narrative script generation, achieving state-of-the-art performance with only 8B parameters.
Novel Training & Evaluation: Introduction of a memory-augmented annotation pipeline, a temporally segmented reward mechanism for RL, and a hierarchical evaluation framework that disentangles semantic accuracy from temporal localization.
Long-Context Solutions: Demonstration that a two-stage segmentation approach (TSG) effectively breaks the "context cliff" where performance degrades linearly with video length in other models.

4. Experimental Results

Performance on 5-Minute Videos: OmniScript (8B) significantly outperforms larger open-source models (e.g., Qwen3VL-235B) and rivals proprietary giants like Gemini-3-Pro.
- Event-Level: Achieves 37.7 Overall F1 and 69.3 tIoU@0.1, surpassing Gemini-3-Pro (38.9 Overall, 64.4 tIoU) in temporal localization and dialogue understanding.
- Efficiency: It achieves these results with 1/30th the parameters of the largest open-source competitors.
Long-Form Video (10–45 mins):
- Context Cliff: Baseline models (including Gemini-3-Pro and OmniScript-LCE) suffer severe performance degradation (linear drop in F1) as video duration exceeds 25–30 minutes.
- TSG Superiority: OmniScript-TSG demonstrates length-invariant robustness, maintaining flat performance curves across 10–45 minute videos, effectively bypassing the context limitations that plague global-context models.
Ablation Studies:
- Audio Input: Removing audio causes a massive drop in Dialogue accuracy (from 68.2% to 52.0%), proving the necessity of audio-visual reasoning.
- Subtitle Robustness: When subtitles are masked, OmniScript degrades moderately, whereas non-omni models (like Qwen3VL) collapse, confirming OmniScript relies on genuine multimodal cues rather than text shortcuts.
- CoT & RL: Both Chain-of-Thought reasoning and Segmented Reward RL significantly boost performance in dialogue and temporal localization.

5. Significance

This work represents a paradigm shift in video understanding, moving from short-clip analysis to holistic narrative reconstruction.

Industry Impact: It provides a viable path for automating script logging, content retrieval, and assisting human screenwriters by converting raw footage into structured, searchable scripts.
Technical Advancement: It proves that parameter efficiency combined with omni-modal fusion and structured reasoning can outperform massive proprietary models.
Methodological Insight: The paper highlights that for long-form video, explicit segmentation and structural planning (Strategy 2) are more effective than simply scaling context windows, offering a blueprint for future long-context multimodal research.