Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing

Tele-Omni is a unified multimodal framework that leverages pretrained large language models to parse diverse text, image, and video instructions for a single diffusion-based model, enabling flexible and high-quality video generation and editing across multiple tasks within a cohesive system.

The Gist

Imagine you have a magical movie studio in your living room. Right now, most AI video tools are like specialized workers: one worker is great at making movies from a script (text), another is great at animating a sketch (image), and a third is great at editing out a background. But if you want to do something complex—like "Take this photo of a cat, put it in a cyberpunk city, make it rain, and then edit the video to make the cat wear sunglasses"—you usually have to hire three different workers and hope they talk to each other. They often don't.

Tele-Omni is the solution to this problem. Think of it as hiring a Super-Producer who can do everything in one go.

Here is a simple breakdown of how Tele-Omni works, using everyday analogies:

1. The Two-Brain System

Tele-Omni isn't just one big brain; it's a team of two specialists working in perfect sync:

• The "Director" (The MLLM): This is the brain that understands language, images, and videos. It's like a seasoned film director who reads your messy notes ("Make the car red, but keep the rain falling") and looks at your reference photos. The Director doesn't paint the frames; instead, they write a very precise shot list and storyboard. They figure out what you want and how to do it.
• The "Cinematographer" (The Diffusion Model): This is the artist who actually paints the movie. They take the Director's shot list and the reference photos and start creating the video frame by frame. They are incredibly skilled at making things look real and keeping the motion smooth.

The Magic: In older systems, the Director and Cinematographer spoke different languages. In Tele-Omni, they speak the same language. The Director translates your complex, mixed instructions (text + photos + video clips) into a clear plan that the Cinematographer can execute perfectly.

2. The "Swiss Army Knife" of Video Tasks

Most video AI tools are like a single-purpose screwdriver. Tele-Omni is a Swiss Army Knife. Because the Director understands the intent behind your request, Tele-Omni can handle a huge variety of jobs without needing to be retrained for each one:

• Text-to-Video: You say, "A dragon flying over a castle," and it makes it.
• Image-to-Video: You upload a photo of a sleeping cat, and it makes the cat wake up and stretch.
• First-and-Last-Frame: You give it a photo of a person sitting down and a photo of them standing up. The Director figures out the "middle" of the story and fills in the movement smoothly.
• Video Editing: You can say, "Remove that ugly trash can from the background," or "Change the summer clothes to winter coats," and it edits the video while keeping the rest of the scene perfect.
• In-Context Generation: You can show a reference video of a dance and say, "Do this dance, but with a robot," and it copies the motion but changes the character.

3. The Secret Sauce: "Task-Aware" Training

You might wonder, "How does the AI know the difference between 'making a new video' and 'editing an old one'?"

Usually, if you mix different types of training data (like mixing cake batter with car engine oil), the AI gets confused. Tele-Omni uses a special Data Pipeline (a recipe for training) that organizes everything neatly.

Think of it like a universal remote control. Instead of having a separate remote for the TV, the stereo, and the AC, Tele-Omni teaches the AI to recognize the buttons you press.

• If you press "Make New," it knows to create from scratch.
• If you press "Edit," it knows to keep the background and change the subject.
• If you press "Reference," it knows to copy the style or motion from a provided clip.

The system is trained on a massive, organized library of examples so it learns to distinguish these tasks instantly, just by looking at your instructions.

4. Why This Matters

Before Tele-Omni, if you wanted to make a complex video, you had to jump between different tools, losing quality and consistency along the way. It was like trying to build a house by switching between a hammer, a saw, and a trowel every five minutes.

Tele-Omni lets you stay in one room. You give it a mix of instructions—text, photos, and video clips—and it acts as a unified production team. It understands that "make the sky blue" applies to the whole video, while "make the car red" only applies to the car, all while keeping the movement smooth and the lighting realistic.

In short: Tele-Omni is the first AI video tool that acts like a true creative partner, capable of understanding your messy, multi-part ideas and turning them into a single, high-quality movie without needing a different tool for every step.

Technical Summary

1. Problem Statement

Despite significant advances in diffusion-based video generation, current state-of-the-art models face three critical limitations:

• Task Specificity: Most models are designed for single tasks (e.g., Text-to-Video) and rely almost exclusively on textual prompts. They struggle to handle multimodal inputs (images, reference videos) or complex, context-aware scenarios.
• Lack of Unification: Video editing methods typically rely on heavily engineered, task-specific pipelines (e.g., separate models for object removal vs. style transfer). This hinders scalability and the ability to compose multiple operations within a single instruction.
• Data Heterogeneity: Training a single model on diverse video tasks (generation vs. editing) is challenging because the conditioning inputs and target outputs vary significantly. Naively mixing these datasets often leads to unstable training and degraded performance.

2. Methodology

Tele-Omni proposes a unified framework that integrates Multimodal Large Language Models (MLLMs) for instruction parsing with Diffusion Transformers (DiT) for high-fidelity video synthesis.

A. Model Architecture

The framework consists of two primary modules connected via a lightweight adaptor:

1. MLLM (Understanding Module): A pretrained MLLM acts as the "brain." It ingests heterogeneous inputs (text, images, reference videos) and parses them into structured semantic representations. It infers the user's intent (generation vs. editing, specific constraints) without requiring explicit task labels.
2. DiT (Generation Module): A Diffusion Transformer serves as the backbone for video synthesis. It receives:
   • Semantic Guidance: Features from the MLLM projected via a trainable adaptor.
   • Visual Conditions: Latent representations of reference images, videos, or specific frames (first/last) encoded by a pretrained VAE.
3. Decoupling Strategy: By separating instruction parsing (MLLM) from video synthesis (DiT), the system avoids information bottlenecks associated with compressing long video sequences into semantic tokens, allowing for finer-grained control.

B. Unified Task Formulation

Tele-Omni unifies five distinct video-centric tasks under a single instruction-driven paradigm:

• Text-to-Video (T2V): Standard generation from text.
• Image-to-Video (I2V): Generation conditioned on a reference image.
• First-Last-Frame Generation: Synthesizing intermediate frames between two boundary frames.
• In-Context Generation: Using multiple visual references (images/videos) to guide generation.
• In-Context Editing: Modifying specific regions or attributes of a video based on text and reference inputs.

C. Positional Encoding & Conditioning

To handle variable numbers of visual conditions, the model employs a specialized 3D Rotary Position Embedding (RoPE) strategy:

• Reference Images: Treated as temporal conditions (shifted by time).
• Conditional Videos: Treated as spatial in-context conditions (shifted by width).
• This allows the model to concatenate diverse visual latents along the temporal dimension while maintaining spatial and temporal coherence.

D. Training Strategy

The training follows a two-stage process to ensure stability and performance:

1. Stage 1 (Adapter Alignment): The MLLM and DiT are frozen; only the adaptor is trained on T2V and I2V tasks. This aligns the MLLM's semantic space with the DiT's generation space.
2. Stage 2 (Joint Fine-tuning): The adaptor and DiT are jointly optimized on a broader dataset including editing, in-context generation, and frame-conditioned tasks. The MLLM remains frozen.

3. Data System & Construction

A key contribution is the Task-Aware Data Processing Pipeline, which unifies heterogeneous data into a structured instruction format:

• Instruction-Based Editing: The authors constructed a pipeline using GPT-4o, FLUX, and Wan2.1 to generate paired data for:
  • Global Editing: Style transfer (e.g., turning a video into an oil painting).
  • Local Editing: Object insertion, removal, and modification (e.g., changing hair color, removing a person).
• Quality Control: A rigorous dual-verification strategy using Qwen3-VL and Gemini3-pro filters out artifacts like identity inconsistency, ghosting, or unintended edits.
• Reference Generation: For image-guided tasks, incomplete object segments are reconstructed using Nano-Banana to create high-quality reference images.

4. Key Contributions

1. Unified Framework: Tele-Omni is the first framework to support a wide spectrum of video generation and editing tasks (T2V, I2V, Editing, In-Context) within a single model using multimodal instructions.
2. Decoupled Architecture: The separation of semantic understanding (MLLM) and visual synthesis (DiT) enables flexible control without sacrificing temporal coherence or visual fidelity.
3. Task-Aware Data Design: The proposed data pipeline successfully bridges the gap between generation and editing tasks, allowing joint training without the instability typically associated with heterogeneous datasets.
4. Advanced Conditioning: The novel RoPE formulation allows the model to handle variable numbers of visual conditions (images, videos, frames) seamlessly.

5. Experimental Results

Tele-Omni demonstrates competitive performance across multiple benchmarks:

• Visual Fidelity & Motion: Generated videos exhibit strong temporal coherence, avoiding static or frozen motions. Particle dynamics (waves, snow) and complex human movements are rendered naturally.
• Editing Capabilities:
  • Object Removal: Produces clean results without ghosting artifacts.
  • Object Insertion: Dynamically inserted entities interact naturally with the scene and maintain spatial stability over time.
  • Style Transfer: Successfully applies complex styles (e.g., Cyberpunk, Anime) while preserving the identity of subjects and scene geometry.
• In-Context Performance: The model successfully inserts objects into videos at specified locations without masks and maintains high identity fidelity in in-context generation tasks.
• First-Last-Frame: The model correctly interprets complex motion evolution (e.g., falling ribbons, lighting changes) rather than simple interpolation.

6. Significance

Tele-Omni represents a significant step toward general-purpose video foundation models. By moving away from task-specific pipelines toward a unified, instruction-driven approach, it addresses the scalability and composability issues plaguing current video editing tools. The framework's ability to handle multimodal inputs (text, images, video) within a single architecture suggests a future where video creation and editing are as intuitive and flexible as natural language interaction, paving the way for next-generation interactive video applications.