💻 computer science

Vorch-Omni: Multi-Task Orchestration of Sight and Sound

Vorch-Omni is a unified, scalable multi-task framework that leverages a single flow-matching diffusion transformer with flexible token-level conditioning and dual visual pathways to seamlessly orchestrate over 10 diverse audio-visual generation, editing, and extension tasks without requiring task-specific architectural changes.

Original authors: Vorch Team, Xiaoyu Chen, Yang Ding, Cong Han, Menglin Han, Yuxin Hong, Jiebo Hou, Zequn Jie, Xiang Li, Jing Liu, Qi Liu, Yulei Lu, Siyuan Luo, Lin Ma, Xin Ma, Yinlong Qian, Peng Shi, Fang Wan, Siqi Wa

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vorch Team, Xiaoyu Chen, Yang Ding, Cong Han, Menglin Han, Yuxin Hong, Jiebo Hou, Zequn Jie, Xiang Li, Jing Liu, Qi Liu, Yulei Lu, Siyuan Luo, Lin Ma, Xin Ma, Yinlong Qian, Peng Shi, Fang Wan, Siqi Wang, Yaohui Wang, Yaole Wang, Yidi Wu, Siqian Yang, Mingyu Yin, Haoran Yu, Gang Yue, Lisai Zhang, Yuting Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to be a master filmmaker. In the past, you'd have to hire a different robot for every single job: one robot that only knows how to draw pictures from words, another that only knows how to make a movie clip longer, a third that can swap a character's face, and a fourth that can add sound effects. It's like having a toolbox where every tool is a separate, heavy machine that you have to carry around and plug in individually. This is the world of "generative AI" today—a field where computers learn to create new images, videos, and sounds instead of just copying old ones.

The big challenge scientists have been facing is that these different "robots" don't talk to each other well. If you want a video where a character sings a song while the background changes, you usually have to stitch together the results from three different models, which often leads to glitches, bad timing, or weird mismatches between what you see and what you hear. The question on everyone's mind is: Can we build one single, super-smart "conductor" that understands all these different jobs at once? Can it decide when to create something new, when to copy something exactly, and when to just change a tiny detail, all while keeping the sound and picture perfectly in sync?

Enter Vorch-Omni, a new project that suggests the answer might be "yes." Think of Vorch-Omni not as a collection of separate robots, but as a single, incredibly versatile orchestra conductor. Instead of hiring a violinist for the music and a drummer for the rhythm, this one conductor can direct the entire symphony. The researchers built a system that treats video and audio as a single, unified language. Whether you want to generate a brand-new scene from a text description, extend a video that just ended, or swap a character's face while keeping their dance moves exactly the same, Vorch-Omni uses the same core brain to do it all.

The secret sauce is how the system "thinks" about the inputs. Imagine you are giving instructions to a chef. Sometimes you say, "Make me a burger from scratch" (this is generating something new). Sometimes you say, "Here is a burger, just add cheese" (this is editing). Sometimes you say, "Here is a photo of a burger, make it look like a painting" (this is referencing). In the past, computers got confused by these different requests. Vorch-Omni solves this by giving every piece of information a specific "name tag" and a "seat number." It knows exactly which part of the input is the "target" (what needs to be created), which part is the "source" (what needs to be kept), and which part is just a "reference" (a style guide). This allows the model to handle over 10 different types of tasks—like turning text into video, cloning voices, or editing specific parts of a movie—without needing to change its internal wiring for each job.

The paper suggests that this approach works remarkably well, especially when it comes to keeping sound and video in perfect sync. When the researchers tested their model against other top-tier systems, they found that while the overall quality was often similar, Vorch-Omni was significantly better at making sure the audio matched the video (like lips moving in time with speech) and that it could stick to the specific details of a reference image or sound clip. It didn't just "guess" the connection; it understood the relationship between the sight and the sound.

However, the authors are careful to note that this isn't a magic wand that solves every problem in filmmaking yet. While the model is great at short clips and specific edits, it still struggles with very long, complex stories that need to stay consistent for hours. It's also not perfect at generating speech in every language or handling extremely fine-grained edits. But, the paper suggests, this "unified conductor" approach is a major step forward. By proving that one model can juggle so many different roles without getting confused, Vorch-Omni opens the door to a future where creating high-quality, synchronized audio-visual content is as simple as giving a single, clear instruction.

Technical Summary: Vorch-Omni: Multi-Task Orchestration of Sight and Sound

Problem Statement

Recent advances in generative video modeling have expanded capabilities from text-conditioned generation to reference-based synthesis, temporal extension, and instruction-guided editing. However, these capabilities remain fragmented across task-specific models and pipelines. A unified model faces a fundamental ambiguity: when target videos, source clips, and multiple references coexist in the input, the model must infer which content to generate, preserve, or use as guidance. This challenge is compounded when extending to joint audio-visual generation, where audio and video can act as targets, temporally contiguous contexts, or independent references.

Existing unified models often rely on architectural fusion alone, which fails to explicitly specify the roles of varying inputs (e.g., distinguishing a source to be edited from a reference to be imitated). Furthermore, sequential pipelines that synthesize modalities separately accumulate errors and require post-hoc synchronization. The core research question is whether a reproducible and extensible task interface can support diverse visual tasks and arbitrary combinations of audio and video conditions and outputs within a single framework.

Methodology

Core Formulation: Arbitrary-Condition-to-Arbitrary-Output

Vorch-Omni is built upon a single flow-matching diffusion transformer. Its central abstraction is a formulation where any task is defined by four collections: video generation targets ( $X_v$ ), video conditions ( $C_v$ ), audio generation targets ( $X_a$ ), and audio conditions ( $C_a$ ).

Token-Level Conditioning Masks: Explicitly distinguish between clean conditioning tokens (observed signals) and noisy generation targets.
Task Identifiers: Per-token embeddings ( $r_i$ ) explicitly distinguish roles such as targets, sources, subject references, frame references, audio references, and editing conditions. This allows the same latent content to serve different purposes (e.g., a clip as a prefix to extend vs. a source to edit) without architectural changes.
Position Types: A complementary notion of temporal composition ( $p$ ) separates temporally contiguous context (merged with the target timeline for extension) from independent references (separate positional grids). This prevents references from being misinterpreted as adjacent temporal segments.

Complementary Multimodal Conditioning

To handle both high-level semantic intent and fine-grained visual structure, the framework employs two parallel visual conditioning paths:

Semantic Path: A Vision-Language Model (VLM) jointly interprets sampled condition frames and text instructions, providing high-level task semantics.
Latent Path: A video Variational Autoencoder (VAE) encodes complete visual conditions into clean latent tokens, preserving fine-grained appearance, layout, identity, and motion details.
Audio conditions utilize the audio latent path and interact natively with visual tokens within the backbone.

Architecture

The model utilizes an asymmetric dual-stream backbone based on LTX-2.3, consisting of 48 transformer blocks.

Video Stream: 32 attention heads with head dimension 128.
Audio Stream: 32 attention heads with head dimension 64.
Cross-Modal Interaction: Gated cross-attention operations occur after modality-specific text attention and before feed-forward networks in every block. This enables bidirectional information exchange (audio-to-video and video-to-audio) while maintaining separate processing streams. If one modality is absent, the model naturally reduces to the available stream.

Data Infrastructure

A distributed, versioned data pipeline curates diverse, temporally aligned audio-visual clips. Key steps include:

Shot-Aware Segmentation: Dividing long-form videos into coherent clips using TransNetV2 and PySceneDetect.
Quality Filtering & Deduplication: Removing overlays, watermarks, and near-duplicates using SigLIP embeddings.
Structured Annotation: Generating dense video captions (dynamic/static schema) and audio metadata.
Task Packing: Instantiating curated clips into diverse task configurations (generation, extension, editing, etc.) with appropriate conditioning masks, task IDs, and position types.
Weighted Sampling: Balancing heterogeneous task distributions to prevent base generation data from overwhelming specialized tasks.

Key Contributions

Unified Framework: Introduction of Vorch-Omni, a single flow-matching diffusion transformer supporting more than 10 tasks (including text-to-video, reference-conditioned synthesis, temporal extension, and editing) without task-specific architectural modifications.
Explicit Role Representation: A formulation using token-level conditioning masks, task identifiers, and position types to explicitly define what to generate, preserve, reference, and how conditions relate temporally.
Complementary Conditioning: The integration of VLM-based semantic reasoning with VAE-based structural guidance to support both instruction understanding and fine-grained fidelity.
Reproducible Data Pipeline: A scalable pipeline that curates aligned audio-visual data, produces structured annotations, and balances task mixtures for training a unified model.

Results

The model was evaluated against Wan 2.7 on a human-evaluation benchmark comprising 537 paired sample groups across text-to-audio-video generation and reference-conditioned manipulation.

Overall Performance: 81.4% of judgments were ties ("Same"), with a net win rate of -1.9% overall. This indicates that for general visual quality, the unified model performs comparably to specialized baselines.
Specific Strengths: The model achieved positive net win rates in dimensions specific to unified multimodal generation:
- Audio-Visual Synchronization: +5.9% overall; +24.3% in audio-driven tasks (specifically +45.6% on lip-audio sync).
- Reference Alignment: +6.6% overall.
- Audio Prompt Following: +7.1%.
- Audio Quality: +3.3%.
Task-Level Analysis: The model showed advantages in multimodal and reference-driven tasks (e.g., audio-driven generation, animate-and-replace, video transform) while maintaining parity on core visual quality metrics.

Significance and Claims

The paper claims that Vorch-Omni demonstrates that explicit condition roles provide a practical path toward scaling a single model across heterogeneous audio-visual tasks. By decoupling task specification from backbone computation, the framework allows new input-output combinations to be expressed primarily through configuration changes rather than architectural modifications.

The authors emphasize that the primary benefit of this unification is not a uniform improvement in unconditional visual preference, but rather enhanced coordination between modalities and better preservation of information from references. The framework successfully reduces the ambiguity of multi-condition inputs and supports bidirectional generative dependencies (e.g., speech determining lip motion, visual events determining sound effects) within a single training objective.

The paper acknowledges remaining challenges, including long-horizon consistency, fine-grained edit locality, and robust speech generation across languages, identifying these as directions for future work.