💻 computer science

Tora3: Trajectory-Guided Audio-Video Generation with Physical Coherence

Tora3 is a trajectory-guided audio-video generation framework that enhances physical coherence and motion-sound synchronization by using object trajectories as a shared kinematic prior to jointly control visual motion and acoustic events, supported by a newly curated large-scale dataset called PAV.

Original authors: Junchao Liao, Zhenghao Zhang, Xiangyu Meng, Litao Li, Ziying Zhang, Siyu Zhu, Long Qin, Weizhi Wang

Published 2026-08-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Junchao Liao, Zhenghao Zhang, Xiangyu Meng, Litao Li, Ziying Zhang, Siyu Zhu, Long Qin, Weizhi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine watching a video of a hammer striking a nail. In a perfect world, the sound of the impact happens at the exact moment the metal meets the wood, and the sharpness of the noise matches how hard the blow was. For years, computers have struggled to create videos where the movement and the sound feel like they belong to the same physical reality. While artificial intelligence has become very good at generating beautiful moving pictures or creating realistic soundscapes from text, combining the two has often resulted in a disconnect. The objects might move in ways that defy gravity, or the sounds might arrive a fraction of a second too late, or they might be too quiet for a loud crash. The core problem has been that the computer generating the video and the computer generating the audio were working in isolation, each guessing what the other should do without a shared understanding of how the object is actually moving through space.

A team of researchers at Alibaba Group and Fudan University has addressed this gap with a new system called Tora3. Their approach is built on a simple but powerful idea: use the path an object takes as a shared blueprint for both the video and the audio. Instead of letting the computer guess how a car should drive or a ball should bounce, the system first defines the exact trajectory, or the line the object follows, and then uses that line to dictate both the visual motion and the accompanying sound. This method treats the path not just as a guide for where the object goes, but as a source of physical information that tells the system when to make a sound and how loud it should be. By anchoring both the sight and the sound to the same mathematical description of movement, the researchers have created videos where the physics feel real and the audio is perfectly synchronized with the action.

The researchers found that previous methods often failed because they tried to generate the video and audio separately and then hoped they would match up, or they used the movement path only as a visual guide for the video while ignoring its potential to guide the sound. Tora3 changes this by using the trajectory as a "kinematic prior," which is a technical way of saying it uses the path as a fundamental rule that both the video and audio generators must follow. When the system sees an object moving along a specific line, it calculates the speed and acceleration at every point. If an object is speeding up, the system knows the sound should get louder or change in tone. If an object hits a surface, the system knows exactly when to trigger a sharp impact sound. This shared reliance on the path ensures that the visual and auditory elements are not just happening at the same time, but are reacting to the same physical forces.

To make this work, the team developed a specific way to feed this path information into the computer's brain. For the video part, instead of building a complex new machine to understand movement, they simply took the visual information from the first frame of the video and moved it along the path of the object as the video progresses. This is a direct and efficient way to tell the computer, "This object is here, and it is moving to there," without needing extra layers of processing that might confuse the system. For the audio, the system looks at the path and calculates how fast the object is moving and how quickly it is speeding up or slowing down. These calculations are then used to shape the sound, ensuring that a fast-moving object produces a different sound than a slow one, and that a sudden stop creates a distinct noise.

The team also created a new method for blending these instructions into the final video. They realized that some parts of the video need to stick strictly to the path to ensure the object moves correctly, while other parts, like the background, should remain flexible to look natural. Their solution was to use two different rules for the video generation process: one rule that tightly binds the object to its path, and another rule that allows the rest of the scene to flow naturally. This hybrid approach prevents the video from looking stiff or unnatural while still ensuring the main action follows the intended trajectory.

To train this system, the researchers gathered a massive collection of 460,000 video clips that focused on movement. They used advanced tools to automatically find the objects in these videos and map out their paths, creating a dataset rich in motion patterns. This allowed the system to learn from a wide variety of real-world movements, from a car driving down a road to a seal slapping water. When tested against other leading systems, Tora3 produced videos that were not only visually sharper but also had a much stronger connection between the movement and the sound. The system was better at matching the timing of sounds to the moment of contact and at adjusting the volume of the sound to match the intensity of the movement.

The results suggest that giving artificial intelligence a shared physical rulebook, in this case the path of an object, is a powerful way to improve the realism of generated content. The system does not just make things look good; it makes them feel like they exist in a world governed by consistent laws of physics. While the technology is still being refined, the success of this approach highlights a promising direction for the future of digital media, where the line between what we see and what we hear becomes indistinguishable from reality. The researchers plan to continue this work by exploring how other physical factors, such as the material of an object or the way sound travels through a room, can be added to this shared framework to create even more convincing experiences.

Technical Summary: Tora3 – Trajectory-Guided Audio-Video Generation with Physical Coherence

1. Problem Statement

While recent advances in generative modeling have significantly improved the perceptual quality and multimodal coherence of text-to-audio-video (T2AV) systems, a critical gap remains in physical coherence. Existing methods, including strong open-source models, often generate object motions that are visually unstable (e.g., inconsistent trajectories, implausible acceleration) and sounds that are only loosely aligned with salient motion or contact events.

The core limitation identified is the lack of an explicit, shared motion-aware structure that constrains both video and audio modalities simultaneously. Current approaches typically rely on semantic alignment rather than explicit motion-sound modeling, leading to:

Visual Instability: Generated objects may follow inconsistent paths or break temporal continuity.
Audio-Visual Misalignment: Impact sounds may miss contact moments, intensity may not reflect interaction strength, and continuous sounds (e.g., sliding) may not evolve with motion dynamics.

The authors argue that object trajectories offer a natural, lightweight, and interpretable shared representation to bridge this gap. Trajectories compactly encode kinematic cues (event timing, motion intensity, temporal evolution) that can constrain plausible motion in video while dictating the timing and amplitude evolution of sound.

2. Methodology: The Tora3 Framework

Tora3 is a trajectory-guided audio-video generation framework built on a twin Diffusion Transformer (DiT) architecture (following the Ovi baseline). It treats object trajectories not merely as an auxiliary control signal for video, but as a shared kinematic prior that jointly guides visual motion and acoustic events. The framework consists of three key technical components:

A. Trajectory-Aligned Motion Representation (Video Branch)

Instead of using a separate motion encoder to process trajectory data (which can entangle optimization and attenuate sparse signals), Tora3 directly injects motion cues into the video latent space:

Mechanism: The system propagates features from the first frame along the prescribed object trajectories to subsequent frames.
Implementation: For a latent video sequence $x_0$ , the system constructs a trajectory-conditioned latent $x_{traj}$ by replacing the latent features at trajectory-aligned locations in frame $i$ with the corresponding features from the first frame ( $z$ ).
Benefit: This avoids distribution shifts and additional feature transformations, providing a direct conditioning signal that preserves local object cues while enforcing trajectory adherence.

B. Kinematic-Audio Alignment Module (Audio Branch)

To address the insufficiency of semantic conditioning alone, Tora3 introduces explicit kinematic conditioning derived from trajectories:

Second-Order Kinematic States: The system computes position ( $\mathbf{r}$ ), velocity ( $\mathbf{v}$ ), and acceleration ( $\mathbf{a}$ ) using central differences. It also extracts the magnitudes of velocity and acceleration ( $\|\mathbf{v}\|, \|\mathbf{a}\|$ ) to disentangle direction from intensity.
Feature Encoding: These 8D kinematic features undergo signed log-compression and are encoded into latent tokens via an MLP.
Fusion: The kinematic tokens are injected into the audio DiT via a cross-attention mechanism. A learnable gating mechanism ( $\sigma(\gamma)$ ) balances the influence of these kinematic cues against the original semantic text conditions, ensuring the audio remains semantically appropriate while becoming physically aligned with motion intensity and timing.

C. Hybrid Flow Matching

Recognizing that different spatial regions require different handling during denoising, Tora3 employs a hybrid flow matching scheme:

Trajectory Regions ( $\Omega_{traj}$ ): In regions defined by the object trajectories, the flow matching target is anchored to the trajectory-conditioned latent ( $x_{traj}$ ) rather than standard noise. This ensures faithful adherence to the motion prior.
Background Regions: Outside the trajectory, the standard flow matching construction (targeting Gaussian noise) is preserved to maintain flexible scene refinement and local coherence.
Loss Balancing: A soft mask (Gaussian blurred) is used to create a region-balanced loss objective, preventing the sparse trajectory regions from being overwhelmed by the dominant background area during training.

3. Key Contributions

Tora3 Framework: A unified T2AV generation framework that uses object trajectories as a shared kinematic prior to jointly guide visual motion and acoustic events, significantly improving physical coherence.
Novel Architectural Components:
- A trajectory-aligned motion representation that injects motion directly into video latents without an explicit motion encoder.
- A kinematic-audio alignment module driven by second-order kinematic states (position, velocity, acceleration, and magnitudes) to synchronize sound events and intensity with motion dynamics.
- A hybrid flow matching scheme that balances motion fidelity in trajectory regions with local coherence in background regions.
PAV Dataset: The curation of PAV, a large-scale (460k clips) audio-video dataset emphasizing motion-relevant patterns. It includes automatically extracted motion annotations (trajectories via CoTracker3) and paired video/audio descriptions, specifically designed for trajectory- and kinematics-aware generation.

4. Experimental Results

Extensive experiments on the PAV dataset demonstrate that Tora3 outperforms strong open-source baselines (including LTX-2, Ovi, MOVA, and AVControl) across multiple dimensions:

Video Quality: Tora3 achieves the lowest Fréchet Video Distance (FVD: 784.1) and highest Aesthetic Score (AS: 4.61), indicating superior visual realism and stability.
Audio-Video Synchronization: The model achieves the highest FGAS (0.234), indicating better alignment between video frames and synchronized audio features.
Motion-Sound Coherence:
- Event Timing: Tora3 shows the lowest Event Timing Error (ETE: 0.181), meaning impact sounds align more precisely with contact events.
- Intensity Correlation: It achieves the highest Motion-Audio Intensity Correlation (MAIC: 0.63), demonstrating that audio loudness evolves consistently with motion strength (e.g., faster movement generates louder sounds).
Trajectory Faithfulness: Tora3 maintains the lowest Trajectory Error (12.13), confirming that generated objects follow the prescribed paths more accurately than baselines.

Ablation studies confirm that using trajectories as a shared prior for both modalities yields the best results, and that the inclusion of second-order kinematic features (acceleration and magnitudes) is critical for high-fidelity sound generation.

5. Significance and Claims

The paper claims that Tora3 represents a significant step forward in making multimodal generation physically coherent rather than just semantically aligned. By introducing object trajectories as a shared kinematic prior, the framework successfully couples visual motion and acoustic generation under a unified structure.

The authors emphasize that this approach:

Solves the problem of "loose" alignment where sound is detached from physical events.
Provides a lightweight, interpretable mechanism to constrain plausible motion and sound evolution without requiring complex physics simulations.
Demonstrates that explicit kinematic conditioning (position, velocity, acceleration) is essential for generating realistic motion-dependent audio.

The work positions Tora3 as a practical solution for generating high-fidelity audio-video content where motion dynamics and sound are intrinsically linked, setting a new standard for physical coherence in generative AI. Future work is suggested to extend this trajectory-level coherence to richer physical factors like material properties and 3D acoustic propagation.