💻 computer science

SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning

This paper introduces SyncDPO, an efficient post-training framework that enhances temporal synchronization in video-audio joint generation by leveraging Direct Preference Optimization with on-the-fly rule-based negative construction and curriculum learning to overcome the limitations of traditional supervised fine-tuning.

Original authors: Xin Cheng, Xihua Wang, Ying Ba, Yuyue Wang, Kaisi Guan, Yinbo Wang, Wenpu Li, Ruihua Song

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xin Cheng, Xihua Wang, Ying Ba, Yuyue Wang, Kaisi Guan, Yinbo Wang, Wenpu Li, Ruihua Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to create a movie where the sound and the picture match perfectly. If the robot sees a person clap their hands, the "clap" sound should happen at the exact same millisecond the hands touch. If the sound comes a split second too late, the movie feels fake and jarring.

This paper introduces a new training method called SyncDPO to fix this timing problem in AI video generators. Here is how it works, broken down into simple concepts:

The Problem: The "Laggy" Robot

Currently, AI models are great at making videos that look real and sound generally correct. However, they often struggle with timing.

The Analogy: Think of a bad karaoke machine where the singer's voice is slightly out of sync with the music. You can hear the words, and you can hear the music, but they don't line up.
The Old Way (SFT): Previously, to fix this, engineers used a method called "Supervised Fine-Tuning." Imagine this as a teacher showing the robot the correct movie and saying, "Try to make your video look exactly like this." The robot tries to copy the picture and sound, but because the "penalty" for being slightly off-time is too small, the robot keeps making tiny timing mistakes. It's like a student who knows the answer is "5" but keeps writing "5.001" because the teacher didn't strictly correct the decimal.

The Solution: The "Bad Example" Lesson

The authors realized that to teach the robot perfect timing, you don't just need to show it the right answer; you need to show it the wrong answers and say, "This is bad, don't do this."

They used a technique called Direct Preference Optimization (DPO).

The Analogy: Instead of just showing the robot a perfect movie, you show it two clips:
1. The Winner: A clip where the gunshot sound happens exactly when the gun fires.
2. The Loser: A clip where the gunshot sound happens two seconds after the gun fires.
  The robot learns, "Ah! I must avoid the second one." This sharpens its sense of timing.

The Innovation: Making "Bad Examples" Instantly

Usually, creating these "Loser" clips is expensive and slow. You would have to generate many videos, wait for humans to rank them, or use other complex AI to find the bad ones. It's like hiring a film critic to watch 100 bad movies just to find one example of bad timing.

SyncDPO changes the game by being a "Rule-Based Chef."
Instead of cooking a whole new meal to find a bad one, the chef takes the perfect meal and instantly messes it up using simple rules:

Speed it up: Make the video fast but keep the audio slow (or vice versa).
Swap it: Take the audio from a different video and paste it onto the current one.
Delay it: Push the sound forward or backward by a few seconds.
Hide it: Mute part of the audio or freeze part of the video.

These "messed up" versions are created instantly while the data is being loaded. No extra humans, no extra waiting, and no extra cost.

The Strategy: The "Video Game" Approach (Curriculum Learning)

The authors also noticed that if you start by showing the robot extremely obvious bad examples (like a 10-second delay), it learns too easily. But if you start with tiny errors (like a 0.1-second delay), the robot gets confused and can't learn.

So, they used a Curriculum Learning strategy, similar to a video game:

Level 1 (Easy): Start with obvious mistakes (like swapping the audio with a completely different sound). The robot learns the basics of "sound must match picture."
Level 2 (Hard): Gradually switch to subtle mistakes (like speeding up the video slightly). Now the robot has to learn fine-grained, precise timing.

By slowly increasing the difficulty, the robot becomes a master of synchronization.

The Results

The paper tested this on four different types of videos:

People talking (lip-sync).
Gunshots and explosions.
Animals making noise.
General environmental sounds.

The Outcome:
SyncDPO was significantly better at aligning sound and picture than previous methods.

It made the "clap" sound happen exactly when the hands touched.
It made the gunshot sound happen exactly when the gun fired.
It worked well even on types of videos it hadn't seen before (generalization).

Crucially, the authors found that this method didn't ruin the quality of the video or audio; it just made the timing perfect. They even had humans watch the results, and the humans consistently preferred the SyncDPO videos because they felt more "real" and immersive.

Summary

In short, SyncDPO is a smarter way to train AI to sync sound and video. Instead of just copying good examples, it teaches the AI by instantly creating "bad" examples using simple rules, starting with easy mistakes and working its way up to tiny, precise ones. The result is AI-generated videos where the sound and action click perfectly together.

Technical Summary: SyncDPO

Problem Statement

Recent advancements in video-audio (V-A) joint generation have achieved high semantic correspondence, yet precise temporal synchronization remains a significant challenge. While models can generate high-quality visual and audio content individually, they often fail to align audio events with their visual triggers at a fine-grained level (e.g., gunshot sounds drifting from animations, or speech mismatching lip movements).

Current post-training methods for V-A models rely heavily on Supervised Fine-Tuning (SFT) using Mean Squared Error (MSE) loss. The paper argues that MSE provides insufficient penalties for subtle temporal misalignments, as it focuses on overall feature reconstruction rather than distinguishing precise temporal offsets. Furthermore, existing Direct Preference Optimization (DPO) approaches, which could offer better discrimination, are often inefficient. They typically depend on costly sampling-and-ranking procedures or human annotation to construct preference pairs, resulting in substantial computational overhead.

Methodology

The authors propose SyncDPO, a post-training framework that leverages DPO to enhance the temporal sensitivity of V-A joint generation models. The methodology consists of three core components:

1. Flow-Matching based DPO

The framework operates on a Flow-Matching generative model. Instead of learning solely from positive examples (as in SFT), SyncDPO explicitly contrasts synchronized pairs against temporally distorted ones. The objective function ( $L_{Flow-DPO}$ ) pulls the predicted velocity field closer to the "winner" (synchronized) sample while pushing it away from the "loser" (misaligned) sample, using a frozen reference model for regularization. This discriminative supervision compels the model to recognize and penalize synchronization errors.

2. On-the-Fly Rule-Based Negative Construction

To address the efficiency bottleneck of conventional DPO, SyncDPO introduces a lightweight, rule-based strategy to construct negative preference pairs during data loading, eliminating the need for additional sampling, external reward models, or human annotation. The method perturbs the ground-truth aligned video-audio pairs using five operations:

Scaling: Rescaling the temporal dimension of video or audio to create speed inconsistencies.
Replacing: Substituting video or audio with another sample from the dataset to introduce semantic and temporal discord.
Shifting: Applying a temporal offset to simulate advance/delay errors.
Masking: Masking parts of the video or audio to create silent/frozen content.
Synthesizing: Replacing modality with a sample from the untuned reference policy.

Experiments indicate that Replacing and Scaling are the most effective strategies for providing discriminative supervision without destroying semantic structure.

3. Curriculum Learning Strategy

Recognizing that DPO training is sensitive to the difficulty of negative examples, the authors implement a curriculum learning approach. The training schedule progressively shifts the sampling probability from Replacing (easier, coarse misalignment with semantic mismatch) to Scaling (harder, fine-grained temporal distortion with semantic consistency). This allows the model to first establish basic temporal awareness before refining its sensitivity to subtle inconsistencies.

Key Contributions

SyncDPO Framework: A DPO-based post-training method that significantly improves temporal alignment in V-A generation by leveraging negative supervision.
Efficient Negative Construction: A novel, cost-effective method for generating preference pairs on-the-fly via rule-based perturbations, avoiding the computational overhead of traditional sampling-and-ranking pipelines.
Curriculum Learning: A difficulty-scheduled training strategy that transitions from coarse to fine-grained misalignment supervision, optimizing the learning trajectory for temporal synchronization.

Experimental Results

The authors evaluated SyncDPO across four diverse benchmarks: LRS2 (lip-speech), AVSync15 and GreatestHits (event-triggered ambient sounds), and VABench (out-of-domain generalization).

Temporal Alignment: SyncDPO consistently outperformed both SFT and vanilla DPO baselines. On LRS2, it achieved the best Lip-Speech synchronization metrics (LSE-D and LSE-C). On ambient sound benchmarks, it significantly reduced the DeSync metric (temporal offset).
Generalization: The method demonstrated superior out-of-distribution (OOD) generalization on the VABench and Koala datasets, capturing intrinsic motion-sound dynamics better than SFT, which tended to overfit to training distribution correlations.
Quality Preservation: Improvements in synchronization did not come at the cost of generation fidelity. Metrics for speech quality (WER, MCD), audio quality (FAD), and video quality (FVD) remained competitive or improved compared to baselines.
Human Evaluation: Blind pairwise human preferences confirmed that SyncDPO outputs were consistently preferred for temporal alignment over SFT, particularly in scenarios involving event triggers and speech.

Significance and Claims

The paper claims that SyncDPO addresses a critical gap in V-A generation: the lack of precise temporal synchronization. By demonstrating that explicit negative supervision via DPO is more effective than MSE-based reconstruction, and by solving the efficiency problem through rule-based negative construction, the authors present a scalable solution for high-fidelity joint generation.

The authors emphasize that their approach captures intrinsic motion-sound dynamics, allowing for robust performance even in out-of-distribution scenarios where traditional SFT fails. The work suggests that future improvements in generative AI for video and audio will require moving beyond simple reconstruction losses toward preference-based optimization that explicitly models temporal causality and alignment.