LongCat-Video-Avatar 1.5 Technical Report
LongCat-Video-Avatar 1.5 is an upgraded open-source framework that prioritizes production readiness and systematic engineering to deliver commercial-grade, stable, and identity-consistent long-video generation with accelerated inference, outperforming leading closed-source systems across diverse scenarios.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you want to create a digital character that can talk, sing, and act just like a real human, but you only have a photo of them and an audio recording of their voice. For a long time, making these characters look "real" for more than a few seconds has been like trying to balance a house of cards in a windstorm: they might look good for a moment, but then their faces glitch, their lips don't match the words, or their bodies start moving strangely.
The LongCat-Video-Avatar 1.5 paper from the Meituan LongCat Team is a technical report about a new, open-source "recipe" for building these digital characters. Instead of inventing a brand-new type of magic, they focused on perfecting the engineering to make the result stable enough for real-world use (like movies, news, or customer service).
Here is how they did it, explained with simple analogies:
1. The "Ear" Upgrade: Whisper Large
In the old version, the model used a standard "ear" (an audio encoder called Wav2Vec2) to listen to the voice. It was okay, but sometimes it missed the subtle nuances of speech.
- The Upgrade: They swapped this for Whisper Large, a much more powerful audio system.
- The Analogy: Think of the old ear as someone listening to a song in a noisy room with earplugs. The new Whisper Large is like putting on high-end noise-canceling headphones in a quiet studio. It hears every tiny detail of the voice, which allows the character's lips to move with perfect precision, matching the sound exactly, even in long videos.
2. The "Gym" for Data: Curating the Training Set
You can't teach a digital actor just by showing them random videos. If you show them a video with a blurry face, a person holding a sign, or two people talking at once, the model gets confused.
- The Fix: They built a rigorous "data gym." They didn't just dump thousands of videos into the system. They sorted them like a librarian organizing a library:
- Silent Data: They taught the model what to do when no one is speaking (e.g., blinking, breathing, looking around) so the character doesn't freeze or look weird when the audio stops.
- Emotion Data: They specifically fed the model videos of people showing different feelings (laughing, crying, reacting) so the character doesn't just look like a robot reading a script.
- Multi-Person Data: They taught the model how to handle a scene with two people talking. They used a special trick (giving the background characters a "silent" audio track) so that only the person who is supposed to talk actually moves their mouth, while the others stay still.
3. The "Coach" (RLHF): Learning from Human Feedback
Even with good data, the model might still make small mistakes, like a hand looking slightly twisted or a face moving unnaturally.
- The Method: They used a technique called Group-Relative Policy Optimization (GRPO).
- The Analogy: Imagine a dance instructor watching a student. Instead of just saying "Good job" or "Bad job," the instructor compares the student's dance to a group of other dancers and says, "Your arm movement is 10% better than the average, but your footwork is 5% worse." The model learns from these comparisons to fine-tune its movements frame-by-frame, ensuring the hands look real and the body moves naturally.
4. The "Turbo Mode": Making it Fast
Usually, high-quality video generation is slow. It's like baking a cake where you have to check the oven every 5 minutes for an hour.
- The Innovation: They used a technique called Distillation to compress the process.
- The Analogy: They taught the model to bake the cake in 8 steps instead of the usual 50. It's like training a runner to sprint the whole race in 8 giant strides instead of 50 small, slow steps. The result is a video that looks just as good but generates much faster, making it practical for real-time use.
5. The Results: How Does It Compare?
The team tested their new model against the best "closed-box" (secret, paid) systems currently on the market, like HeyGen and Kling Avatar.
- The Verdict: In a blind test with over 500 different scenarios (including talking heads, singing, anime styles, and even animals), LongCat-Video-Avatar 1.5 was often rated as better or equal to these expensive commercial systems.
- Key Wins:
- Lip Sync: The mouth movements matched the audio perfectly.
- Stability: The character didn't flicker or change faces during long videos.
- Identity: The character looked like the same person from start to finish.
- Complexity: It handled tricky situations, like a person holding a guitar or two people talking, without the background characters accidentally moving their lips.
Summary
LongCat-Video-Avatar 1.5 is essentially a "production-ready" toolkit. It takes the messy, experimental nature of AI video generation and polishes it with better ears (Whisper), better training data (Silent/Emotion/Multi-person), a strict coach (RLHF), and a speed boost (Distillation). The goal wasn't just to make a cool demo, but to build a system that works reliably enough to be used in real businesses, and they proved it works as well as (or better than) the top paid tools available today.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.