InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervision
InternVideo-Next introduces a novel two-stage pretraining framework that disentangles the encoder-decoder design into an Encoder-Predictor-Decoder structure with a conditional diffusion decoder to overcome the limitations of pixel-level reconstruction and shortcut learning, thereby achieving state-of-the-art performance in general video representation learning without relying on noisy video-text supervision.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you want to teach a robot how to understand the world just by watching videos, without anyone ever telling it what's happening in words. This is the challenge the paper InternVideo-Next tackles.
Here is the story of how they did it, using some simple analogies.
The Problem: Two Bad Ways to Learn
Previously, researchers tried two main ways to teach robots to watch videos, but both had flaws:
The "Subtitle Reader" (Text-Supervised):
Imagine teaching a student by showing them a movie with subtitles. They learn great vocabulary and can tell you who is doing what (e.g., "A man is kicking a ball"). But, because they are just reading subtitles, they miss the subtle physics. They don't really understand how the ball spins, the 3D depth of the room, or the gravity pulling it down. They are good at the "story" but bad at the "reality."The "Pixel Painter" (Masked Video Modeling):
Imagine a game where you cover up part of a video and ask the student to draw the missing part. This forces them to look at the raw pixels. They get really good at drawing the exact colors and shapes. But, they often get "lazy." Instead of understanding the physics of the scene, they just guess, "Oh, the sky is usually blue, so I'll draw blue." They miss the deeper logic of how objects move and interact.
The Gap: The "Subtitle Reader" knows the words but not the physics. The "Pixel Painter" knows the colors but not the meaning. We needed a student who understands both.
The Solution: The "Architect, The Predictor, and The Builder"
The authors realized the old way of building these AI brains was too messy. They broke the process down into three distinct roles, like a construction crew:
- The Architect (Encoder): Looks at the video and figures out the big picture.
- The Predictor (The "World Model"): This is the star. It tries to guess what the missing parts of the video should look like based on the laws of physics and logic.
- The Builder (Decoder): Takes the Predictor's guess and tries to turn it back into a real video frame.
The magic of InternVideo-Next is how they trained this team in two specific stages.
Stage 1: Learning the "Vocabulary" of Reality
In the first stage, they wanted the Predictor to understand both the big ideas (semantics) and the tiny details (pixels).
- The Old Way: They used a simple "Builder" (a linear decoder) that forced the Predictor to be very simple. It was like asking a genius architect to draw a blueprint using only a stick figure. It couldn't capture complex details.
- The New Way (Diffusion Decoder): They replaced the simple Builder with a Diffusion Decoder. Think of this like a master artist who can take a rough sketch and slowly refine it into a photorealistic painting, adding noise and then removing it to find the perfect image.
- The Secret Sauce: They also gave the Predictor a "cheat sheet" from a super-smart image model (SigLIP). This cheat sheet told the Predictor, "Hey, that's a dog, not a cat," ensuring the AI learned the meaning of the objects while still learning how to draw the fur and the tail.
Result: The Predictor now has a "latent space" (a mental map) that is full of rich meaning and sharp details.
Stage 2: Learning the "Physics" of the World
Now that the Predictor has a great mental map, Stage 2 is about learning how the world moves.
- The Game: They hide big chunks of the video (like a whole second of action) and ask the Predictor to guess what happened in that gap.
- The Trick: They freeze the "Teacher" (the model from Stage 1) so it doesn't change. The "Student" has to predict the Teacher's answer.
- Why it works: Because the Teacher's answers are so detailed and meaningful (thanks to Stage 1), the Student can't cheat. It can't just guess "blue sky" or "static background." It has to actually understand causality (if I push this cup, it falls) and 3D geometry (that car is moving away, so it gets smaller).
The Results: A Super-Student
When they tested this new model, it was a game-changer:
- It knows the story: It's great at recognizing actions (like "a person is dancing").
- It knows the physics: It's surprisingly good at guessing depth (how far away things are) and tracking objects moving in 3D space, even though it was never explicitly taught to do this.
- It's efficient: It learned all this using only public, unlabeled videos. It didn't need expensive human-written captions.
The Big Picture
Think of InternVideo-Next as the first AI that didn't just memorize a script or just copy-paste pixels. It actually watched the world, figured out the rules of physics and cause-and-effect, and built a "World Model" inside its brain.
This is a huge step toward creating AI that can truly understand our physical world, which is essential for things like self-driving cars, robots that can help us in our homes, and future AI that can have deep conversations about what it sees.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.