MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation
MilliVid addresses the challenge of long-range consistency in video generation by employing a hierarchical autoencoder to compress frames into multi-scale tokens and utilizing a coarse-to-fine rollout strategy within a video diffusion model, thereby preserving global geometry and object permanence while reducing computational costs.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to tell a very long story to a friend, but you only have a tiny notepad. If you try to write down every single detail of every scene—every leaf on a tree, every crack in the sidewalk, every texture on a wall—you will run out of space after just a few sentences. By the time you get to the end of the story, you've forgotten the beginning entirely.
This is the exact problem video AI models face today. They want to generate long, coherent videos, but their "notepads" (computer memory) are too small to hold all the details of a long sequence. As a result, they start to hallucinate, forgetting where objects went or changing the layout of the room as the video progresses.
The paper MILLIVID proposes a clever solution: Stop trying to remember everything at once. Instead, remember the big picture first, and fill in the tiny details later.
Here is how they did it, using some everyday analogies:
1. The "Russian Nesting Doll" Memory System
Most video AI models try to store every frame at full resolution. MILLIVID changes the game by using a hierarchical tokenizer. Think of this like a set of Russian nesting dolls or a map with different zoom levels:
- The Coarse Level (The Big Picture): This is the smallest doll or the map at the lowest zoom. It doesn't show the texture of the grass or the color of the bricks. It only remembers the shape of the room, where the walls are, and where the main objects are. Because it's so simple, the AI can remember hundreds of these "big picture" frames at once.
- The Fine Level (The Details): This is the largest doll or the map at maximum zoom. It adds the grass texture, the brick patterns, and the lighting. But because it's so detailed, the AI can only remember a few of these frames before its memory runs out.
The Analogy: Imagine you are describing a city to a friend.
- Old Way: You try to describe every single window on every building for the whole trip. You get tired and forget the city layout after 10 minutes.
- MILLIVID Way: First, you describe the layout of the city (where the park is, where the river flows) for the whole hour. Then, as you get closer to a specific building, you add the details (the color of the door, the flowers in the window). You keep the "big picture" in your head the whole time so you never get lost.
2. The "Coarse-to-Fine" Rollout
The paper introduces a new way of generating the video called coarse-to-fine rollout.
Instead of generating the video frame-by-frame with full detail (which causes the AI to forget the past), the model works in stages:
- Stage 1: It generates a long sequence of "blurry" or low-detail frames. Because they are simple, the AI can keep track of 100+ frames at once. This ensures the story stays consistent (the car stays on the road, the building doesn't disappear).
- Stage 2: It goes back and "sharpens" the recent frames, adding the high-definition details.
- The Magic Trick: Crucially, when it adds the details to the recent frames, it looks at the low-detail versions of the distant frames to make sure the story still makes sense.
The Analogy: Think of it like sketching a painting.
- First, you draw a rough stick-figure outline of the whole scene to get the poses and positions right. You can draw the whole scene this way without messing up.
- Then, you go back and add the muscles, the clothes, and the facial expressions to the people in the foreground.
- You don't try to paint the muscles of the person in the background before you know where they are standing, or you might paint them in the wrong spot.
3. The "Minecraft" Test
To prove this works, the researchers didn't just use pretty movies. They used Minecraft.
- Why? Minecraft is a 3D world where you walk around. If the AI forgets the layout, you might walk through a wall or see a tree disappear and reappear in a different spot.
- The Result: They tested their method against the current best models (like FramePack). The old models would generate a video where the player walked for a while, and then the world would "glitch"—buildings would shift, or the path would loop back on itself incorrectly.
- MILLIVID's Performance: Because it kept the "coarse" map of the world in its memory the whole time, it could generate long videos where the player walked hundreds of steps, turned around, and saw the exact same buildings they passed earlier, perfectly consistent.
The Bottom Line
The paper claims that by accepting that we can't remember every tiny detail of the distant past, we can remember the structure of the past much better.
- Old Approach: "I will remember every detail of the last 5 seconds perfectly, but I will forget everything that happened 10 seconds ago."
- MILLIVID Approach: "I will remember the general shape and location of everything that happened 10 seconds ago, and I will only remember the tiny details of the last 5 seconds."
This trade-off allows the AI to generate videos that stay coherent for much longer, keeping the "story" of the video consistent without needing supercomputers that cost millions of dollars. The authors tested this specifically on Minecraft gameplay and found it produced significantly more consistent results than existing methods.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.