Evaluating Design Video Generation: Metrics for Compositional Fidelity
This paper introduces a fully automated, four-dimensional evaluation framework to assess compositional fidelity in design video generation, addressing the lack of standardized benchmarks for ensuring structured constraints like layout preservation and precise motion control.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a director asking a very talented, but slightly confused, robot to animate a digital poster. You give the robot a blueprint: "Make the logo slide in from the left, make the text fade in gently, and keep the background still."
The robot tries its best, but sometimes it makes mistakes. Maybe it slides the background instead of the logo, or it makes the text spin wildly when you just wanted it to fade in. Until now, there was no standardized way to grade the robot's homework. Most people just watched the video and said, "Looks okay," or "Looks weird."
This paper introduces a strict, automated grading system for these design animations. Instead of relying on human opinions, the authors built a "robot teacher" that checks the video against the original blueprint with surgical precision.
Here is how their system works, broken down into simple concepts:
1. The Two Test Tracks
The authors created two different types of tests to see how the robots perform:
- The "Solo Act" (Single-Component): Imagine isolating just one element, like a single dancing button on a white background. This is the easy test to see if the robot understands basic moves like "pop" or "slide."
- The "Full Orchestra" (Full-Layout): This is the hard test. It's a whole poster with 10 or 20 different things happening at once. The robot has to make sure the text moves one way, the image moves another, and the background stays perfectly still.
2. The Four Grading Categories
The "robot teacher" checks the video on four specific dimensions, much like a music teacher checking a student's performance:
- Motion Type (The Dance Move): Did the robot do the right dance? If you asked for a "fade" (disappearing like a ghost), did it do that? Or did it accidentally do a "spin" or a "slide"?
- The Catch: The paper notes that some moves look very similar to a camera. For example, a "fade" and a "slide" can look almost the same if the robot is just jittering a little bit. The system groups these similar-looking moves into "observable classes" to be fair.
- Motion Direction (The Compass): If the robot was told to move "Up," did it go Up? Or did it go "Up-Right"? The system checks the angle of the movement to see if it matches the blueprint.
- Timing (The Stopwatch): How long did the move last? If the blueprint said "slide for 0.5 seconds," did it slide for 0.5 seconds, or did it drag on for 3 seconds? The system measures the exact duration of the action.
- Text Recoverability (The Reading Test): If the poster has words, can you still read them? The system takes snapshots of the video and tries to read the text. If the robot distorted the letters so badly they look like gibberish, it gets a low score.
3. The "Robot Teacher" Tools
To grade these videos, the authors built two special tools:
- The "Spotter" (Tracker): For simple videos, it just looks for anything that isn't the background color. For complex videos, it uses a super-trained eye (a modified YOLO detector) that knows exactly where every piece of the puzzle should be, so it can spot if a piece moved when it shouldn't have.
- The "Rulebook" (Classifier): This is a set of strict "If-Then" rules. It doesn't guess; it calculates. For example: "If the object gets bigger quickly but doesn't move far, it's a 'Pop'. If it moves far, it's a 'Pan'."
4. What They Found (The Report Card)
The authors tested two of the world's most advanced video robots: Sora-2 and Veo-3.1. Here is what their "robot teacher" found:
- The "One-Size-Fits-All" Problem: The robots often struggle to follow specific instructions for specific parts. They tend to apply one big motion to the whole screen (like making the whole poster slide) instead of moving just one element.
- The "Ghost" Problem: The robots often fail at "fades." Because the robots can't easily "see" transparency changes (opacity), they often mistake a gentle fade for a jittery slide.
- The "Text" Struggle: While the robots are getting better at moving things around, they still struggle to keep text perfectly readable, especially in complex layouts.
- The Gap: Even the best robots (Sora and Veo) are still far from perfect. They can get about 60-65% of the motion types right in complex scenes, whereas a perfect system (based on the original blueprint) would get nearly 70% right. This gap shows that the robots are still "hallucinating" movements that weren't asked for.
The Bottom Line
This paper doesn't invent a new robot; it invents the ruler used to measure them. It proves that current AI video generators are great at making "cool" videos, but they are still bad at following the strict, structured rules required for professional design work. By using this new grading system, developers can finally see exactly where their robots are failing so they can fix them.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.