💻 computer science

Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation

This paper introduces Animation2Code, a benchmark comprising 1,069 video-animation pairs and novel human-aligned metrics to evaluate and reveal the limitations of current vision-language models in reconstructing executable web animations with accurate temporal dynamics.

Original authors: Anya Ji, Abhijith Varma Mudunuri, David M. Chan, Alane Suhr

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anya Ji, Abhijith Varma Mudunuri, David M. Chan, Alane Suhr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magic movie projector. You feed it a video of a bouncing ball, a spinning logo, or a waving hand. Your goal? To ask a super-smart computer to write the exact computer code (the "recipe") that would recreate that video from scratch.

This is what the paper "Animation2Code" is all about. It's a new test to see if today's most advanced AI models can look at a moving video and write the instructions to make that motion happen again.

Here is a simple breakdown of their findings, using some everyday analogies:

1. The Challenge: Static vs. Dynamic

Think of current AI models like photographers. They are amazing at looking at a single photo of a webpage or a chart and writing the code to draw it perfectly. If you show them a picture of a blue square, they can write code to draw a blue square.

But this paper asks: Can they be choreographers?
Can they watch a video of a square that spins, shrinks, and then turns red, and write code that makes the square do exactly that sequence of moves? That requires understanding time and motion, not just a single snapshot.

2. The Test: "Animation2Code"

The researchers built a giant "exam" called Animation2Code.

The Study Guide: They collected 1,069 short web animation videos (like bouncing balls, loading spinners, and waving flags) along with the actual code that created them.
The Exam: They showed these videos to top AI models (like Gemini, GPT-4, Claude, etc.) and asked: "Write the code to make this video."
The Grading: They didn't just check if the code ran. They used two special "rulers" to grade the results:
- The Look-Ruler (Appearance): Does the generated video look like the original? (Is the ball blue? Is it round?)
- The Move-Ruler (Temporal): Does the generated video move like the original? (Does it bounce at the right speed? Does it spin in the right direction?)

3. The Big Discovery: "Good at Drawing, Bad at Dancing"

The results were surprising and a bit funny.

The AI is a Great Artist: The models were fantastic at the "Look-Ruler." They could write code that made a video that looked almost identical to the original. The colors, shapes, and styles were spot on.
The AI is a Clumsy Dancer: However, when it came to the "Move-Ruler," the models struggled badly. Even when the video looked perfect, the motion was often wrong.
- Analogy: Imagine an AI trying to recreate a video of a person doing a backflip. The AI writes code that draws a perfect human body in the right clothes. But in the video, the person just stands there, or falls over, or spins the wrong way. The "costume" is perfect, but the "dance" is broken.

4. Why This Matters

The paper found that even when the AI was given more video frames (more "clues" about the motion) or was given extra training to fix its mistakes, it still couldn't figure out the timing and physics of the movement.

The Gap: There is a huge gap between seeing a motion and understanding the logic of that motion well enough to write code for it.
The Limit: The models seem to be guessing the "vibe" of the movement rather than calculating the actual trajectory. They can mimic the look of a wave, but they can't write the code to make the water actually flow.

5. The Bottom Line

The researchers created this benchmark to prove that while AI is getting great at static tasks (like drawing a webpage from a photo), it is still very bad at temporal reasoning (understanding how things move over time) when asked to generate code.

They aren't saying the AI is useless; they are saying it's like a student who memorized the textbook definitions of "jumping" and "spinning" but has never actually learned how to do a physical backflip. The code they write looks right on paper, but when you run it, the animation doesn't move the way it should.

In short: Current AI can draw a perfect picture of a dancing robot, but it can't write the code to make the robot actually dance.

Technical Summary: Animation2Code

Problem Statement

While Vision-Language Models (VLMs) have demonstrated significant proficiency in static visual-to-code tasks (e.g., generating code for webpages, charts, or SVGs from images), their ability to recover temporal dynamics from video remains unexplored. Static de-rendering requires mapping a single image to a program, whereas dynamic de-rendering requires reasoning over sequences of frames, tracking object trajectories, and understanding motion timing. Current benchmarks for video understanding primarily focus on question-answering or classification, which do not require the model to produce executable code that deterministically reproduces complex motion. There is a lack of a benchmark that evaluates whether models can infer continuous, time-driven visual changes and translate them into programmatic representations (HTML/CSS/JavaScript).

Methodology

The Animation2Code Benchmark

The authors introduce Animation2Code, a benchmark designed to evaluate temporal visual reasoning through the task of animation de-rendering.

Dataset: The dataset consists of 1,069 high-quality web animation videos sourced from CodePen. Each video is paired with its ground-truth executable code (HTML with inline CSS and/or JavaScript).
Data Diversity: The animations cover diverse visual appearances and motion patterns, including rotation, translation, scaling, and opacity changes. The code complexity ranges from 7 to over 10,000 lines, with 86% being pure CSS and 14% incorporating JavaScript.
Task Definition: Given a reference video ( $V_{ref}$ ) rendered from ground-truth code, the model must generate a self-contained, executable HTML file ( $C$ ). When executed, $C$ must produce a rendered video ( $V_{gen}$ ) that matches $V_{ref}$ in both spatial appearance and temporal dynamics.
Rendering Pipeline: To ensure fair evaluation, the benchmark uses a headless Chromium pipeline to deterministically render code into MP4 videos (1024×768, 30 FPS), synchronizing CSS and JavaScript animations.

Evaluation Metrics

To disentangle visual fidelity from temporal alignment, the authors propose two automated, human-aligned metrics:

Appearance Similarity: Measures the visual resemblance between the generated and reference animations.
- Process: Animated regions are cropped to remove positional confounds. Frames are aligned using Dynamic Time Warping (DTW).
- Scoring: Uses DreamSim embeddings (trained on human perception) to calculate the mean similarity along the optimal alignment path.
Temporal Similarity: Measures the similarity of motion trajectories independent of visual appearance.
- Process: Uses CoTracker3 to extract motion tracklets. To handle the lack of pixel-level correspondence in generated vs. ground-truth animations, the system initializes tracking points programmatically on a grid.
- Scoring: Computes the correlation between tracklet displacements based on direction agreement (cosine similarity) and speed agreement (magnitude ratio). The final score aggregates pairwise correlations using a Chamfer distance.

Experimental Setup

The authors evaluated a suite of state-of-the-art commercial and open-source VLMs (Gemini 3 Flash Preview, Qwen3-VL-8B-Instruct, GPT-5.4, Claude Sonnet 4.6, and LLaMA 4 Scout) under three settings:

Zero-Shot Prompting: Direct generation from video input (native video or 2 FPS sampled frames).
Supervised Finetuning (SFT): Finetuning Qwen3-VL-8B-Instruct on the dataset using LoRA and full-parameter methods.
Iterative Refinement: A multi-agent framework (adapted from METAL) where the model iteratively compares its rendered output against the target and revises the code.

Key Results

Execution Success: Most models achieved high code execution rates (up to 100%), indicating they can generate syntactically valid HTML/CSS/JS.
Appearance vs. Temporal Gap:
- Models achieved strong appearance similarity (up to 0.84 for GPT-5.4), suggesting they can reproduce static visual layouts effectively.
- Temporal similarity remained uniformly low across all models and settings (maximum score of 0.31 for Gemini 3 Flash Preview).
Impact of Training and Refinement:
- Finetuning: While SFT improved execution success, it significantly degraded both appearance and temporal similarity, suggesting models relied on learned code priors rather than grounding in the input video.
- Iterative Refinement: This approach yielded the most consistent improvements, particularly in the first iteration, but temporal scores remained low compared to appearance scores.
Input Modality: Increasing the frame rate of input (from 2 FPS to 24 FPS) provided only marginal gains, indicating that simply providing denser temporal signals is insufficient for current models to learn continuous animation dynamics.
Qualitative Failures: Models often failed to reproduce correct trajectories (e.g., circular motion becoming linear or static) and struggled with spatial relationships, even when the general visual style was correct.

Significance and Claims

The paper claims to be the first benchmark to evaluate the de-rendering of web animations into executable code, specifically targeting temporal visual reasoning.

Revealing a Fundamental Limitation: The primary finding is that current VLMs possess a significant gap in their ability to interpret and replicate motion via code generation. They can reconstruct "what" an object looks like but fail to reconstruct "how" it moves over time.
Evaluation Suite: The proposed metrics (Appearance and Temporal Similarity) are validated against human judgments, showing strong correlation and providing a reliable, automated proxy for evaluating animation quality.
Research Direction: The benchmark isolates the challenge of recovering temporal dynamics, which is often confounded with appearance mismatches in existing benchmarks. It serves as a testbed for future research in multimodal program synthesis and temporal reasoning.
Ethical Sourcing: The dataset is constructed from MIT-licensed CodePen projects, aiming to mitigate risks associated with unlicensed cloning of web visuals.

The authors conclude that while models can reliably produce executable code and strong static visuals, they consistently fail to capture correct temporal dynamics, highlighting a critical area for future improvement in VLMs.