🤖 machine learning

Training and Benchmarking Code Generation for Physics-Inspired Animations

This paper introduces SimuScene, a comprehensive dataset and benchmark for training and evaluating large language models on generating executable code for physics-inspired animations, demonstrating that even top models struggle with this task but can be significantly improved through a novel reinforcement learning pipeline utilizing visual rewards.

Original authors: Yanan Wang, Renxi Wang, Yongxin Wang, Xuezhi Liang, Fajri Koto, Timothy Baldwin, Xiaodan Liang, Haonan Li

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yanan Wang, Renxi Wang, Yongxin Wang, Xuezhi Liang, Fajri Koto, Timothy Baldwin, Xiaodan Liang, Haonan Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can read a storybook and instantly write the computer code to make a movie of that story. You tell it, "A ball rolls down a hill and bounces off a wall," and you expect to see a ball rolling down a hill and bouncing off a wall. This is the dream of Large Language Models (LLMs): computers that can understand human language and turn it into action. But there's a tricky part. Just because the robot writes code doesn't mean the movie it makes will actually look like the story. The code might run, but the ball might float like a ghost, roll uphill, or turn into a square. This paper explores a specific corner of science where we ask: Can these AI robots not just write the script, but actually direct the physics of the movie so it feels real? It's like asking a chef to not just write a recipe for a cake, but to actually bake one that tastes like a cake, not a brick.

The researchers behind this study, led by a team from Mohamed bin Zayed University of Artificial Intelligence and Sun Yat-sen University, decided to put these AI robots to the ultimate test. They created a new game called SimuScene. Think of it as a giant, automated playground with 52 different physics themes, like gravity, light, electricity, and how fluids flow. They built a system that generates thousands of unique scenarios—like "a spinning disk on a slippery table" or "light bending through a lens"—and then asks the AI to write Python code to animate them. But here's the catch: the AI doesn't just get a pass for writing code that runs. The code has to produce a video that actually looks like the physics described. To check this, they used a special "referee" (a Vision-Language Model) that watches the generated video and answers questions like, "Did the ball bounce?" or "Did the light bend the right way?" If the video fails the questions, the AI fails the test.

The results were a bit of a reality check. Even the smartest, most advanced AI models available today struggled mightily. When the researchers asked 10 different top-tier AIs to generate these animations, the very best one only got about 21.5% of the scenarios right on average. That means for every 100 attempts, the AI only successfully created a video that matched the physics description about 21 times. Most of the time, the code would run, but the animation would be weird: objects might pass through walls, move in the wrong direction, or ignore gravity entirely. It turns out that teaching an AI to write code is one thing, but teaching it to understand the invisible rules of how the world moves and to visualize those rules is a much harder puzzle.

However, the paper doesn't just stop at saying "it's hard." The team also tried a clever trick to make the AIs better. They used a method called Reinforcement Learning, which is like training a dog with treats. Instead of just telling the AI "good job" or "bad job" based on the text, they let the AI generate a video, watched the video, and gave it a "treat" (a reward) only if the video actually looked right. They used a vision-based judge to give these rewards. After this training, the AI models improved significantly. A smaller model that started with a 0% success rate managed to reach an 11.1% pass rate after training, and a larger model jumped from 18.3% to 34.4%. This suggests that while the AIs aren't perfect physics engines yet, they can learn to get better if you show them the actual movies they create and let them see their own mistakes.

In short, this paper shows that while our AI friends are getting great at writing code, they are still learning how to be good directors of physics. They can write the script, but making the movie look real is still a work in progress. The team built a massive dataset of over 7,600 physics scenarios to help train them, and their experiments prove that watching the results (the videos) and learning from them is a powerful way to teach these models. It's a hopeful sign that with the right training, we might soon have AI that can turn our wildest physical descriptions into accurate, moving pictures.

Technical Summary: SimuScene – Training and Benchmarking Code Generation for Physics-Inspired Animations

Problem Statement

While Large Language Models (LLMs) have demonstrated strong capabilities in mathematical reasoning, complex coding, and scientific problem solving, their ability to generate executable code that visually depicts physical scenarios and their qualitative dynamics remains underexplored. Existing benchmarks typically evaluate symbolic reasoning, static coding tasks, or perceptual understanding in isolation. They do not require models to produce code whose rendered animations are visually aligned with dynamic physical descriptions.

The core challenge lies in the intersection of physical-scenario understanding and animation-oriented code generation. Producing such animations requires not only qualitative physical understanding (e.g., motion patterns, temporal evolution, object interactions) but also the coding ability to represent these concepts in executable visual form. The paper posits that the goal is not numerically precise simulation, but rather the generation of code that produces visually interpretable animations consistent with qualitative physical descriptions.

Methodology

1. SimuScene Dataset Construction

The authors introduce SimuScene, a novel task, dataset, and benchmark designed to evaluate LLMs on physics-inspired animation generation.

Scope: The dataset covers 52 concepts across five classical physics domains: Mechanics, Electromagnetism, Optics, Fluid Mechanics, and Thermodynamics.
Pipeline: A multi-stage automated pipeline was developed to ensure data quality:
1. Scenario Generation: Starting from 52 concepts, GPT-4o generates basic physical scenarios and additional conditions (e.g., "Considering air resistance") to control difficulty.
2. Plausibility Filtering: Scenarios are automatically checked for physical plausibility.
3. Verification Question Generation: For valid scenarios, GPT-4o generates visual verification questions describing expected qualitative dynamics.
4. Question Validation: A second review stage (using DeepSeek-R1-0528) assesses whether the question set is sufficient to judge the full motion described.
5. Human Verification: A 334-example test set was manually verified to ensure clear descriptions, plausible scenarios, and comprehensive verification questions.
Statistics: The final dataset contains 7,659 validated scenarios (7,325 for training, 334 for testing). The training data includes 12,556 correct responses (visually verified) and 4,325 scenarios used for Reinforcement Learning (RL).

2. Evaluation Pipeline

The evaluation process involves three steps:

Code Generation: The LLM generates Python code based on a scenario description.
Video Rendering: The code is executed to render a video depicting the qualitative dynamics.
VLM-Based Verification: A Vision-Language Model (VLM) evaluates the rendered video against the verification questions. A response is considered correct only if all verification questions are answered as "True."

Reliability: The VLM judge (Qwen2.5-VL-72B-Instruct) was validated against human annotators, achieving 87.8% agreement, confirming the reliability of automated evaluation.

3. Training Approach: Code-Video-Judge RL

To improve performance, the authors propose a training pipeline combining Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) with visual rewards.

Reward Signal: A Code-Video-Judge reward is computed by aggregating scores from multiple verification questions answered by VLMs over code-generated videos.
Binary Reward: The default reward is binary ( $r_{binary}$ ), assigning a pass only if all verification questions are true.
RL Algorithm: Group Relative Policy Optimization (GRPO) is used to optimize the policy based on these visual rewards.
Reward Hacking Mitigation: Strategies include multi-question aggregation, ensemble VLM voting, and separating training and evaluation VLMs. Human evaluation confirmed no evidence of systematic reward hacking.

Key Contributions

SimuScene Benchmark: The first systematic study and dataset for evaluating LLMs on generating executable code for physics-inspired animations, featuring a rigorously human-verified test set of 334 scenarios and quality-controlled training data.
Comprehensive Benchmarking: Evaluation of 10 frontier LLMs (including GPT-5, DeepSeek-R1, Qwen3, and Gemini) reveals significant limitations, with the strongest model achieving only 21.5% Avg@8 accuracy.
Visual Reward Training: Introduction of a reinforcement learning pipeline using vision-based rewards from code-generated videos to train text-only LLMs. This approach yields substantial gains, with a trained 7B model achieving an 11.1% pass rate (compared to 0% for the baseline) and a 32B model reaching 72.2% Pass@8.

Results

Baseline Performance

Low End-to-End Accuracy: Even the strongest models struggle to generate animations that are both executable and visually aligned with descriptions. DeepSeek-R1-0528 achieved only 21.5% Avg@8 accuracy.
High Execution, Low Alignment: Models generally achieve high Executable Rates (E.R.) and Render Rates (R.R.), but lower Playable Rates (P.R.) and Accuracy (Acc.), indicating difficulty in translating scene descriptions into plausible qualitative dynamics.
Domain Variance: Performance varies by domain. Mechanics (27.7% Avg@8) and Thermodynamics (26.0%) perform better than Optics (8.4%), likely due to data scarcity in the latter.

Training Improvements

SFT Impact: Supervised Fine-Tuning substantially improves all metrics over the base model.
RL Gains: RL further boosts performance, particularly in converting rendered videos to playable ones and improving final accuracy.
- The 7B model improved from 0.0% to 34.4% Pass@8 accuracy after RL.
- The 32B model achieved 72.2% Pass@8, surpassing untrained frontier models like DeepSeek-R1.
Reward Ablation: Visual feedback (VLM-as-judge) proved superior to text-only LLM-as-judge rewards, providing more faithful supervision by observing actual motion and interactions rather than inferring them from static code.

Significance and Claims

The paper claims that SimuScene fills a critical gap in evaluating LLMs at the intersection of physical reasoning and code generation. It highlights that current models lack the consistency to generate animations that faithfully capture qualitative physical dynamics, even when they can generate syntactically correct code.

The work demonstrates that training with curated physics data and vision-based rewards effectively improves a model's ability to generate text-to-code-to-video pipelines that are visually consistent. The authors emphasize that this is among the first works to use vision-based rewards to train text-only models for such tasks, validating the reliability of VLM-based judging without evidence of reward hacking.

The study suggests that while current models face significant challenges, targeted training can bridge the gap between textual physical descriptions and executable visual simulations, supporting downstream applications in educational visualization and controllable animation prototyping.