🤖 AI

PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning

This paper introduces PRISM, a large-scale, multilingual benchmark and a comprehensive evaluation framework designed to rigorously assess the spatial-temporal reasoning capabilities of language models in programmatic video generation, revealing a significant gap between code executability and visual spatial coherence.

Original authors: Qiran Zhang, Yuheng Wang, Runde Yang, Lin Wu, Jingru Fan, Shu Yao, Jie Zhang, Tianle Zhou, Huatao Li, Ruijie Shi, Yihan Li, Chen Qian

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qiran Zhang, Yuheng Wang, Runde Yang, Lin Wu, Jingru Fan, Shu Yao, Jie Zhang, Tianle Zhou, Huatao Li, Ruijie Shi, Yihan Li, Chen Qian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a robot architect to build a moving, animated movie about math or history. You give the robot a script (the prompt), and it writes the code to build the scene.

For a long time, we've been asking: "Did the robot finish building the house?" If the code runs without crashing, we said, "Great job!"

But the paper PRISM argues that's not enough. Just because the house is built doesn't mean the rooms are in the right place, the furniture isn't floating in mid-air, or the walls aren't collapsing on the actors.

Here is the paper explained in simple terms:

1. The Problem: "Running" vs. "Looking Right"

Think of Pixel-level AI (like standard video generators) as a painter who tries to paint a movie frame by frame. They are great at making things look realistic, but they often mess up the logic. A character might walk through a wall, or text might appear upside down.

Programmatic AI (what this paper studies) is different. It's like a robot that writes a precise set of instructions (code) to build the movie. It's supposed to be perfect because it follows rules.

The Old Test: Did the robot finish writing the instructions? (Yes/No)
The New Problem: The robot might finish the instructions perfectly, but the instructions might tell the robot to put a giant tree inside a tiny house. The code runs, but the movie looks broken.

2. The Solution: PRISM (The "Stress Test")

The authors created a massive new test called PRISM.

The Scale: They gathered over 10,000 examples (20 times bigger than any previous test). These cover 437 different topics, from "How to solve an algebra problem" to "The history of lychees."
The Bilingual Twist: They tested this in both English and Chinese.
The Human Touch: They didn't just let computers grade the work. Humans checked the "blueprints" to make sure the code actually made sense for the story being told.

3. The New Grading System: The "Funnel"

Instead of just giving a pass/fail grade, PRISM uses a funnel with four layers to catch different types of mistakes:

The Gatekeeper (Code Reliability): Does the code even run? If it crashes, it's out.
The Architect (Spatial Reasoning): This is the big one. Does the layout make sense?
- Overlap: Did two objects smash into each other?
- Out-of-Bounds: Did an object float off the screen?
- Leakage: Did a word spill out of its box?
The Director (Dynamic Complexity): Is the video too boring (like a static slideshow) or too chaotic (too much spinning and jumping)? The test checks if the movement matches the story.
The Editor (Temporal Density): Is the video moving at a good pace, or is it flashing too fast?

4. The Shocking Discovery: The "Execution-Spatial Gap"

The authors tested 7 of the smartest AI models available (including big names like GPT, Gemini, and Claude).

The Result:

The Good News: The AIs are getting very good at writing code that runs. Most of the time, the code doesn't crash.
The Bad News: There is a massive gap between "running" and "looking right."
- On average, 41% of the videos that ran successfully were spatially broken.
- Imagine a robot that can build a car engine perfectly, but then puts the wheels on the roof. The engine runs, but the car doesn't drive.

Key Findings:

Thinking Longer Doesn't Help: The authors tried letting the AI "think" longer before answering (a feature called "Thinking Mode"). It didn't fix the spatial problems. The AI just got more confident in its wrong answers.
Too Much Action is Bad: When the AI tried to make the video too exciting with lots of movement, the layout fell apart.
Language Matters: The AI struggled differently with English vs. Chinese, often making more layout errors with Chinese text.

5. The Bottom Line

The paper concludes that we can no longer just ask, "Does the code run?" We have to ask, "Does the code create a logical, organized world?"

Currently, even the smartest AIs are like talented but clumsy stagehands. They can follow the script to turn the lights on and move the props, but they often forget to check if the props are blocking the actors' faces or if the scenery is falling over. PRISM is the new rulebook to force them to learn how to be better stage managers.

Technical Summary: PRISM - A Benchmark for Programmatic Spatial-Temporal Reasoning

1. Problem Statement

While recent advances in Large Language Models (LLMs) have expanded automated content creation to video, existing approaches face significant limitations in programmatic video generation. Current methods generally follow two routes:

Pixel-level generation (Diffusion Models): These achieve high visual fidelity but struggle with spatial consistency, accurate symbol rendering, and long-horizon logical coherence due to a lack of explicit geometric constraints.
Programmatic generation: This approach executes scripts (e.g., using Manim) to define objects, layouts, and temporal dynamics, offering superior geometric precision and controllability.

However, a critical gap remains in evaluating whether LLMs can synthesize video programs that are not only executable but also spatially and temporally correct. Existing benchmarks are insufficient because:

They focus on static outputs (charts, webpages) rather than continuous temporal sequences requiring consistent object relations.
They are small-scale and fail to cover diverse real-world scenarios.
Evaluation protocols rely on coarse-grained "VLM-as-judge" assessments or simple execution checks, missing fine-grained rendering failures like layout misalignment or out-of-bounds objects that appear only in specific frames.

The core challenge is the "Execution-Spatial Gap": the discrepancy between code that runs successfully and code that produces visually coherent, spatially correct animations.

2. Methodology

2.1 PRISM Benchmark Construction

The authors introduce PRISM (Programmatic Reasoning In Spatial Modalities), a large-scale, bilingual (English/Chinese) benchmark containing 10,372 human-calibrated instruction-code pairs.

Scale & Scope: It is 20× larger than prior programmatic video benchmarks, spanning 437 subject categories grounded in real-world knowledge visualization (e.g., educational animations, scientific demonstrations).
Data Curation Framework: A hierarchical, human-in-the-loop pipeline was employed to filter over 30,000 raw candidates:
1. Deterministic Hard Filtering: Automated scripts parse the Manim rendering pipeline to detect geometric violations (e.g., coordinate thresholds) and remove samples with layout defects.
2. Boundary Calibration: A gold subset of 3,000 expert-annotated examples was used to refine exemption rules and tolerance thresholds, minimizing false positives.
3. Human Spot-Check: Final semantic review ensures animations match educational content and are pedagogically appropriate.
Task Definition: The task is formulated as a constrained code generation problem $f_\theta: I \to O$ , mapping natural language instructions (educational outlines) to a Manim program ( $C$ ) and its rendered video ( $V$ ).

2.2 Evaluation Framework

PRISM proposes a funnel-style evaluation framework with four complementary metrics:

Code-Level Reliability: Measures execution robustness. It includes the basic pass rate (first-attempt success), fine-grained error categorization (API hallucination, misuse, text-rendering errors, etc.), and execution complexity (mean render time).
Spatial Reasoning: The core end-to-end metric assessing layout planning over the full animation sequence. It detects three failure modes:
- Layout Overlap: Unintended occlusion of visual elements.
- Out-of-Bounds (OOB): Elements extending beyond the visible frame.
- Layout Leakage: Geometric extensions exceeding local regions (e.g., text escaping a box).
- Mechanism: Uses an automated audit pipeline capturing key snapshots, extracting object point sets and bounding extents, and applying deterministic checks.
Prompt-Aware Dynamic Visual Complexity (PADVC): Quantifies input-relevant dynamic richness. It distinguishes between near-static outputs (under-dynamic) and excessive decorative motion (over-dynamic). It calculates a "centered" score ( $PADVC_c$ ) based on the geometric evolution energy of non-text objects relative to the prompt's visual density.
Temporal Density (TD): Measures the rate of frame-level visual change (temporal activity). Like PADVC, it uses a centered score ( $TD_c$ ) to identify videos that are either too static or suffer from redundant transitions.

3. Key Results

The authors evaluated seven mainstream LLMs (four closed-source: GPT-5.4, Gemini 3.1 Pro, Kimi K2, Claude Sonnet 4.6; three open-source: Qwen3.5, GLM-5, DeepSeek-V3.2) on the benchmark.

The Execution-Spatial Gap: A striking finding is the average drop of ~41% from execution success rate to spatial pass rate. For example, GPT-5.4 achieved a 94.5% execution rate in English but only a 30.8% spatial pass rate. This confirms that runnable code does not guarantee spatial coherence.
Model Performance:
- Closed-source models generally led in code-level reliability (execution success) but often suffered from high spatial failure rates due to "visually ambitious" outputs violating constraints.
- Gemini 3.1 Pro Preview demonstrated the best alignment, achieving the highest spatial scores with one of the smallest Execution-Spatial gaps.
- Open-source models showed varied performance; GLM-5 was the most balanced, while others struggled with API misuse (Qwen3.5) or boundary control (DeepSeek-V3.2).
Dynamic vs. Spatial Trade-off: High dynamic intensity (high PADVC or TD) correlates with lower spatial quality. Models producing excessive motion or text expansion often triggered layout violations.
Thinking Mode Ablation: Enabling "thinking" modes did not reliably improve spatial organization. In some cases (e.g., Claude Sonnet 4.6), thinking led to bolder layout changes that increased spatial errors (overlap/OOB) without improving the final spatial pass rate.

4. Significance and Claims

The paper claims that PRISM provides a principled benchmark for advancing spatially coherent code generation, moving beyond the limitations of current evaluation methods.

Beyond Executability: The work establishes that evaluating programmatic video generation must go beyond simple code execution to include rigorous spatial and temporal reasoning metrics.
Identifying Bottlenecks: The results highlight that long-horizon spatial reasoning is the primary bottleneck for current LLMs, rather than basic code generation capabilities.
Resource Provision: By releasing a large-scale, high-quality dataset with a deterministic evaluation framework, PRISM offers a standardized testbed for future research into spatially grounded programmatic video generation.

The authors conclude that while LLMs can generate executable code, they currently lack the robustness to maintain geometric consistency over extended animated sequences, a gap that PRISM is designed to measure and help bridge.