💻 computer science

GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models

This paper introduces GTASA, a corpus of multi-actor videos with precise spatiotemporal ground truth generated by the GEST-Engine, which addresses the challenges of evaluating video generation quality and reveals that self-supervised encoders capture spatial structure more effectively than VLM visual encoders.

Original authors: Nicolae Cudlenco, Mihai Masala, Marius Leordeanu

Published 2026-04-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nicolae Cudlenco, Mihai Masala, Marius Leordeanu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the world by showing it millions of home videos. The robot watches people cooking, playing, and fighting. But here's the problem: the robot has no idea why things are happening, or even if what it's seeing is physically possible. It just sees pixels moving.

This paper introduces a solution called GTASA (Ground Truth Annotations for Spatiotemporal Analysis). Think of it as a "perfect simulator" built inside a video game engine (specifically Grand Theft Auto: San Andreas) that generates videos with a built-in "cheat sheet" for the robot.

Here is the breakdown of what they did, using simple analogies:

1. The Problem: The "Magic Trick" vs. The "Blueprint"

Current AI video generators (like Sora or VEO) are like magicians. They can make a video of a cat riding a skateboard look incredibly realistic. But if you ask the magician, "Did the cat actually touch the ground? Did the skateboard break?", they might not know. The AI often hallucinates: objects disappear, people morph into different people, or physics breaks (like a cup floating through a table).

The authors say, "We need a blueprint." They built a system (called GEST-Engine) that doesn't just guess what a video should look like; it constructs the video from a strict set of rules, just like a video game does.

The Analogy: Imagine a puppet show. A neural generator is like a magician pulling a rabbit out of a hat—it looks cool, but you don't know how it works. The GEST-Engine is like a puppeteer with visible strings. You can see exactly where every hand, foot, and object is at every single millisecond.

2. The Solution: The "Perfect Cheat Sheet"

The system takes a story (e.g., "Two people meet, shake hands, and one gives the other a coffee") and runs it in a 3D game world. Because it's a game engine, it knows the exact 3D coordinates of every pixel.

The Output: It produces a video and a massive data sheet (the "Ground Truth") that says: "At frame 50, Person A is 2 meters from Person B. At frame 51, they are 1.5 meters away."
Why it matters: This data sheet is perfect. It has zero errors. Real-world cameras can't give you this; they only give you a blurry 2D picture.

3. The Three Big Tests (The "Report Card")

The authors put their system through three major tests to see if it helps AI learn better.

Test Q1: The "Reality Check" (Human Evaluation)

They asked humans to watch videos made by their game engine vs. videos made by top AI generators (VEO and WAN).

The Result: The AI generators were beautiful but broken. About 80-87% of the time, the AI videos had "magic tricks" (people disappearing, objects changing shape).
The Winner: The game-engine videos were less "pretty" (they look like a 2004 video game), but they were 100% logical. If the script said "shake hands," the hands actually shook. If the script said "walk through a door," the character didn't walk through the wall.
Takeaway: For training robots to understand logic, a "ugly but logical" video is better than a "pretty but broken" one.

Test Q2: The "Student" (Training Video Captioning)

They tried to teach AI models to describe videos (Video Captioning). They gave the AI two types of study materials:

Real videos (hard to learn from because the data is messy).
Synthetic videos from their game engine (perfect data).

The Result: The AI that studied the "perfect game data" became a much better student. It learned to describe actions and sequences more accurately than the AI that only studied real videos or videos made by other AI generators.
Analogy: It's like learning to drive. You can learn by watching chaotic real-world traffic (hard to predict), or you can learn in a driving simulator where the rules are perfect. The simulator-trained driver actually performed better in the real world.

Test Q3: The "X-Ray Vision" (Probing the AI's Brain)

They wanted to see what current AI models "know" about space and time. They used their perfect "cheat sheet" to quiz the AI: "Is Person A closer to the camera than Person B?" or "Did the car move left or right?"

The Result: They found a surprising split.
- Self-Supervised AI (models that learn by watching videos alone) were surprisingly good at understanding 3D space and movement.
- Vision-Language AI (models trained to talk about images) were actually quite bad at understanding spatial relationships. They could describe a cat, but they didn't really "know" where the cat was in 3D space.
Takeaway: Just because an AI can talk about a video doesn't mean it understands the physics of the world inside it.

4. The "Blind Spot" Discovery

In a final twist, they asked the most advanced AI models (like GPT-5 or Gemini) to look at the "broken" AI videos and say, "Is this physically possible?"

The Shock: The AI models failed miserably. They couldn't tell the difference between a real video and a video where a person's head suddenly turned into a pumpkin. They just kept describing the video as if everything was normal.
Meaning: Current AI is "blind" to physical impossibilities. It sees the pattern, not the reality.

Summary

The paper argues that to build truly smart AI that understands the physical world, we can't just feed it millions of random YouTube videos. We need structured, perfect data generated by game engines.

The Metaphor: Current AI is like a child who has seen a million cartoons but has never touched a real ball. They know what a ball looks like, but they don't know how it rolls. The authors built a "playground" (the game engine) where the AI can learn the rules of physics perfectly, so it can eventually understand the real world.

The Bottom Line: Don't just make pretty pictures. Build a world where the rules are known, so the AI can learn the truth behind the pixels.

1. Problem Statement

The paper addresses three critical limitations in current video generation and understanding research:

Lack of Ground Truth: Evaluating neural video generators is difficult because they lack "ground truth" for physical plausibility and semantic faithfulness. Current models often exhibit physically impossible interactions (e.g., objects appearing/disappearing) and fail to maintain consistent multi-actor narratives.
Annotation Trade-offs: Existing datasets either offer massive scale with coarse labels (e.g., Kinetics) or dense annotations (scene graphs) at a prohibitive human cost and limited scale. Synthetic datasets often lack complex multi-actor coordination or narrative structure.
Blind Spots in Encoder Understanding: It remains unclear whether current video encoders (especially Vision-Language Models) actually learn 3D spatial structures and dynamic entity relationships, as real-world datasets lack exact 3D pairwise spatial ground truth for every frame.

2. Methodology: GEST-Engine

The authors introduce GTASA (Ground Truth Annotations for Spatiotemporal Analysis), a dataset and system built upon GEST-Engine.

Core Concept: Instead of relying on neural generators to "hallucinate" video, the system uses a 3D game engine (GTA San Andreas via Multi Theft Auto) to execute explicit, deterministic specifications.
Input Representation (GEST): The system takes a Graph of Events in Space and Time (GEST). This graph encodes actors, actions, objects, and their temporal/spatial constraints.
Pipeline Stages:
1. Procedural Graph Generation: A generator creates complex multi-actor narratives by selecting episodes, assigning actors, and defining action chains with temporal constraints (using Allen's relations and Floyd-Warshall constraint satisfaction).
2. Simulation Execution: The graph is executed in the 3D engine. The engine handles collision, occlusion, and physics deterministically.
3. Artifact Collection: The system captures perfect ground truth at zero marginal cost:
  - Frame-synchronized pairwise spatial relation graphs (Euclidean distance, relative direction, angular offsets).
  - Exact event-to-frame temporal mappings.
  - Instance segmentation and depth (optional).
4. Text Generation: The graph is converted into a "proto-language" description, which is then refined by an LLM (GPT-4o) into natural language descriptions for training and evaluation.
5. Dataset Assembly: The system generates 938 unique multi-actor video clips (14.25 hours total) containing 29.6K events and 892M pairwise spatial relations.

3. Key Contributions

GEST-Engine System: An open-source, scalable pipeline for generating controllable multi-actor narrative videos with perfect spatiotemporal annotations (3D positions, relations, and event boundaries) derived from game engine simulation.
GTASA Dataset: A public corpus of 938 procedurally generated videos with 28.2K action clips, 892M spatial relations, and 29.6K exact temporal mappings.
Comprehensive Evaluation Framework: The paper establishes three distinct research questions (Q1–Q3) to validate the utility of this synthetic data:
- Q1: Can game engines generate physically valid videos superior to neural generators?
- Q2: Can this synthetic data train better video captioning models?
- Q3: What spatiotemporal information do frozen video encoders actually encode?

4. Experimental Results

Q1: Human Evaluation of Video Generation

Setup: Human annotators compared videos generated by GEST-Engine against state-of-the-art neural generators (VEO 3.1 and WAN 2.2) using identical text prompts.
Metrics: Physical Validity (binary) and Semantic Alignment (5-point Likert scale).
Findings:
- Validity: GEST-Engine achieved a 69% validity rate, significantly outperforming VEO 3.1 (18%) and WAN 2.2 (13%). Neural generators frequently violated physics (objects morphing, disappearing).
- Semantics: GEST-Engine scored 4.09/5 on semantic alignment, while neural models scored ~1.75–2.50.
- Conclusion: While neural models are visually photorealistic, they fail at semantic coherence and physical plausibility in complex multi-actor scenarios.

Q2: Video Captioning Training

Setup: Fine-tuned two VLMs (VideoLLaMA3 and Qwen3-VL) under three conditions: (G) GEST synthetic data $\to$ Real data, (V) VEO synthetic data $\to$ Real data, and (R) Real data only.
Findings:
- Models trained with GEST data (G) consistently outperformed those trained with VEO data (V) and Real-only data (R) across BLEU, METEOR, CIDEr, and ROUGE-L metrics.
- An LLM jury (GPT-5, Gemini, Claude) preferred captions from the (G) condition over (V) and (R), confirming that the structural accuracy of GEST data compensates for the visual domain gap (low-fidelity GTA graphics).

Q3: Spatiotemporal Probing of Encoders

Setup: Probed four frozen video encoders (V-JEPA 2, VideoMAE V2, Qwen3-VL, VideoLLaMA3) on 11 tasks using GTASA's exact 3D ground truth. Tasks included actor counting, depth ordering, motion presence, and relative direction.
Findings:
- Self-Supervised > VLM: Self-supervised encoders (V-JEPA 2, VideoMAE V2) significantly outperformed VLM visual encoders (Qwen3-VL) in encoding spatial structure (Avg. 60.6% vs 52.4% balanced accuracy).
- Dynamic Tasks: The gap was largest in dynamic tasks (e.g., Motion Presence: 88.6% for V-JEPA vs 63.1% for Qwen3-VL).
- Weaknesses: All models struggled with fine-grained pairwise compass direction (near random baseline). VLMs generally failed to detect physically invalid videos (accuracy ~56%), revealing a blind spot in current foundation models.

5. Significance and Impact

Benchmarking Standard: GTASA provides the first dataset with exact 3D pairwise spatial ground truth for every frame, enabling rigorous probing of what video models actually "know" about the world.
Training Paradigm: Demonstrates that structurally perfect synthetic data (even with low visual fidelity) is more effective for training video understanding models than high-fidelity but semantically noisy neural generations or real-world data alone.
Diagnostic Tool: Reveals that current frontier VLMs are poor at detecting physical impossibilities and lack robust internal models of 3D spatial dynamics, suggesting a need for training objectives that prioritize structural reasoning over purely visual reconstruction.
Future Direction: The authors plan to port the system to GTA V for higher visual fidelity, bridging the gap between synthetic precision and real-world visual realism.