Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos

This paper introduces Synthetic Visual Genome 2 (SVG2), a large-scale automated panoptic video scene graph dataset with over 636K videos, and presents TRaSER, a novel model that leverages trajectory-aligned token mechanisms to significantly outperform existing baselines in scene graph generation and downstream video question answering tasks.

The Gist

Imagine you are watching a busy street scene in a video. A human brain doesn't just see "a car" and "a person." It instantly understands a complex story: "The red car is speeding past the tall man, who is looking at his phone while waiting for the green light."

For decades, computers have been terrible at this. They can recognize objects, but they struggle to understand the relationships, the timing, and the details all at once. They get lost in the chaos of a moving video.

This paper introduces SVG2 (Synthetic Visual Genome 2) and TraSeR, a new system designed to teach computers how to "read" a video like a human does. Here is the breakdown in simple terms.

1. The Problem: The "Blind" Computer

Think of existing video datasets as a library with only a few thousand books, and those books are missing half their pages.

• Too Small: Old datasets had very few videos.
• Too Lazy: They often only described the first second of a video, missing what happened later.
• Too Expensive: To fix this, humans would have to watch millions of hours of video and write down every detail (e.g., "blue shirt," "running," "holding a cup"). This is too slow and expensive to do manually.

2. The Solution: The "Robot Librarian" (SVG2)

The authors built a fully automated pipeline to create a massive new library called SVG2. Instead of hiring humans, they built a team of AI "robots" that work together to watch videos and write the story.

• The Tracker (The Eye): First, they use a robot (SAM2) that acts like a super-accurate highlighter. It draws a mask around every single object in every frame of the video, even if the object disappears behind a wall and comes back. It keeps a perfect "ID card" for every object so it knows, "That's still the same red car."
• The Describer (The Voice): Next, another robot (DAM) looks at each highlighted object and writes a detailed description. Instead of just "dog," it writes "a small, fluffy, golden dog."
• The Storyteller (The Brain): Finally, a powerful AI (GPT-5) looks at the whole scene and figures out the relationships. It connects the dots: "The dog is chasing the ball," or "The ball is under the bench."

The Result: They created a dataset with 636,000 videos and millions of details. It's like turning a small pamphlet into a 100-volume encyclopedia of video life. They even had a few humans double-check the work, and the robots were right 93% of the time.

3. The New Model: TraSeR (The "Smart Reader")

Now that they have this massive library, they needed a student to learn from it. They built a new AI model called TraSeR.

Most AI models try to watch a video like a human watches a movie—frame by frame, getting overwhelmed by the sheer amount of data. TraSeR is different. It uses a clever trick called "Token Resampling."

• The Analogy: Imagine trying to read a 10-hour movie script. If you read every single word, you'll get tired and forget the beginning by the time you reach the end.
• TraSeR's Trick: Instead of reading every word, TraSeR creates a "summary card" for each character.
  • The Global Card: It summarizes the whole life of an object (e.g., "The man was wearing a hat the whole time").
  • The Moment Card: It also zooms in on short, specific moments to catch quick actions (e.g., "At second 14, the man dropped the hat").

By organizing the video data this way, TraSeR can process the whole story in one go without getting confused.

4. Why This Matters: The "Superpower"

The paper tested TraSeR on several tasks, and the results were impressive:

• Better Detection: It found objects and relationships much better than previous open-source models (improving by 15–40%).
• Beating the Giants: It even beat the most advanced commercial AI models (like GPT-5) at predicting what objects are doing and what they look like.
• The "Cheat Sheet" Effect: The most exciting part? When they fed TraSeR's "story notes" (the scene graph) to another AI to answer questions, that AI got smarter.
  • Without notes: "What is the man doing?" -> "I think he's walking." (Maybe wrong).
  • With notes: "The man is walking while holding a coffee cup." -> "He is walking and drinking coffee." (Correct).

Summary

Think of SVG2 as a massive, perfectly organized library of video stories written by robots. TraSeR is the brilliant student who learned to read that library so well that it can now understand videos better than almost any other computer.

This is a huge step forward because it moves computers from just "seeing" pixels to truly understanding the dynamic, moving world around them. It's the difference between a security camera that just records footage and a security guard who can tell you exactly what happened, who did it, and why.

Technical Summary

1. Problem Statement

Video understanding requires moving beyond static object recognition to capturing structured, dynamic interactions between entities over time. Spatio-temporal scene graphs (VSGs) serve as a crucial intermediate representation for this, encoding objects, attributes, and their evolving relationships. However, the field faces significant bottlenecks:

• Data Scarcity: Existing VSG datasets (e.g., VidOR, PVSG) are small-scale, often limited to thousands of videos, and rely on expensive, sparse manual annotations.
• Annotation Limitations: Manual efforts often miss objects that appear/disappear, fail to capture dense frame-level attributes, and suffer from long-tail bias and inconsistent temporal labeling.
• Model Generalization: Models trained on these limited datasets struggle to generalize to open-vocabulary scenarios, new objects, and complex temporal dynamics.
• Computational Complexity: Generating VSGs from raw video involves handling massive token sequences and combinatorial relation candidates, making dense encoding computationally infeasible for standard Vision-Language Models (VLMs).

2. Methodology

The authors propose a two-pronged solution: (1) SVG2, a massive synthetic dataset created via an automated pipeline, and (2) TraSeR, a specialized model architecture designed to generate VSGs efficiently.

A. Synthetic Visual Genome 2 (SVG2) Dataset

SVG2 is a large-scale, panoptic video scene graph dataset containing 636K videos, 6.6M objects, 52M attributes, and 6.7M relations. It is constructed using a fully automated, three-phase pipeline:

1. Phase 1: Panoptic Trajectory Generation

   • Core Tech: Uses SAM2 (Segment Anything Model 2) with multi-scale grid prompts to generate dense per-frame masks.
   • Tracking Innovation: Introduces an Online-Offline Two-Stage Tracking mechanism.
     • Online: Propagates masks forward while monitoring uncovered regions to detect new objects (emergence) and assign new IDs.
     • Offline: Replays the video to re-initialize objects at their exact first appearance, ensuring global temporal consistency and identity preservation.
   • Verification: Uses a lightweight post-filtering module to remove redundant tracks.

2. Phase 2: Object Description & Parsing

   • Description: Uses Describe Anything Model (DAM) to generate detailed textual descriptions for each object trajectory.
   • Parsing: Employs GPT-4.1-nano to extract structured object names and attributes (e.g., color, shape, material) from the descriptions.
   • Verification: A SAM3-based verification step cross-checks extracted labels against instance trajectories to filter out hallucinations.

3. Phase 3: Inter-Object Relation Extraction

   • Inference: Uses GPT-5 to infer spatio-temporal relations between objects.
   • Strategy: To avoid bias toward trivial spatial relations (e.g., "left of"), the pipeline splits extraction into two passes: one for Spatial relations and one for Non-Spatial relations (functional, stateful, motion, social, attentional, event-level).
   • Output: Produces temporally grounded relations with specific start/end frame intervals.

B. TraSeR Model Architecture

TraSeR (Trajectory-grounded Scene graph Reasoner) is a VLM-based model designed to convert raw videos and panoptic trajectories into structured VSGs in a single forward pass. It addresses the challenge of aligning visual tokens with object identities and temporal dynamics.

• Trajectory-Aligned Token Arrangement:

  • Instead of processing raw video patches, TraSeR binds ViT tokens directly to object trajectories based on segmentation masks.
  • It creates ordered token streams for each object, separated by special [TRJ] tokens, ensuring the model maintains consistent object identity over time.

• Dual Resampler Module:

  • Object-Trajectory Resampler: Aggregates all tokens for a specific object across its entire lifespan to capture global semantic context.
  • Temporal-Window Resampler: Partitions the video into short windows (e.g., 4 seconds) and resamples tokens within each window. This preserves local motion dynamics and fine-grained temporal changes that global aggregation might lose.
  • The final representation concatenates global summaries with temporal window summaries, providing a compact, time-aware, and object-centric input for the language model.

3. Key Contributions

1. SVG2 Dataset: The largest and most diverse video scene graph dataset to date, offering an order-of-magnitude increase in scale (636K videos) and diversity compared to prior benchmarks. It includes dense, frame-level annotations for objects, attributes, and relations.
2. Automated Pipeline: A robust, fully automated synthesis framework combining SAM2, DAM, and GPT-5 that achieves high annotation accuracy (93.8% for objects, 88.3% for attributes, 85.4% for relations) verified by human evaluation.
3. TraSeR Architecture: A novel VLM design featuring trajectory-aligned token arrangement and dual resamplers (global and temporal) that effectively handles the combinatorial complexity of video scene graph generation.
4. SVG2test Benchmark: A high-quality, expert-annotated diagnostic benchmark (100 videos) with open-vocabulary categories and multi-granularity annotations, enabling rigorous evaluation beyond closed-set metrics.

4. Results

• Scene Graph Generation Performance:

  • TraSeR outperforms the strongest open-source baselines (e.g., Qwen2.5-VL, InternVL) by +15–20% in relation detection and +30–40% in object prediction.
  • It surpasses GPT-5 by +13% in object prediction and +15% in attribute prediction, demonstrating that specialized architecture and high-quality training data can outperform massive proprietary models on this specific task.
  • On the SVG2test benchmark, TraSeR achieves state-of-the-art performance across all metrics (triplet, relation, object, attribute).

• Video Question Answering (VQA):

  • When TraSeR-generated scene graphs are used as an intermediate representation for VQA (feeding them into GPT-4.1), it yields absolute accuracy gains of +1.5% to +4.6% over using video-only inputs or video augmented with Qwen2.5-VL's generated graphs.
  • This proves that explicit spatio-temporal scene graphs help VLMs reason more effectively about video content.

• Ablation Studies:

  • Training on the full SVG2 dataset (including the synthetic PVD subset) significantly boosts generalization compared to training only on academic datasets.
  • Both the object-trajectory and temporal-window resamplers are essential; removing either degrades performance in object recognition or relation localization, respectively.

5. Significance

• Scalability: SVG2 demonstrates that high-quality, dense video annotations can be synthesized at a scale previously unattainable through manual effort, breaking the data bottleneck for video reasoning.
• Representation Learning: The success of TraSeR highlights the importance of trajectory-aligned tokenization and temporal resampling in VLMs, offering a blueprint for future video understanding models that need to handle long-term dependencies and fine-grained interactions.
• Foundation for Reasoning: By providing a structured, intermediate representation (scene graphs), the work bridges the gap between raw pixel data and high-level semantic reasoning, enabling better performance in downstream tasks like VQA, robotic navigation, and video retrieval.
• Open Science: The authors release the SVG2 dataset, model checkpoints, and code, fostering further research into large-scale video scene graph generation and open-vocabulary video understanding.