Technical Summary: LychSim

Problem Statement

While self-supervised and weakly-supervised pretraining have reduced the computer vision community's reliance on manually curated synthetic datasets for direct supervised training, simulation remains indispensable for two critical objectives: (1) rigorous Out-of-Distribution (OOD) evaluation and robustness analysis, and (2) closed-loop optimization for embodied AI and interactive agents. However, existing modern simulation platforms often present steep technical barriers. They require extensive expertise in computer graphics, C++, and game engine architecture (specifically Unreal Engine 5), creating a friction point that limits their adoption by vision researchers who lack this background. Furthermore, existing solutions often lack unified interfaces for diverse asset types or fail to provide the rich, semantically aligned 3D ground truths necessary for advanced 3D representation learning.

Methodology

The authors present LychSim, a controllable and interactive simulation framework built upon Unreal Engine 5 (UE5) designed to bridge the gap between complex 3D graphics engines and the needs of vision research. The system is defined by three core design pillars:

1. Streamlined Python API and Asset Abstraction

To lower the entry barrier, LychSim provides a high-level Python API that abstracts the underlying complexities of UE5 and C++.

• Unified Interface: The API handles the heterogeneity of UE5 assets (StaticMesh, SkeletalMesh, Blueprints) through a single add_obj command, allowing researchers to spawn and manipulate diverse objects without managing engine-specific workflows.
• Efficient Workflow: Researchers can programmatically control scene layouts, adjust 3D coordinates, and retrieve synchronized ground truths (RGB, depth, segmentation, point maps) via simple function calls, eliminating the need for prior game engine experience.

2. Procedural Data Pipeline with Rich Ground Truths

LychSim employs a hybrid approach to scene generation, leveraging high-quality, artist-created assets from the UE5 Fab Asset Marketplace and augmenting them with procedural rules.

• Data Extensions: The system introduces two key data extensions to raw assets:
  • Object Annotations: Semantic categories, canonical scales, and pose alignments to ensure semantically aligned 3D ground truths.
  • Procedural Rules: Scene-level constraints (e.g., navigable floors, road areas, pedestrian trajectories) that guide structural generation while maintaining semantic faithfulness.
• Ground Truth Diversity: Beyond standard 2D labels, LychSim generates comprehensive 3D ground truths, including:
  • Beyond Visible Regions: Instance-level depth buffers and geometric projections to recover occlusion relationships and truncation ratios for objects extending outside the image plane.
  • Part-Level Segmentation: Customized render targets outputting object part IDs and per-pixel 3D vertex positions (dense point maps).
  • Environmental Variables: Macroscopic parameters for lighting, fog, and rain to simulate challenging weather and illumination shifts.

3. Native Model Context Protocol (MCP) Integration

To enable interactive, closed-loop research, LychSim integrates the Model Context Protocol (MCP).

• Agentic Interaction: A dedicated MCP server exposes the Python API as a suite of standardized tools, allowing reasoning agentic Large Language Models (LLMs) to autonomously navigate, query scene states, capture visual feedback, and manipulate objects in real-time.
• Closed-Loop Playground: This integration transforms the simulator from a static data generator into a dynamic environment where agents can iteratively plan, execute, and verify 3D scene layouts based on natural language instructions.

Key Contributions

1. LychSim Framework: A publicly available, highly controllable simulation framework built on UE5 that democratizes access to high-fidelity 3D simulation for vision researchers.
2. Unified Python API: A streamlined interface that abstracts UE5 asset complexities, enabling unified control over diverse object types (static, skeletal, blueprints) and automated data generation.
3. Advanced Ground Truths: A novel pipeline generating semantically aligned 3D ground truths, including part-level segmentation, dense point maps, and quantitative occlusion/truncation metrics that are difficult to obtain from real-world data.
4. Agentic Integration: The first native integration of MCP with a high-fidelity UE5 simulator, facilitating interactive, language-driven scene planning and adversarial testing.
5. Open Release: The authors commit to releasing the full C++ and Python source code, MCP server implementation, procedural rules, and object-level annotations under permissive licenses (MIT for code, CC BY 4.0 for annotations).

Results and Case Studies

The paper demonstrates LychSim's utility through three primary case studies:

• Synthetic Data Engine: LychSim serves as a scalable engine for generating post-training data to enhance spatial understanding in Vision-Language Models (VLMs). It is used to diagnose weaknesses in spatial reasoning models (e.g., Unreal3DSpace, PerceptualTaxonomy) by creating targeted OOD scenarios with varying visual complexities.
• Adversarial Examiners: The framework powers reinforcement learning-based adversarial examiners. In a specific study, a Gaussian policy was trained to systematically explore 3D camera viewpoints to minimize the Intersection-over-Union (IoU) of the Segment Anything Model (SAM). Results showed that even on common objects in simple environments, the adversarial examiner could effectively identify model weaknesses and failure modes.
• Interactive Scene Planning: Leveraging agentic LLMs (e.g., Claude Opus 4.6, Gemma 4) via MCP, the system facilitates multi-turn, language-driven scene layout generation. Agents can plan scenes based on specifications, navigate to verify physical plausibility (e.g., detecting floating objects), and iteratively edit layouts based on user feedback. While the paper notes failure patterns such as physically implausible layouts due to current LLM spatial reasoning limits, it highlights the potential for this closed-loop approach.

Significance

The authors position LychSim as a critical tool for advancing computer vision research toward a more accurate understanding and generation of the 3D world. By removing the technical barriers of game engine development, the framework enables a broader community to:

• Conduct rigorous, controlled evaluations of model robustness against OOD shifts.
• Develop and test interactive AI systems in safe, high-fidelity virtual playgrounds.
• Explore richer 3D representations using novel ground truths like part-level segmentation and occlusion metrics.

The paper concludes that graphics-based simulation remains a vital, scalable data engine for model training and a rigorous evaluation framework, with LychSim serving as a unifying platform that bridges rule-based procedural generation and LLM-driven agentic planning.