OrigamiBench: An Interactive Environment to Synthesize Flat-Foldable Origamis

1. Problem Statement

Current AI benchmarks often treat visual perception and programmatic reasoning as isolated tasks. While Vision-Language Models (VLMs) excel at pattern recognition, they struggle to develop a causal understanding of physical processes required for sequential decision-making in the real world. Specifically, there is a lack of benchmarks that:

• Integrate visual perception with geometric and physical constraints.
• Require multi-step planning where actions must be physically valid (e.g., adhering to laws of flat-foldability).
• Distinguish between superficial visual similarity and genuine causal reasoning about state transitions.

The authors posit that origami synthesis is an ideal domain to address this gap because it requires transforming a 2D sheet into a target shape through a sequence of folds constrained by strict geometric rules (Maekawa's and Kawasaki's theorems), demanding both visual analysis and recursive logical planning.

2. Methodology: OrigamiBench

The authors introduce OrigamiBench, an interactive, closed-loop benchmark designed to evaluate VLMs on geometric reasoning and sequential planning.

A. Dataset Organization

• Source: 366 distinct origami designs from the "Flat-Folder" project.
• Format: Encoded in .fold files (standardized crease patterns) containing vertex coordinates, edge connectivity, fold assignments (Mountain/Valley), and face definitions.
• Taxonomy: Data is categorized by Semantic Category (e.g., animals, geometry, roles) and Complexity Level (Easy, Medium, Hard) based on vertex/crease counts.
• Rendering: A pipeline converts .fold files into SVG/PNG images showing both the crease pattern and the folded state (front and back views).

B. Interactive Environment

The environment operates as a closed-loop system:

1. State: Maintained as a CreasePattern object.
2. Observation: The agent receives a multimodal prompt including:
   • Target folded shape (PNG).
   • Current folded state (PNG).
   • Current 2D crease pattern (PNG).
   • Binary feedback on the feasibility of the previous action.
3. Action: The agent outputs a structured JSON command to add a new crease (defining vertices and Mountain/Valley assignment).
4. Validation & Execution:
   • Schema Check: Validates JSON structure.
   • Physics Check: Uses the Flat-Folder solver to verify compliance with Maekawa's and Kawasaki's theorems. If the action creates an invalid state, it is rejected.
   • Update: If valid, the state updates, and new observations are rendered.

C. Evaluation Tasks

The benchmark employs two distinct evaluation modes:

1. One-Step Multiple-Choice Evaluation:
   • Goal: Assess causal understanding of a single fold.
   • Setup: Given a reference state and four candidates (A, B, C, D), the model must identify the state resulting from exactly one additional fold.
   • Variants:
     • Associative: Distractors are from unrelated origamis (tests visual similarity).
     • Causal: Distractors are from the same folding sequence but represent incorrect steps (tests causal reasoning).
2. Full-Step Interactive Evaluation:
   • Goal: Assess multi-step planning and synthesis.
   • Setup: The model starts with a blank sheet and must iteratively fold to match a target shape.
   • Metrics:
     • Query Efficiency (QE): Percentage of steps contributing to the final shape.
     • Geometric Similarity (GS): Intersection over Union (IoU) of the silhouette.
     • Semantic Similarity (SS): Cosine similarity of CLIP embeddings (fine-tuned on origami data).

3. Key Contributions

• OrigamiBench: The first interactive benchmark integrating visual perception, geometric constraints, and sequential planning for physical synthesis.
• Causal vs. Associative Distinction: A novel evaluation framework that explicitly separates surface-level pattern matching from deep causal reasoning about physical transformations.
• Closed-Loop Simulation: A robust environment using a physics-compliant solver (Flat-Folder) to provide real-time feedback on action validity, preventing models from hallucinating physically impossible states.
• Domain-Specific Metrics: Introduction of IoU-based geometric and fine-tuned CLIP-based semantic metrics specifically for origami synthesis.

4. Experimental Results

The authors evaluated modern VLMs (Qwen3-VL, Qwen2.5-VL, InternVL, MiniCPM-V) on the benchmark.

A. One-Step Evaluation Findings

• Associative Setting: Models performed exceptionally well (e.g., Qwen3-VL-32B achieved 96.18%), indicating strong capability in visual pattern matching and recognizing local geometric correspondences.
• Causal Setting: Performance dropped drastically. The best model (Qwen3-VL-32B) only achieved 43.37%, while others hovered near random guessing (25%).
• Conclusion: Current VLMs struggle to infer the underlying physical operation (the "cause") rather than relying on visual similarity. Scaling model size alone does not solve this; the gap suggests a failure in integrating language rules with visual representations.

B. Full-Step Interactive Evaluation Findings

• Planning Failure: Models could interact with the environment and produce syntactically valid actions (high QE), but failed to generate coherent multi-step plans.
• Outcome: Models tended to synthesize very simple structures or get stuck, failing to reach complex targets.
• Root Cause: The inability to plan stems directly from the lack of causal understanding observed in the one-step task. Without understanding why a fold works, models cannot sequence actions to reach a goal.

5. Significance and Future Directions

• Limitation of Current AI: The study demonstrates that simply increasing model parameters does not yield causal reasoning about physical transformations. VLMs currently lack a robust internal "language" that links visual observations to physical constraints.
• Path Forward: The authors propose:
  • Explicit Intermediate Representations: Training models to predict intermediate states (e.g., crease graphs, layer ordering) rather than just raw pixels.
  • Reinforcement Learning: Using the closed-loop simulator for execution-based supervision and reward learning based on foldability.
  • Curriculum Learning: Designing training curricula that emphasize "hard-negative" discrimination within the same folding sequence to force causal understanding.

Summary: OrigamiBench reveals a critical bottleneck in current AI: the inability to translate visual perception into causal, sequential physical reasoning. While models are excellent at "seeing" what a fold looks like, they fail to understand "how" and "why" a fold transforms the object, highlighting the need for new architectures that better integrate geometry, physics, and language.