💻 computer science

Automatically Improving Simulation Physics for Articulated Objects

This thesis addresses the challenge of creating simulation-ready articulated objects by defining "interaction-readiness," proposing a quantitative evaluation framework, and introducing a multi-modal, simulator-in-the-loop method that automatically refines incomplete 3D assets into physically consistent models to enhance robot learning stability and performance.

Original authors: Anh-Quan Pham

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anh-Quan Pham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to do chores, like opening a suitcase, closing a laptop, or pressing a stapler. To do this safely and efficiently, you usually don't use the real robot in the real world immediately; that would be too slow and risky. Instead, you teach it in a video game-like simulation.

However, there is a major problem with these simulations: the objects in them are often "fake" in a very specific way.

The Problem: The "Ghost" Objects

Think of the 3D models of objects we have today (like a digital stapler or a cabinet) as ghosts. They look perfect. If you take a photo of them, they look real. If you ask a computer to recognize them, it knows exactly what they are.

But if you try to touch them in the simulation, they fall apart.

If you try to close a laptop lid, it might snap shut instantly like a magnet, or it might float in mid-air.
If you try to push a drawer, it might vibrate uncontrollably or pass right through the table.
If you try to pick up a box, it might be too heavy to lift or so light it flies away.

This happens because the digital files only have the shape and the look of the object. They are missing the physics: how heavy it is, how much friction the hinges have, how the joints move, and how the parts balance. Fixing this manually is like trying to tune a thousand different musical instruments by ear; it takes forever and requires a human expert for every single object.

The Solution: Asset2Sim (The "Simulator-in-the-Loop" Chef)

The author, Anh Quan Pham, created a new method called Asset2Sim. Think of this method as a smart chef who is trying to cook a perfect meal (a realistic simulation) using a recipe that is missing some ingredients (the physics data).

Here is how Asset2Sim works, using a simple analogy:

The Guess (The VLM): First, the system uses a very smart AI (a Vision-Language Model) that has seen millions of videos of real life. The AI looks at the "ghost" object and says, "Okay, this looks like a stapler. Staplers are usually made of plastic, they have a spring inside, and they weigh about 200 grams." The AI makes a guess at the physics.
- The Problem with just guessing: If the AI guesses wrong, the simulation might still crash. The stapler might be too heavy, or the spring might be too stiff.
The Taste Test (The Simulator): This is the magic part. Instead of just trusting the AI's guess, Asset2Sim puts the object into a physics simulator (the "kitchen") and tries to use it.
- It tries to close the lid.
- It tries to press the button.
- It checks: "Did the parts crash into each other? Did the object vibrate? Did it fall over?"
The Correction (The Loop): If the simulation fails (e.g., the lid crashes through the table), the system doesn't give up. It tells the AI: "Your guess was wrong. The object is too heavy, or the joint is too loose. Try again."
- The AI adjusts its guess based on this feedback.
- The system tests it again.
- It repeats this process a few times until the object behaves exactly like a real-world object.

Why This Matters: "Interaction-Readiness"

The paper introduces a new concept called Interaction-Readiness.

Not Ready: An object that looks good but breaks when you touch it.
Ready: An object that looks good and behaves correctly when you push, pull, or lift it.

The author created a report card to grade these objects. Instead of just asking, "Did the robot finish the task?" (which can be lucky), they check:

Stability: Does the object sit still when you aren't touching it?
Realism: Does it move the way a human expects it to? (e.g., Does a heavy door swing slowly?)
Learning: Can a robot learn to use it quickly?

The Results

The paper tested this method on many objects like staplers, suitcases, cabinets, and laptops.

Old Methods: When the AI just guessed the physics without checking, the simulations were often broken. Robots couldn't learn, or the objects exploded in the simulation.
Asset2Sim: By using the "taste test" loop, the system fixed the objects. The robots were able to successfully close the suitcases, press the staplers, and open the drawers in the simulation.
Speed: Even though it checks the physics multiple times, it is still much faster than a human trying to fix every single object by hand.

Summary

In short, this paper solves the problem of "fake" objects in robot training. It takes digital objects that look real but act fake, and uses a smart AI + a physics simulator working together to "tune" them until they act exactly like real-world objects. This allows robots to learn from simulations much faster and more reliably, without needing a human to fix every single toy, tool, or appliance by hand.

Technical Summary: Automatically Improving Simulation Physics for Articulated Objects

Problem Statement

Modern robot learning relies heavily on scalable simulation, yet the effectiveness of these systems is bottlenecked by the quality of object assets. While large-scale 3D datasets (e.g., PartNet-Mobility, ShapeNet) provide rich geometric and kinematic representations, they typically lack the physical properties required for stable and realistic interaction. Constructing "interaction-ready" articulated objects—those that can be reliably simulated under manipulation—currently requires significant manual engineering to assign mass, inertia, friction, damping, and joint dynamics.

Existing automated approaches focus on recovering geometric structure and kinematic relationships but fail to assign physically consistent parameters. Purely predictive methods, including those leveraging Vision-Language Models (VLMs), often produce parameters that are semantically plausible but physically inconsistent, leading to simulation artifacts such as self-interpenetration, spontaneous oscillation, or unstable contact dynamics. Furthermore, there is no widely adopted, quantitative definition of "interaction-readiness" that allows for systematic evaluation independent of downstream task success.

Methodology: Asset2Sim

The thesis proposes Asset2Sim, a multi-modal, simulator-in-the-loop approach for generating interaction-ready articulated objects from incomplete 3D assets. The method reframes physical property estimation not as a static prediction problem, but as a grounded, iterative refinement process. The pipeline consists of three primary stages:

Structured Feature Extraction: An incomplete URDF and its associated meshes are converted into a multi-modal representation ( $I$ ) comprising metadata, geometric data (link properties, joint definitions, mesh volumes), semantic annotations (link labels like "lid" or "hinge"), rendered visualizations, and mesh analysis (volumetric properties).
Physical Property Synthesis: A Vision-Language Model (VLM) acts as a physical reasoner, taking the structured representation and user guidance to produce a structured "overlay" of modifications. This overlay specifies global scale, link inertial properties (mass, inertia tensors), and joint dynamics (damping, friction, stiffness). Crucially, the VLM provides latent priors (e.g., material density based on visual texture), while analytical formulas enforce rigid-body constraints (e.g., calculating mass from volume and density, ensuring inertia tensors are positive definite).
Closed-Loop State Refinement: To resolve instabilities caused by imperfect initial states (e.g., self-collisions), the method employs a simulator-in-the-loop refinement loop. Given a user-specified target state (e.g., "slightly open stapler"), the system simulates the asset, computes a penetration score, and iteratively adjusts joint configurations. The VLM proposes updates to specific "focus joints" implicated in collisions, and the simulator validates whether these updates reduce penetration while preserving the semantic intent. This process continues until a low-penetration, physically stable configuration is found.

Key Contributions

The thesis makes two primary contributions:

An Interaction-Readiness Evaluation Protocol: The work introduces a quantitative framework to decompose interaction-readiness into measurable components, moving beyond downstream task success as the sole metric. The framework includes:
- Physical Robustness: Quantifying equilibrium and stability via penetration depth, position/orientation drift, and joint oscillation.
- Semantic Accuracy: Benchmarking alignment with expert-curated references and user specifications regarding scale and joint configuration.
- Behavioral Fidelity: Assessing whether zero-shot Vision-Language-Action (VLA) policies trained on real data can successfully manipulate the asset in simulation, measured via Sim-to-Real Correlation Coefficient (SRCC).
- Learning Feasibility: Evaluating whether Reinforcement Learning (RL) policies can converge to successful manipulation strategies within the simulated environment.
- Visual and Interaction Realism: Using VLMs as objective judges and human teleoperation to validate motion plausibility and dynamic consistency.
The Asset2Sim Refinement Pipeline: A practical method for automatically generating interaction-ready assets. By integrating VLM reasoning with iterative simulator feedback, the approach corrects inconsistent parameters and ensures physical consistency. It bridges the gap between geometric/kinematic asset construction and the deployment of physically realistic simulations.

Experimental Results

Experiments were conducted across 93 articulated assets spanning 10 object categories (e.g., staplers, cabinets, laptops) and 11 manipulation tasks. The proposed method was compared against baselines including Human-Annotated assets (GenSim2), Direct VLM (one-shot prediction), and Inverse-Variance Weighted VLM (VLM-IVW).

Physical Robustness: Asset2Sim achieved a 100% pass rate on position/orientation stability and joint oscillation tests, significantly outperforming Direct VLM (66.7% and 77.8% pass rates, respectively) and VLM-IVW (77.8% and 77.8%). It also demonstrated the lowest penetration depth among automated methods.
Semantic Fidelity: Asset2Sim achieved the highest prompt alignment score (97.22%), indicating superior adherence to user specifications regarding object scale and initial joint states compared to VLM-IVW (82.22%) and Direct VLM (86.11%).
Behavioral Fidelity: In zero-shot VLA evaluations, Asset2Sim assets yielded the highest simulation success rates and the strongest correlation with real-world performance (SRCC of 0.52–0.68 across different VLA models), whereas baselines showed weak or negligible correlations.
Learning Feasibility: RL policies trained on Asset2Sim assets achieved a 97.53% average success rate across articulation tasks, compared to 45.68% for VLM-IVW and 11.11% for Human (GenSim2) and Direct VLM baselines.
Realism: Human evaluators rated Asset2Sim assets significantly higher (mean score 7.04/10) than all baselines. VLM-as-a-Judge evaluations also ranked Asset2Sim highest, confirming that the refined dynamics align with perceptual expectations of physical realism.
Scalability: On a large-scale set of 93 assets, Asset2Sim achieved a 96% success rate in producing simulation-ready outputs, significantly outperforming VLM-only baselines which suffered from high rates of structural failures and invalid configurations.

Significance and Claims

The thesis argues that the fidelity of object assets is a central factor in enabling scalable and reliable robot learning, rather than a secondary concern. It claims that:

Interaction-readiness is a measurable construct that can be systematically evaluated, revealing failure modes (e.g., numerical instability, semantic misalignment) that qualitative inspection or downstream task success alone cannot detect.
Simulator-in-the-loop refinement is necessary to bridge the gap between foundation model reasoning and physical consistency. Purely predictive approaches, even with VLMs, fail to guarantee stable dynamics because they lack closed-loop validation.
Physical realism directly impacts downstream performance. Improvements in object quality lead to more stable simulations, better alignment with real-world physics, and improved performance in both zero-shot interaction and policy learning settings.

The work concludes that while challenges remain regarding computational overhead and robustness to noisy inputs, the proposed framework and method provide a practical pathway for constructing physically realistic, interaction-ready environments at scale, addressing a critical bottleneck in the trajectory of simulation-based robot learning.