ArtLLM: Generating Articulated Assets via 3D LLM

ArtLLM is a novel framework that leverages a 3D multimodal large language model to autoregressively predict kinematic structures and generate high-fidelity articulated 3D assets directly from complete meshes, significantly outperforming existing methods in accuracy and generalization for applications like robotics and simulation.

The Gist

Imagine you want to build a digital world for a video game or a robot training simulation. You need more than just static statues; you need objects that can move, like a door that swings, a drawer that slides, or a robot arm that bends. These are called articulated assets.

For a long time, creating these moving parts has been like trying to build a complex Lego set without instructions, or worse, trying to find the exact right Lego bricks in a giant, dusty warehouse where the boxes are all labeled "Miscellaneous."

ArtLLM is a new tool that changes the game. Think of it as a super-smart digital architect that can look at a picture of a chair or a robot and instantly say, "Ah, I see! This chair has four legs and a seat that can tilt. Let me build you a perfect, moving version of it right now."

Here is how it works, broken down into simple steps:

1. The "Brain" (The 3D LLM)

Most AI models are good at making pictures, but they don't really understand how things move. ArtLLM uses a special kind of AI called a 3D Large Language Model.

• The Analogy: Imagine a master carpenter who speaks a language of blueprints. Instead of just looking at a photo of a door, this carpenter reads the "language" of the door. They understand that a door needs a hinge, a handle, and a frame.
• How it works: ArtLLM looks at a 3D shape (like a cloud of points representing the object) and "speaks" a blueprint. It doesn't just guess the shape; it writes a list of instructions: "Part A is the box. Part B is the lid. Connect them with a hinge here, and make sure the lid only opens 90 degrees."

2. The "Builder" (Part Generation)

Once the "Brain" writes the blueprint, it hands it off to a specialized "Builder."

• The Analogy: In the past, if you wanted a drawer, you had to pull a pre-made drawer out of a limited catalog. If the catalog didn't have the exact right size or style, you were stuck with a mismatch.
• How it works: ArtLLM's builder doesn't use a catalog. It creates the parts from scratch based on the blueprint. If the blueprint says "a drawer with a curved handle," the builder sculpts that exact curved handle. This means every object is unique and fits the original image perfectly.

3. The "Safety Inspector" (Physics Check)

Sometimes, even a good blueprint has a flaw. Maybe the drawer is designed to slide out so far that it hits the wall and breaks.

• The Analogy: Imagine building a toy car. You might build the wheels, but if you don't check the limits, the wheels might spin off the car.
• How it works: ArtLLM has a final safety step. It simulates the movement in its head. If it sees that a door would crash into the wall if opened too wide, it automatically adjusts the "stop" on the hinge. It ensures the object moves realistically without breaking or crashing into itself.

Why is this a Big Deal?

• No More "Cookie Cutter" Objects: Old methods were like using a stamp; they could only make things that already existed in a database. ArtLLM is like a sculptor; it can make anything you can imagine.
• Speed: Old methods took hours or days to figure out how a single object moves. ArtLLM does it in seconds.
• Realism for Robots: This is huge for robotics. Right now, robots train in simulations. If the simulation objects don't move like real objects, the robot learns the wrong lessons. ArtLLM creates "Digital Twins" that move exactly like the real world, helping robots learn faster and safer.

In a nutshell:
ArtLLM is like giving a robot a pair of eyes and a brain that understands not just what an object looks like, but how it works. It turns a static picture into a fully functional, moving digital toy, ready to be played with in a video game or used to train a real-life robot.

Technical Summary

1. Problem Statement

The creation of interactive digital environments for gaming, robotics, and simulation relies heavily on articulated 3D objects (e.g., doors, drawers, machinery) that possess both geometric parts and kinematic structures (joints). Existing methods for generating these assets face significant limitations:

• Optimization-based methods: Require slow, per-object joint fitting, often handle only simple single-joint objects, and produce low-fidelity geometry.
• Retrieval-based methods: Assemble parts from fixed libraries, leading to repetitive geometry, poor generalization to novel shapes, and a lack of geometric novelty.
• General 3D Generation: While high-fidelity 3D generation has advanced, there is a fundamental disconnect between geometry and motion. Existing models lack awareness of kinematic structures, often resulting in parts that look correct but cannot move mechanically as intended.

The core challenge is to generate high-quality, physically grounded articulated assets directly from inputs (images or text) that feature novel geometries, accurate part layouts, and correct kinematic relationships.

2. Methodology: ArtLLM Framework

The authors propose ArtLLM, a novel framework that treats articulated asset generation as a language modeling problem. The pipeline consists of four main stages:

A. 3D Articulation Language Model (ArtLLM)

Instead of regressing continuous parameters directly, ArtLLM reformulates the kinematic structure of an object as a tokenized language sequence.

• Input: The model takes a point cloud representation of an object (derived from images, text, or existing meshes).
• Architecture: It uses a Point Transformer v3 encoder to process the point cloud, projecting features into a language model backbone (Qwen3 0.6B).
• Output: The model autoregressively predicts a "blueprint" consisting of:
  1. Part Layouts: Defined as 3D axis-aligned bounding boxes (AABBs) with discrete tokenized coordinates.
  2. Kinematic Joints: Defined as structured text tokens specifying joint type (Revolute, Continuous, Prismatic, Screw), connectivity (parent/child IDs), axis direction, origin, and motion limits.
• Quantization: To ensure numerical stability, all continuous geometric and kinematic parameters (coordinates, angles, limits) are discretized into bins (e.g., 128 bins for coordinates, 48 for rotation angles) and mapped to a vocabulary.

B. Training Strategy

The model is trained on a large-scale, curated dataset of 20,673 articulated objects (combining PartNet-Mobility, PhysX3D, and procedurally generated Infinite-Mobility data).

• Multi-Task Learning: The training involves three supervised fine-tuning (SFT) tasks:
  1. Part Layout Prediction (from point cloud).
  2. Kinematic Prediction (from point cloud + ground-truth layout).
  3. End-to-End Articulation Prediction (from point cloud).
• Two-Stage Training:
  • Stage 1: Train only on Part Layout Prediction to establish a robust geometric foundation for the 3D encoder (initialized with P3SAM weights).
  • Stage 2: Fine-tune the full model on all three tasks to refine kinematic reasoning.

C. Part-Aware Geometry Synthesis

Once the structural blueprint (part boxes and joints) is predicted, a part-aware generative model (specifically XPart) is conditioned on these bounding boxes to synthesize high-fidelity 3D mesh geometries for each part.

• Robustness: A bounding box expansion step ensures that the generated geometry fully covers the input point cloud, preventing truncation artifacts.

D. Physically-Constrained Joint Limit Correction

To ensure the generated assets are physically plausible and collision-free:

• The system performs a post-processing step where it simulates the articulation of the predicted joints.
• It detects collisions between moving parts and static parts.
• If collisions occur, the joint limits are refined hierarchically to the point of first contact, ensuring the final asset is safe for simulation.

3. Key Contributions

1. Unified Framework: ArtLLM is the first framework to jointly reason over geometry and articulation using a 3D Large Language Model, bridging the gap between visual semantics and mechanical function.
2. Tokenized Articulation: By converting kinematic structures into discrete tokens, the method achieves stable autoregressive prediction of variable numbers of parts and joints, overcoming the instability of direct regression.
3. High-Fidelity & Novelty: Unlike retrieval-based methods, ArtLLM generates novel geometries that match the input shape while maintaining correct kinematic structures.
4. Physics-Grounded Output: The inclusion of a collision-based limit correction module ensures the generated assets are ready for physical simulation without manual tuning.
5. Scalable Dataset: The creation of a large-scale, diverse training corpus (20k+ objects) specifically curated for articulation learning.

4. Experimental Results

The method was evaluated on the PartNet-Mobility dataset (7 categories) and compared against state-of-the-art baselines (URDFormer, SINGAPO, Articulate-Anything).

• Quantitative Performance: ArtLLM significantly outperforms baselines across all metrics:
  • Part Layout (mIoU): 0.6884 (vs. 0.4330 for SINGAPO).
  • Joint Type Accuracy: 90.84% (vs. 84.57% for Articulate-Anything).
  • Joint Axis Error: 0.1271 (significantly lower than competitors).
  • Graph Accuracy: 77.41% (vs. 61.42% for Articulate-Anything).
• Efficiency: The inference time is 19 seconds, drastically faster than optimization-based methods (e.g., 183s for URDFormer) and retrieval-based methods (522s for Articulate-Anything).
• Generalization: The model generalizes well to real-world images and objects outside the training categories.
• Real-to-Sim Application: In a robotic teleoperation test (Franka Panda arm), ArtLLM successfully reconstructed real-world objects into simulation-ready URDFs, allowing the robot to reproduce real-world tasks (closing a laptop, moving a bucket) in a simulated environment with high fidelity.

5. Significance

ArtLLM represents a paradigm shift in 3D content creation for robotics and simulation.

• For Robotics: It enables scalable robot learning by rapidly generating high-fidelity digital twins of real-world objects, reducing the "Sim-to-Real" gap.
• For Simulation: It provides a pipeline to create diverse, physically valid interactive environments without manual modeling.
• For AI Research: It demonstrates the potential of 3D LLMs to handle complex structural reasoning, moving beyond simple shape generation to functional, articulated object synthesis.

The work establishes a new standard for generating articulated assets that are not only visually accurate but also mechanically functional and ready for deployment in physical simulations.